Compare commits
No files matched your search
+161
-10
@@ -386,10 +386,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
|
||||
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
|
||||
verified email, roles and memberships from groups, `no_passwords`) and
|
||||
SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
|
||||
signed with a per-process key, so with two control planes behind a
|
||||
balancer the callback must reach the one that started it (sticky
|
||||
sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||||
SCIM 2.0 (`/scim/v2` Users and Groups). The sign-in cookie's key comes
|
||||
from the store (2026-10-10), so a callback may reach either control
|
||||
plane behind a balancer. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||||
`internal/web/account.go`; map IdP groups to org/team/admin.
|
||||
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
|
||||
project), `keys rotate --overlap`, kinds and expiry in `keys list`.
|
||||
@@ -405,8 +404,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand; each control plane sends
|
||||
budget events on its own, so two behind a balancer may both send one.
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
|
||||
templates driven by `server init` or the console; llmgw-native webhooks
|
||||
@@ -1091,16 +1091,167 @@ at a third of its CPUs.
|
||||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||||
took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU
|
||||
profile on llama.cpp, which needs a source per profile (the catalog has
|
||||
one per model; the import swapped the entry by hand), and an admission
|
||||
queue limit that suits bulk imports (30 s turned some away).
|
||||
took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
|
||||
profile, so an embedding model's GPU profile runs vLLM and its CPU
|
||||
profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
|
||||
`Retry-After` on a busy refusal, which is what had turned passages away
|
||||
(llmgw-kb retries only what says when to).
|
||||
- Retrieval on 40 questions written from the corpus, document in the top
|
||||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||||
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
|
||||
chat model, and a reranker.
|
||||
|
||||
## G. Jobs: 3D generation through ComfyUI (2026-10-10)
|
||||
|
||||
A request for a 3D model takes minutes on the one 24 GB card, and the card has to be
|
||||
emptied of the chat model first. That is not a request to hold a socket for
|
||||
— `internal/router/activate.go:46` already says so — it is a job: queued,
|
||||
answered with an id, asked after, its result downloaded. The job machinery
|
||||
is built first and tested without a GPU; the ComfyUI generalisation and the
|
||||
custom-node install come after. Decided 2026-10-10: a `jobs` table of its
|
||||
own rather than a one-line batch (the JSONL shape would be artificial and
|
||||
has nowhere for progress), and progress is shown, not just a status.
|
||||
|
||||
Which models (decided 2026-10-10): the assets go into a game sold on
|
||||
Steam, worldwide, the EU included, so the licence of the output decides
|
||||
before the quality does. Hunyuan3D (every version) is out: Tencent's
|
||||
licence excludes the EU, UK and South Korea, and whether a game shipped
|
||||
there with its outputs is "use in the Territory" is not a question to
|
||||
leave to a court. In, in this order: **TRELLIS.2** shape (MIT; the best
|
||||
geometry; its own texture baking goes through nvdiffrec and nvdiffrast,
|
||||
both under NVIDIA's non-commercial source licence, so that stage is not
|
||||
run), **Step1X-3D** (Apache 2.0, StepFun; shape and texture in one, the
|
||||
EU-safe textured path — its ComfyUI node and VRAM need are to be
|
||||
checked), **Direct3D-S2** (MIT, DreamTech; high-detail shape, has a
|
||||
ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and
|
||||
InstantMesh (permissive but superseded), Stable Fast 3D (Stability's
|
||||
community licence, revenue-capped), SAM 3D Objects (Meta's own licence,
|
||||
to be read before it is considered). Steam's disclosure form (changed
|
||||
2026-01-16) lists pre-generated 3D models as content that must be
|
||||
declared on the store page: the licence notice G3 writes into the GLB is
|
||||
what the studio answers that form from.
|
||||
|
||||
### G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch `jobs-g1`)
|
||||
Built as written, with two choices: a job found running after its gateway
|
||||
died is run again from the start (the prompt id column is there for G2 to
|
||||
follow the same work instead), and the request is validated at the door
|
||||
only as far as costs nothing — key, model, allowlist, the request's shape
|
||||
— with limits, guardrails and admission applied when the worker replays it,
|
||||
said as the job's error. The result file's `content_type` is the job's; the
|
||||
files endpoint still serves every file as an octet stream.
|
||||
- Store: `jobs` (id, owner key/principal, kind, model, status, input
|
||||
(JSON), progress (JSON: queue position, percent, step), result file id,
|
||||
error, created/started/finished, lease owner/until), migrations for
|
||||
sqlite and postgres beside `internal/store/batches.go`, whose
|
||||
claim/renew/release lease pattern is copied, not shared: a job is one
|
||||
unit of work, a batch is many lines.
|
||||
- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any
|
||||
gateway on the store claims a queued job, asks for the model with
|
||||
`activator.Want` and waits as long as it takes (no `ActivationWait`
|
||||
bound: nobody is holding a connection), runs it, stores the output
|
||||
through the files store, writes usage and price, marks it done. A
|
||||
gateway that dies mid-job loses its lease and another one picks it up;
|
||||
the ComfyUI prompt id is in the job row so the second gateway asks
|
||||
after the same work rather than queueing it again.
|
||||
- Endpoints: `POST /v1/3d/generations` validates like any inference
|
||||
request (key, allowlist, limits, admission, guardrails on the prompt),
|
||||
writes the job, answers 202 with `{id, status: "queued"}`;
|
||||
`GET /v1/3d/generations/{id}` returns status, progress and, when done,
|
||||
`output_file_id` and the usage; `POST .../{id}/cancel`; the file comes
|
||||
from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results
|
||||
are kept for the batch files' retention and deleted with them: a
|
||||
departure from `images.go`'s "kept nowhere", because a mesh is a
|
||||
download, not a JSON field; the retention is the bound.
|
||||
- Done when: a job survives the gateway that took it being killed, a
|
||||
cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two
|
||||
gateways never both run one job (test with the lease), and the usage
|
||||
row carries the job's price.
|
||||
|
||||
### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`)
|
||||
Built as written: the gate's websocket under its own client id, the
|
||||
client_id of every /prompt rewritten, /progress/{prompt_id} on the gate,
|
||||
the router reading it and /queue into the job (only into a job of the same
|
||||
key). Left: a node that reports no steps (TripoSR's) shows its name and no
|
||||
percent; a vanished replica is said by the job failing, not by progress.
|
||||
The gate (`internal/comfy/gate.go`) passes seven paths and no websocket.
|
||||
Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once,
|
||||
with `golang.org/x/net/websocket` (already a dependency), under a client
|
||||
id of its own, and rewrites the `client_id` of every `/prompt` it
|
||||
forwards to that one. It keeps the last `executing`, `progress`
|
||||
(value/max/node) and `execution_error` message per prompt id and serves
|
||||
them at `GET /progress/{prompt_id}` on the gate; the router polls that
|
||||
beside `/history` and `/queue` and writes queue position, percent and the
|
||||
current node's class into the job's progress. The gate's buffer is
|
||||
bounded (the last N prompts) and the websocket reconnects. Done when a
|
||||
3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in
|
||||
`ctl jobs` and on the account page, and a job whose replica vanished
|
||||
says so within one poll.
|
||||
|
||||
### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR
|
||||
Built: output kinds, image upload through the gate, every output list
|
||||
fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh
|
||||
and usage.meshes (migration 0045), custom nodes cloned at their commit by
|
||||
the gate with packages beside the replica (a node's named packages at its
|
||||
range, the rest at ComfyUI's versions). Not as written: the model. On the
|
||||
PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3,
|
||||
its published background remover (RMBG-2.0) is non-commercial, and its
|
||||
sparse attention and Direct3D-S2's need flash-attention (Ampere and
|
||||
newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture
|
||||
in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT,
|
||||
as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated
|
||||
encoder, a missing node refusing the replica by name.
|
||||
- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an
|
||||
output kind; `ServesComfyUI` stays). 3D weight folders in the gate's
|
||||
`modelFolders`; the workflow's placeholders gain `image` (the input
|
||||
picture, base64 in the request, written to the replica's `input/` by
|
||||
the gate through a new `POST /upload/image` the gate allows — or a
|
||||
base64-loading node, whichever TRELLIS.2's wrapper wants).
|
||||
- `comfy.Client.fetch` reads every output key, not `images` only, and
|
||||
fetches each file named by `/view`; `Image` becomes `Artifact` with its
|
||||
content type (model/gltf-binary, application/octet-stream for .obj).
|
||||
- Custom nodes: a model (or the engine config) names git repositories
|
||||
and a pip requirements list; the agent's provision step clones them
|
||||
into the replica's `custom_nodes/` (the gate's `--base-directory` is
|
||||
where ComfyUI looks, not the install directory) and installs the
|
||||
requirements into the engine's venv with the same PyTorch index and
|
||||
lock discipline as A8. A replica with a missing node refuses to start
|
||||
with the node's name in the error rather than ComfyUI's traceback.
|
||||
- Licence of the output (decided 2026-10-10: it matters). A catalog model
|
||||
carries `license: {name, url, notice}`; a job's answer, `/v1/models`
|
||||
and the portal page show it beside the result, and the mesh's GLB
|
||||
`asset.extras` carries the notice too, since the file travels without
|
||||
the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and
|
||||
its outputs free, but its texture baking depends on nvdiffrast, whose
|
||||
NVIDIA Source Code License is non-commercial — a commercial service
|
||||
must render textures another way or skip that stage. Hunyuan3D 2.0 and
|
||||
2.1 claim no rights in outputs, but the licence excludes the EU, UK and
|
||||
South Korea, needs Tencent's permission above 1 M monthly active users,
|
||||
forbids using outputs to train other models, and asks for "Powered by
|
||||
Tencent Hunyuan"; the notice ships with any distribution. TripoSR is
|
||||
MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse,
|
||||
some of whose objects are CC-BY-NC: a provenance question no licence
|
||||
text settles, to be said in the notice rather than hidden.
|
||||
- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`,
|
||||
`Outcome.Meshes`, the usage sums and exports (as eaea671 did for
|
||||
pictures).
|
||||
- Done when: TRELLIS.2 on cuda01 with `scaling: {min: 0,
|
||||
idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint
|
||||
with nvdiffrast and nvdiffrec not installed in its venv (the licence
|
||||
is kept by the install, not by a note), the GLB carries the MIT
|
||||
notice,
|
||||
the card is free again after the idle timeout, and the same request
|
||||
against flux-schnell is refused as "makes pictures, not meshes".
|
||||
|
||||
### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`)
|
||||
Built: /3d with model-viewer served from the gateway, progress, download
|
||||
and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs
|
||||
was taken by the control plane's own work). Left: the console's Jobs view.
|
||||
A 3D page beside `/images` (upload a picture, watch progress, download
|
||||
the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
|
||||
jobs list|get|cancel`, the console's Jobs view. The page sends down the
|
||||
same path as the API, like `imagesSend` does.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
+83
-1
@@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
@@ -26,6 +26,88 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## v0.14.3 — 2026-10-10
|
||||
|
||||
3D models from a picture, as jobs: made by ComfyUI on the cluster's cards,
|
||||
with their progress, their licence written into the mesh, a price per mesh
|
||||
and a page to make them. Also: a profile's own weights, a band for bulk
|
||||
work, and fixes for control planes behind a balancer and agents that
|
||||
restart.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains tables and columns (migrations 0043–0045: notify_sent,
|
||||
jobs, usage meshes); nothing to do.
|
||||
- **Nodes** need this release for their engines' output to survive an
|
||||
agent restart and for ComfyUI's custom nodes and progress; engines
|
||||
started by an older agent keep its pipe until they are restarted.
|
||||
- A 3D model needs a model that makes them (see install.md, "A model that
|
||||
makes them"); none is added.
|
||||
|
||||
### Added
|
||||
|
||||
- **Jobs: 3D models asked after, not waited for.** `POST
|
||||
/v1/3d/generations` answers 202 with a job id; `GET
|
||||
/v1/3d/generations/{id}` says its status, progress and, when done, the
|
||||
file its result is in (`/v1/files/{id}/content`); `POST .../cancel` stops
|
||||
it, in the engine's queue too. The request is replayed by whichever
|
||||
gateway holds the job's lease through the same path a client's request
|
||||
takes — key, allowlist, limits, admission, guardrails, accounting — so
|
||||
it is charged like any request, tagged `job=<id>`, and a model still
|
||||
loading is waited out for as long as the job has (24 h) rather than
|
||||
refused. A job survives the gateway that took it; results are kept for
|
||||
`router.batch.retain_days`. Migration 0044.
|
||||
- **ComfyUI models that make meshes** (`comfyui.output: meshes`). The
|
||||
picture is uploaded to the replica's ComfyUI and the workflow's
|
||||
`{{image}}`; the mesh the workflow saves is the job's file, an OBJ turned
|
||||
into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards
|
||||
stronger ones can be added the same way.
|
||||
- **Custom nodes** (`comfyui.nodes`): a repository at a full commit, put in
|
||||
each replica's own `custom_nodes` by the gate, with its packages
|
||||
installed beside the replica, never into ComfyUI's environment.
|
||||
- **A job's progress**: the gate listens to ComfyUI's websocket and the job
|
||||
says queued and how many ahead, then the node running, its step and
|
||||
percent.
|
||||
- **A model's `license`** (name, url, notice), in `/v1/models`, in the job,
|
||||
on the portal and in the GLB's `asset.extras`.
|
||||
- **`per_mesh`** in `auth.pricing`; usage rows, sums and exports count
|
||||
`meshes`.
|
||||
- **`/3d` on the portal**, beside `/images`: a picture chosen, the job
|
||||
followed, the mesh in a viewer with a download and its licence; and
|
||||
**`llmgw ctl 3d list|get|cancel`** for every owner's jobs.
|
||||
- **A profile's own `source`**: the weights it loads instead of the
|
||||
model's, so an embedding model can run on vLLM from its repository on a
|
||||
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
|
||||
nodes holding each profile's weights, `ctl model add` checks every
|
||||
source, and `ctl weights` collects none a profile loads.
|
||||
- **A band's own `queue_ttl`** (`router.admission.bands[].queue_ttl`): how
|
||||
long its requests may wait for room, instead of the admission's 30 s. A
|
||||
band for bulk work — a knowledge-base import — waits behind people rather
|
||||
than being turned away.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An engine's output no longer breaks when its agent restarts.** Engines
|
||||
wrote to a pipe of the agent's, which an agent that restarted and adopted
|
||||
them left with no reader; an engine's next large write failed, and in a
|
||||
Python engine took the request it was reporting on with it. Each process
|
||||
now writes to its own file in the agent's work directory, and an adopted
|
||||
engine's log keeps what it wrote before the restart.
|
||||
- **Single sign-on behind a balancer.** The cookie that carries a sign-in
|
||||
across the identity provider was signed with a key of each process's own,
|
||||
so with two control planes a callback that reached the other one was
|
||||
refused. The key is now a secret in the store, the same on every control
|
||||
plane.
|
||||
- **Notifications sent once by two control planes.** Each kept its own
|
||||
hour of quiet per event, so a budget spent on a team both served was
|
||||
announced twice. The control plane that claims an event in the store
|
||||
(migration 0043) sends it; with the store unreachable it is sent anyway.
|
||||
- **A request turned away for a busy cluster says when to come back.** A
|
||||
request that waited as long as it may (504) or found the queue full (503)
|
||||
had no `Retry-After`, so `llmgw-kb` counted the passage as failed instead
|
||||
of sending it again; both now say 10 seconds.
|
||||
- The usage sums and exports count pictures (`images`), as the request
|
||||
records did; pictures made before are added to the sums.
|
||||
|
||||
## v0.14.2 — 2026-10-10
|
||||
|
||||
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
|
||||
|
||||
@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
|
||||
@@ -29,6 +29,9 @@
|
||||
{
|
||||
"name": "capacity"
|
||||
},
|
||||
{
|
||||
"name": "client-jobs"
|
||||
},
|
||||
{
|
||||
"name": "eval"
|
||||
},
|
||||
@@ -280,6 +283,133 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/client-jobs": {
|
||||
"get": {
|
||||
"operationId": "getClientJobs",
|
||||
"summary": "Jobs clients asked for — 3D models — of every owner, newest first",
|
||||
"tags": [
|
||||
"client-jobs"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "status",
|
||||
"in": "query",
|
||||
"description": "Only jobs in this status: queued, running, cancelling, completed, failed, cancelled, expired.",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "limit",
|
||||
"in": "query",
|
||||
"description": "The most jobs to answer with; 50 when absent.",
|
||||
"schema": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/JobView"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/client-jobs/{id}": {
|
||||
"get": {
|
||||
"operationId": "getClientJobsId",
|
||||
"summary": "One job a client asked for",
|
||||
"tags": [
|
||||
"client-jobs"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The job's ID, job_….",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/JobView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/client-jobs/{id}/cancel": {
|
||||
"post": {
|
||||
"operationId": "postClientJobsIdCancel",
|
||||
"summary": "Stop a job, in the engine's queue too",
|
||||
"tags": [
|
||||
"client-jobs"
|
||||
],
|
||||
"parameters": [
|
||||
{
|
||||
"name": "id",
|
||||
"in": "path",
|
||||
"required": true,
|
||||
"description": "The job's ID, job_….",
|
||||
"schema": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
{
|
||||
"$ref": "#/components/parameters/Actor"
|
||||
}
|
||||
],
|
||||
"responses": {
|
||||
"200": {
|
||||
"description": "OK",
|
||||
"content": {
|
||||
"application/json": {
|
||||
"schema": {
|
||||
"$ref": "#/components/schemas/JobView"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"401": {
|
||||
"$ref": "#/components/responses/Unauthorized"
|
||||
},
|
||||
"default": {
|
||||
"$ref": "#/components/responses/Error"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"/api/v1/eval/runs": {
|
||||
"get": {
|
||||
"operationId": "getEvalRuns",
|
||||
@@ -6016,6 +6146,52 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"JobView": {
|
||||
"type": "object",
|
||||
"description": "JobView is one job.",
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string"
|
||||
},
|
||||
"owner": {
|
||||
"type": "string"
|
||||
},
|
||||
"key_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"endpoint": {
|
||||
"type": "string"
|
||||
},
|
||||
"model": {
|
||||
"type": "string"
|
||||
},
|
||||
"status": {
|
||||
"type": "string"
|
||||
},
|
||||
"progress": {
|
||||
"description": "Any JSON value."
|
||||
},
|
||||
"output_file_id": {
|
||||
"type": "string"
|
||||
},
|
||||
"content_type": {
|
||||
"type": "string"
|
||||
},
|
||||
"error": {
|
||||
"type": "string"
|
||||
},
|
||||
"created_at": {
|
||||
"type": "string",
|
||||
"format": "date-time"
|
||||
},
|
||||
"started_at": {
|
||||
"type": "string"
|
||||
},
|
||||
"finished_at": {
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"JudgeAskRequest": {
|
||||
"type": "object",
|
||||
"description": "JudgeAskRequest is a text to have a judge read.",
|
||||
@@ -7599,6 +7775,10 @@
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"meshes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"cost_micros": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
@@ -8573,6 +8753,10 @@
|
||||
"images": {
|
||||
"type": "boolean",
|
||||
"description": "Images is a model that makes pictures, asked for at /v1/images/generations. Implied by an external model whose API is ComfyUI's."
|
||||
},
|
||||
"meshes": {
|
||||
"type": "boolean",
|
||||
"description": "Meshes is a model that makes 3D models from a picture, asked for as a job at /v1/3d/generations. Implied by a ComfyUI model whose output is meshes."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8587,6 +8771,38 @@
|
||||
"checkpoint": {
|
||||
"type": "string",
|
||||
"description": "Checkpoint is the file the workflow's {{checkpoint}} names: the source's own name unless set, which a source that is a set of files must."
|
||||
},
|
||||
"output": {
|
||||
"type": "string",
|
||||
"description": "Output is what the workflow makes: pictures (the default), asked for at /v1/images/generations, or meshes, asked for as a job at /v1/3d/generations from a picture."
|
||||
},
|
||||
"nodes": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"$ref": "#/components/schemas/catalog.CustomNode"
|
||||
},
|
||||
"description": "Nodes are custom nodes the workflow needs, put in each replica's own custom_nodes at the commit named."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.CustomNode": {
|
||||
"type": "object",
|
||||
"description": "CustomNode is a ComfyUI node from a git repository.",
|
||||
"properties": {
|
||||
"repo": {
|
||||
"type": "string",
|
||||
"description": "Repo is the repository, https only."
|
||||
},
|
||||
"commit": {
|
||||
"type": "string",
|
||||
"description": "Commit is the full commit the node is used at: a branch moves, and a node that changed under a model is a model that changed unasked."
|
||||
},
|
||||
"requirements": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Requirements are the PyPI packages the node needs, by name and version range only. They are installed beside the replica, never into ComfyUI's environment: a package named here at the range named, and every other one at the version ComfyUI already runs. One named here is first on ComfyUI's path, so its range must suit ComfyUI too."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8635,6 +8851,10 @@
|
||||
"workflow": {
|
||||
"type": "string",
|
||||
"description": "Workflow is the ComfyUI workflow, in its API format: a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane."
|
||||
},
|
||||
"output": {
|
||||
"type": "string",
|
||||
"description": "Output is what it makes: pictures, or meshes (see ComfyUI.Output)."
|
||||
}
|
||||
}
|
||||
},
|
||||
@@ -8655,6 +8875,23 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.License": {
|
||||
"type": "object",
|
||||
"description": "License is the terms what a model makes comes under: said beside every result, and written into a mesh, which travels without the API.",
|
||||
"properties": {
|
||||
"name": {
|
||||
"type": "string",
|
||||
"description": "Name is the licence's own: MIT, Apache-2.0."
|
||||
},
|
||||
"url": {
|
||||
"type": "string"
|
||||
},
|
||||
"notice": {
|
||||
"type": "string",
|
||||
"description": "Notice is what must accompany what was made: attribution, terms on the output, what the weights were trained on."
|
||||
}
|
||||
}
|
||||
},
|
||||
"catalog.Model": {
|
||||
"type": "object",
|
||||
"description": "Model is a catalog entry as clients see it.",
|
||||
@@ -8701,6 +8938,10 @@
|
||||
"$ref": "#/components/schemas/catalog.ComfyUI",
|
||||
"description": "ComfyUI is how a model whose profiles run engine comfyui makes its pictures: the workflow, and the file it loads."
|
||||
},
|
||||
"license": {
|
||||
"$ref": "#/components/schemas/catalog.License",
|
||||
"description": "License is the terms of what the model makes, for a model whose output somebody ships: shown with every result and written into a mesh."
|
||||
},
|
||||
"adapters": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
@@ -8837,6 +9078,10 @@
|
||||
"gpu_class": {
|
||||
"type": "string"
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source is the weights this profile loads, when not the model's: an embedding model served by vLLM from its repository on a card and by llama.cpp from a GGUF on CPUs is one model with two sources. Empty loads the model's. Updates and canaries move the model's source; a profile with its own keeps it."
|
||||
},
|
||||
"tp": {
|
||||
"type": "integer"
|
||||
},
|
||||
@@ -10288,6 +10533,14 @@
|
||||
"CostMicros": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"Images": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"Meshes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
+2
-1
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
|
||||
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
|
||||
| 403 | The key may not use that model. |
|
||||
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
|
||||
|
||||
## Priority
|
||||
|
||||
|
||||
+137
@@ -1038,6 +1038,27 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
|
||||
No restart, and it survives one: models added this way are stored, while
|
||||
anything in `server.yaml` is declarative and returns on every start.
|
||||
|
||||
A profile may load weights of its own. An embedding model, for example, can
|
||||
be served by vLLM from its repository on a card and by llama.cpp from a GGUF
|
||||
on CPUs, as one model:
|
||||
|
||||
```yaml
|
||||
name: qwen3-embed-4b
|
||||
source: Qwen/Qwen3-Embedding-4B # what a profile loads unless it says
|
||||
capabilities: { embeddings: true }
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: vllm
|
||||
gpu_class: 6000-24g
|
||||
args: ["--runner", "pooling"]
|
||||
- name: cpu
|
||||
engine: llamacpp
|
||||
source: hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf
|
||||
```
|
||||
|
||||
Updates and canaries move the model's `source`; a profile with its own keeps
|
||||
it.
|
||||
|
||||
### Models that load when someone asks
|
||||
|
||||
A cluster with more models than GPUs can hold every model in its catalog and
|
||||
@@ -1672,6 +1693,7 @@ and accounting exactly as an API call does.
|
||||
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper |
|
||||
| `POST /v1/moderations` | answered by the guardrails' judge model (`role: guard`): `flagged`, and the judge's own categories |
|
||||
| `/v1/files`, `/v1/batches` | batches; see [Batches](#batches) |
|
||||
| `/v1/3d/generations` | 3D models, as jobs asked after; see [Jobs](#jobs-3d-models) |
|
||||
| `GET /v1/models` | the models this key may use |
|
||||
|
||||
## Watching it
|
||||
@@ -2030,6 +2052,121 @@ Each line of the file is `{"custom_id": …, "method": "POST", "url":
|
||||
or past their retention — a project under a guardrails policy with
|
||||
`no_store` is refused it.
|
||||
|
||||
## Jobs: 3D models
|
||||
|
||||
A 3D model from a picture takes minutes on a card that first has to be
|
||||
emptied of whatever else was on it. No client should hold a connection for
|
||||
that, so it is a job: queued, answered with an id, asked after, its result
|
||||
downloaded.
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/3d/generations -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-d '{"model": "trellis2", "image": "<base64>"}'
|
||||
# → 202 {"id": "job_…", "status": "queued", …}
|
||||
curl http://gateway:8080/v1/3d/generations/job_… -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
# → {"status": "completed", "output_file_id": "file-…", "content_type": "model/gltf-binary", "usage": {…}}
|
||||
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY" -o model.glb
|
||||
curl -X POST http://gateway:8080/v1/3d/generations/job_…/cancel -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
- **Status:** `queued`, `running`, `completed`, `failed` (with `error`),
|
||||
`cancelling`, `cancelled`, `expired` (not finished within 24 h).
|
||||
`progress` carries what the engine said of the work so far, once it says
|
||||
anything; `GET /v1/3d/generations` lists the caller's jobs.
|
||||
- **Refused at the door:** no key, a model that is not there or not in the
|
||||
key's allowlist, a request the model cannot serve. Refused when served,
|
||||
and said as the job's error: a spent budget, the guardrails, a model
|
||||
that will not come up.
|
||||
- **How it is served:** the request is kept as it was sent and replayed by
|
||||
whichever gateway holds the job's lease, through the same path a
|
||||
client's request takes — limits, admission in the key's band, guardrails,
|
||||
accounting. It is one usage record under the job's id, tagged
|
||||
`job=<id>`, charged like any request; `GET` shows the usage once it is
|
||||
done. A model still loading, a rate limit or a full queue is waited out
|
||||
for as long as the job has: nobody is holding a connection, which is the
|
||||
point.
|
||||
- **Surviving a restart:** a gateway that stops hands its jobs to another
|
||||
within two minutes; a job it was running is run again from the start.
|
||||
- **What it keeps:** the request as sent, and the result as a file owned
|
||||
by the key's project, kept `router.batch.retain_days` (30) like a batch's
|
||||
files; the job's record goes with it. A project under a `no_store`
|
||||
policy is refused jobs, as it is batches.
|
||||
|
||||
A request may also send `seed` and `resolution` (the model's own grid, a
|
||||
multiple of 32 from 64 to 2048). The picture should show one object on a
|
||||
plain background: the model makes a mesh of what it sees, background
|
||||
included.
|
||||
|
||||
### A model that makes them
|
||||
|
||||
A 3D model is made by a ComfyUI model whose `comfyui.output` is `meshes`
|
||||
(see [Pictures from ComfyUI](#pictures-from-comfyui) for engine `comfyui`).
|
||||
Its workflow is a JSON file in ComfyUI's API format: `{{image}}` is the
|
||||
uploaded picture, which a `LoadImage` node reads; `{{seed}}`,
|
||||
`{{resolution|256}}` and `{{checkpoint}}` are as for pictures. The node that
|
||||
saves the mesh may save a GLB or an OBJ; an OBJ leaves the gateway as a GLB,
|
||||
vertex colours kept. TripoSR (MIT) on any card of 8 GB:
|
||||
|
||||
```yaml
|
||||
- name: triposr
|
||||
source: hf:stabilityai/TripoSR/model.ckpt
|
||||
comfyui:
|
||||
workflow: /etc/llmgw/workflows/triposr.json # on every control plane
|
||||
output: meshes
|
||||
nodes:
|
||||
- repo: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR
|
||||
commit: a0c94ac60a7cc062604f61aeeea6d0d493521de3
|
||||
requirements: [omegaconf, trimesh, scikit-image, imageio,
|
||||
"transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers]
|
||||
license:
|
||||
name: MIT
|
||||
url: https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE
|
||||
notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). ..."
|
||||
scaling: {min: 0, max: 1, idle_timeout: 10m}
|
||||
profiles:
|
||||
- {name: gpu, engine: comfyui, gpu_class: 6000-24g, tp: 1}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}},
|
||||
"3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["1", 0],
|
||||
"geometry_resolution": "{{resolution|256}}", "threshold": 25.0}},
|
||||
"4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}}
|
||||
}
|
||||
```
|
||||
|
||||
- **Custom nodes** are named by https repository and full commit (a branch
|
||||
is refused: it moves). The gate puts each in the replica's own
|
||||
`custom_nodes` and installs its `requirements` beside the replica, never
|
||||
into ComfyUI's environment. A package named there is installed at the
|
||||
range named; every other one at the version ComfyUI already runs. A
|
||||
package named is first on ComfyUI's path, so its range must suit ComfyUI
|
||||
as well. TripoSR's checkpoint loads with transformers 4, which ComfyUI's
|
||||
`>=4.50.3` allows.
|
||||
- **Progress:** the gate listens to ComfyUI's websocket and the router
|
||||
writes what it hears into the job: `{"state": "queued", "ahead": 2}`,
|
||||
then `{"state": "running", "node_class": "KSampler", "step": 12,
|
||||
"steps": 30, "percent": 40}` for a node that counts its steps, and
|
||||
`done` at the end.
|
||||
- **Licence:** a model's `license` is shown in `/v1/models`, in the job and
|
||||
on the portal, and written into the GLB's `asset.extras.license`, since
|
||||
the file travels without the API. Choose the model for its licence before
|
||||
its quality when what it makes is shipped. TRELLIS.2 needs Meta's gated
|
||||
DINOv3 and, as published, a background remover under a non-commercial
|
||||
licence; Hunyuan3D's licence excludes the EU, the UK and South Korea.
|
||||
- **Price:** `per_mesh` in `auth.pricing`, beside `per_image`; each usage
|
||||
row says how many meshes it made (`meshes`).
|
||||
- **Cards:** the strongest models today (TRELLIS.2, Direct3D-S2) need
|
||||
flash-attention, which runs on Ampere and newer. A Turing card such as
|
||||
the RTX 6000 runs TripoSR.
|
||||
|
||||
People make them at `/3d` on the portal: they choose a picture and watch
|
||||
the job, and the mesh is shown in a viewer (model-viewer, served from the
|
||||
gateway) with a download link and the licence. `llmgw ctl 3d list`, `get`
|
||||
and `cancel` show and stop every owner's jobs.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
|
||||
@@ -146,6 +146,24 @@ Rules that apply:
|
||||
- Every document added or removed is in the audit log (`llmgw ctl audit`),
|
||||
by name. The text is never recorded.
|
||||
|
||||
### A large import
|
||||
|
||||
A few thousand documents ask the embedding model for far more than it can
|
||||
answer at once. Give the key that sends them a band of its own, below the
|
||||
people's, that may wait for room for minutes rather than the 30 seconds the
|
||||
others wait:
|
||||
|
||||
```sh
|
||||
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
|
||||
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
### Get it
|
||||
|
||||
@@ -61,8 +61,11 @@ rather than repeated.
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
| `images` | `bool` | [Capabilities](#capabilities) |
|
||||
| `meshes` | `bool` | [Capabilities](#capabilities) |
|
||||
| `workflow` | `string` | [ComfyUI](#comfyui) |
|
||||
| `checkpoint` | `string` | [ComfyUI](#comfyui) |
|
||||
| `output` | `string` | [ComfyUI](#comfyui) |
|
||||
| `nodes` | `[]CustomNode` | [ComfyUI](#comfyui) |
|
||||
| `dir` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `python` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `interval` | `time.Duration` | [Control](#control) |
|
||||
@@ -76,6 +79,9 @@ rather than repeated.
|
||||
| `eval_dir` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `repo` | `string` | [CustomNode](#customnode) |
|
||||
| `commit` | `string` | [CustomNode](#customnode) |
|
||||
| `requirements` | `[]string` | [CustomNode](#customnode) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
| `url` | `string` | [Endpoint](#endpoint) |
|
||||
| `api_key` | `string` | [Endpoint](#endpoint) |
|
||||
@@ -98,6 +104,7 @@ rather than repeated.
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `output` | `string` | [External](#external) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -126,6 +133,9 @@ rather than repeated.
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
| `docs_sign_in` | `bool` | [Landing](#landing) |
|
||||
| `name` | `string` | [License](#license) |
|
||||
| `url` | `string` | [License](#license) |
|
||||
| `notice` | `string` | [License](#license) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -144,6 +154,7 @@ rather than repeated.
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `comfyui` | `*ComfyUI` | [Model](#model) |
|
||||
| `license` | `*License` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
@@ -220,6 +231,7 @@ rather than repeated.
|
||||
| `engine` | `string` | [Profile](#profile) |
|
||||
| `engine_version` | `string` | [Profile](#profile) |
|
||||
| `gpu_class` | `string` | [Profile](#profile) |
|
||||
| `source` | `string` | [Profile](#profile) |
|
||||
| `tp` | `int` | [Profile](#profile) |
|
||||
| `pp` | `int` | [Profile](#profile) |
|
||||
| `gpus_per_node` | `int` | [Profile](#profile) |
|
||||
@@ -353,9 +365,11 @@ rather than repeated.
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
| `priority` | `int` | [admission.Band](#admissionband) |
|
||||
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
|
||||
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
|
||||
| `default_band` | `string` | [admission.Config](#admissionconfig) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
|
||||
@@ -385,9 +399,11 @@ rather than repeated.
|
||||
| `name` | `string` | [config.Band](#configband) |
|
||||
| `priority` | `int` | [config.Band](#configband) |
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
@@ -736,6 +752,14 @@ Images is a model that makes pictures, asked for at
|
||||
/v1/images/generations. Implied by an external model whose API is
|
||||
ComfyUI's.
|
||||
|
||||
### `meshes`
|
||||
|
||||
`bool`
|
||||
|
||||
Meshes is a model that makes 3D models from a picture, asked for as a
|
||||
job at /v1/3d/generations. Implied by a ComfyUI model whose output is
|
||||
meshes.
|
||||
|
||||
## ComfyUI
|
||||
|
||||
### `workflow`
|
||||
@@ -754,6 +778,21 @@ Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
source's own name unless set, which a source that is a set of files
|
||||
must.
|
||||
|
||||
### `output`
|
||||
|
||||
`string`
|
||||
|
||||
Output is what the workflow makes: pictures (the default), asked for
|
||||
at /v1/images/generations, or meshes, asked for as a job at
|
||||
/v1/3d/generations from a picture.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]CustomNode`
|
||||
|
||||
Nodes are custom nodes the workflow needs, put in each replica's own
|
||||
custom_nodes at the commit named.
|
||||
|
||||
## ComfyUIConfig
|
||||
|
||||
### `dir`
|
||||
@@ -854,6 +893,31 @@ see the claim. Long enough to survive a slow query or a brief network
|
||||
stall, short enough that a standby takes over before anyone has finished
|
||||
reading the alert. Zero takes the default.
|
||||
|
||||
## CustomNode
|
||||
|
||||
### `repo`
|
||||
|
||||
`string`
|
||||
|
||||
Repo is the repository, https only.
|
||||
|
||||
### `commit`
|
||||
|
||||
`string`
|
||||
|
||||
Commit is the full commit the node is used at: a branch moves, and a
|
||||
node that changed under a model is a model that changed unasked.
|
||||
|
||||
### `requirements`
|
||||
|
||||
`[]string`
|
||||
|
||||
Requirements are the PyPI packages the node needs, by name and version
|
||||
range only. They are installed beside the replica, never into
|
||||
ComfyUI's environment: a package named here at the range named, and
|
||||
every other one at the version ComfyUI already runs. One named here
|
||||
is first on ComfyUI's path, so its range must suit ComfyUI too.
|
||||
|
||||
## Endpoint
|
||||
|
||||
### `id`
|
||||
@@ -997,6 +1061,12 @@ Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
(flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
plane.
|
||||
|
||||
### `output`
|
||||
|
||||
`string`
|
||||
|
||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1227,6 +1297,25 @@ llmgw in general — no address, model or key of this installation —
|
||||
which the project's repository publishes anyway. On, a visitor is sent
|
||||
to sign in, and without accounts the portal needs an API key.
|
||||
|
||||
## License
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name is the licence's own: MIT, Apache-2.0.
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
### `notice`
|
||||
|
||||
`string`
|
||||
|
||||
Notice is what must accompany what was made: attribution, terms on
|
||||
the output, what the weights were trained on.
|
||||
|
||||
## LlamaCppConfig
|
||||
|
||||
### `bin`
|
||||
@@ -1392,6 +1481,16 @@ pictures: the workflow, and the file it loads.
|
||||
|
||||
Takes a [ComfyUI](#comfyui) block.
|
||||
|
||||
### `license`
|
||||
|
||||
`*License`
|
||||
|
||||
License is the terms of what the model makes, for a model whose
|
||||
output somebody ships: shown with every result and written into a
|
||||
mesh.
|
||||
|
||||
Takes a [License](#license) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
@@ -1923,6 +2022,16 @@ version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
|
||||
|
||||
`string`
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the weights this profile loads, when not the model's: an
|
||||
embedding model served by vLLM from its repository on a card and by
|
||||
llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
loads the model's. Updates and canaries move the model's source; a
|
||||
profile with its own keeps it.
|
||||
|
||||
### `tp`
|
||||
|
||||
`int`
|
||||
@@ -2994,6 +3103,13 @@ PerImage is what one picture costs, for a model that makes them: a
|
||||
picture's cost is the card's time, which its prompt's few tokens do
|
||||
not measure.
|
||||
|
||||
### `per_mesh`
|
||||
|
||||
`int64`
|
||||
|
||||
PerMesh is what one 3D model costs: minutes of a card, which no
|
||||
token count measures.
|
||||
|
||||
## admission.Band
|
||||
|
||||
### `name`
|
||||
@@ -3013,6 +3129,15 @@ Priority orders bands; higher is served first.
|
||||
SaturationCeiling is the pool saturation above which this band waits.
|
||||
1.0 means "never hold this band back".
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL, set, is how long this band's requests may wait instead of
|
||||
the controller's: a band for bulk work — a knowledge-base import, a
|
||||
backfill — whose requests had rather wait ten minutes behind people
|
||||
than be turned away after thirty seconds.
|
||||
|
||||
## admission.Config
|
||||
|
||||
### `bands`
|
||||
@@ -3226,6 +3351,14 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`float64`
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL is how long this band's requests may wait for room; the
|
||||
admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
had rather wait behind people than be turned away.
|
||||
|
||||
## config.Price
|
||||
|
||||
### `prompt_per_1k`
|
||||
@@ -3243,6 +3376,12 @@ Takes a [Tracing](#tracing) block.
|
||||
PerImage is the cost of one picture, for a model that makes them
|
||||
(/v1/images/generations), on top of its prompt's tokens.
|
||||
|
||||
### `per_mesh`
|
||||
|
||||
`int64`
|
||||
|
||||
PerMesh is the cost of one 3D model (/v1/3d/generations).
|
||||
|
||||
## engines.Config
|
||||
|
||||
### `disabled`
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,20 +39,20 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000, 0); got != 4000 {
|
||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3); got != 20+150000 {
|
||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
|
||||
@@ -22,6 +22,9 @@ type Price struct {
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
// PerMesh is what one 3D model costs: minutes of a card, which no
|
||||
// token count measures.
|
||||
PerMesh int64 `koanf:"per_mesh"`
|
||||
}
|
||||
|
||||
// Zero reports whether this model spends nothing.
|
||||
@@ -29,15 +32,17 @@ type Price struct {
|
||||
// It matters beyond arithmetic: a zero-cost model stays available to a key
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 }
|
||||
func (p Price) Zero() bool {
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
}
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens, images int64) int64 {
|
||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage
|
||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||
}
|
||||
|
||||
func divUp(n, d int64) int64 {
|
||||
|
||||
@@ -124,5 +124,14 @@ func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
if s.APIKey != "" {
|
||||
env["LLMGW_ENGINE_KEY"] = s.APIKey
|
||||
}
|
||||
// A model's custom nodes, which only the catalog sets — a profile's env
|
||||
// may not name the variable — and so taken before baseEnv drops it.
|
||||
if nodes := p.Env[comfyNodesEnv]; nodes != "" {
|
||||
env[comfyNodesEnv] = nodes
|
||||
}
|
||||
return []Command{{Name: "comfyui", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
|
||||
// comfyNodesEnv is the custom nodes the catalog gives a ComfyUI model, as
|
||||
// JSON; catalog.NodesEnv and comfy.NodesEnv by the same name.
|
||||
const comfyNodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
@@ -0,0 +1,139 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// An engine's output goes to a file of its own, not to a pipe of the
|
||||
// agent's. A pipe has one reader, and that reader is the agent that
|
||||
// started the engine: when that agent restarts, the engine it left running
|
||||
// — and adopts a moment later — writes to a pipe nobody reads, and its next
|
||||
// write past the pipe's buffer fails with EPIPE. A Python engine's progress
|
||||
// bar then raises, and takes the request it was reporting on with it. A
|
||||
// file has no reader to lose: the engine appends to it, every agent reads
|
||||
// it, and the output from before a restart is there for the one after.
|
||||
|
||||
// maxOutputBytes bounds one process's file: past it the file is emptied and
|
||||
// the process goes on appending to it. A progress bar rewriting itself for
|
||||
// an hour would otherwise fill the disk.
|
||||
const maxOutputBytes = 64 << 20
|
||||
|
||||
// outputPath is where a shard's process writes.
|
||||
func (s *Supervisor) outputPath(shardID, name string) string {
|
||||
if name != "" {
|
||||
return filepath.Join(s.cfg.WorkDir, "shard-"+shardID+"."+name+".log")
|
||||
}
|
||||
return filepath.Join(s.cfg.WorkDir, "shard-"+shardID+".log")
|
||||
}
|
||||
|
||||
// openOutput is the file a process is started with: new, for a process
|
||||
// that is new.
|
||||
func (s *Supervisor) openOutput(shardID, name string) (*os.File, error) {
|
||||
return os.OpenFile(s.outputPath(shardID, name), os.O_CREATE|os.O_WRONLY|os.O_APPEND|os.O_TRUNC, 0o600) //nolint:gosec // the agent's own directory
|
||||
}
|
||||
|
||||
// follow reads a process's file into the shard's log as it is written,
|
||||
// until the shard's output is stopped and the file has been read to its
|
||||
// end. A file an earlier agent began is read from the start, so the output
|
||||
// from before a restart is in the log too.
|
||||
func (s *Supervisor) follow(sh *Shard, path, name string) {
|
||||
defer sh.outDone.Done()
|
||||
f, err := os.Open(path) //nolint:gosec // the agent's own directory
|
||||
if err != nil {
|
||||
sh.logs.add("[" + name + "] output unavailable: " + err.Error())
|
||||
return
|
||||
}
|
||||
defer f.Close()
|
||||
var offset int64
|
||||
var rest []byte
|
||||
buf := make([]byte, 64<<10)
|
||||
read := func() {
|
||||
for {
|
||||
st, err := f.Stat()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if st.Size() < offset {
|
||||
// Emptied, by this agent or another: begin again.
|
||||
offset, rest = 0, nil
|
||||
}
|
||||
if st.Size() > maxOutputBytes {
|
||||
_ = os.Truncate(path, 0)
|
||||
offset, rest = 0, nil
|
||||
continue
|
||||
}
|
||||
n, err := f.ReadAt(buf, offset)
|
||||
if n > 0 {
|
||||
offset += int64(n)
|
||||
rest = append(rest, buf[:n]...)
|
||||
for {
|
||||
i := bytes.IndexByte(rest, '\n')
|
||||
if i < 0 {
|
||||
break
|
||||
}
|
||||
s.addLine(sh, name, rest[:i])
|
||||
rest = rest[i+1:]
|
||||
}
|
||||
// A progress bar is a line rewritten with \r and never ended:
|
||||
// its latest state is what is wanted, not a line a megabyte
|
||||
// long. Kept, as llama.cpp and tqdm write them.
|
||||
if len(rest) > 1<<20 {
|
||||
s.addLine(sh, name, rest)
|
||||
rest = nil
|
||||
}
|
||||
}
|
||||
if err != nil || n == 0 {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
t := time.NewTicker(200 * time.Millisecond)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-sh.outStop:
|
||||
read()
|
||||
if len(rest) > 0 {
|
||||
s.addLine(sh, name, rest)
|
||||
}
|
||||
return
|
||||
case <-t.C:
|
||||
read()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Supervisor) addLine(sh *Shard, name string, b []byte) {
|
||||
// The last state of a line rewritten with \r, as a terminal would show.
|
||||
if i := bytes.LastIndexByte(b, '\r'); i >= 0 {
|
||||
b = b[i+1:]
|
||||
}
|
||||
line := string(b)
|
||||
if name != "" {
|
||||
// Prefixed so the two processes of a Ray shard stay distinguishable
|
||||
// in a single log tail.
|
||||
line = "[" + name + "] " + line
|
||||
}
|
||||
sh.logs.add(line)
|
||||
s.log.Debug("engine", "shard", sh.ID, "line", line)
|
||||
}
|
||||
|
||||
// removeOutput drops a shard's files once nothing writes to them.
|
||||
func (s *Supervisor) removeOutput(shardID string) {
|
||||
matches, _ := filepath.Glob(filepath.Join(s.cfg.WorkDir, "shard-"+shardID+".*log"))
|
||||
for _, m := range matches {
|
||||
_ = os.Remove(m)
|
||||
}
|
||||
_ = os.Remove(s.outputPath(shardID, ""))
|
||||
}
|
||||
|
||||
// stopOutput has every follower read what is left and stop. Called once
|
||||
// the process has ended, before the shard is done, so what it wrote last
|
||||
// is in the log by the time anyone reads the log for why it ended.
|
||||
func (s *Supervisor) stopOutput(sh *Shard) {
|
||||
sh.outOnce.Do(func() { close(sh.outStop) })
|
||||
sh.outDone.Wait()
|
||||
}
|
||||
@@ -1,12 +1,10 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
@@ -104,6 +102,12 @@ type Shard struct {
|
||||
ID string
|
||||
Spec engines.ShardSpec
|
||||
|
||||
// outStop ends the followers of the processes' output files, and
|
||||
// outDone is their having read to the end.
|
||||
outStop chan struct{}
|
||||
outOnce sync.Once
|
||||
outDone sync.WaitGroup
|
||||
|
||||
mu sync.Mutex
|
||||
state State
|
||||
pid int
|
||||
@@ -248,6 +252,7 @@ func (s *Supervisor) Start(ctx context.Context, id string, spec engines.ShardSpe
|
||||
logs: newRingBuffer(s.cfg.LogLines),
|
||||
started: time.Now(),
|
||||
done: make(chan struct{}),
|
||||
outStop: make(chan struct{}),
|
||||
}
|
||||
s.shards[id] = sh
|
||||
s.mu.Unlock()
|
||||
@@ -315,14 +320,15 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
// CommandContext kills only the leader, so replace that with a group kill.
|
||||
c.Cancel = func() error { return killGroup(c.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := c.StdoutPipe()
|
||||
out, err := s.openOutput(sh.ID, "")
|
||||
if err != nil {
|
||||
cancel()
|
||||
return err
|
||||
}
|
||||
c.Stderr = c.Stdout.(io.Writer) // interleave, as the engine writes them
|
||||
|
||||
if err := c.Start(); err != nil {
|
||||
c.Stdout, c.Stderr = out, out // interleaved, as the engine writes them
|
||||
err = c.Start()
|
||||
out.Close() // the process holds its own
|
||||
if err != nil {
|
||||
cancel()
|
||||
return fmt.Errorf("starting %s: %w", cmd.Path, err)
|
||||
}
|
||||
@@ -339,7 +345,8 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
"shard", sh.ID, "err", err)
|
||||
}
|
||||
|
||||
go s.pump(sh, stdout, cmd.Name)
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, ""), cmd.Name)
|
||||
go s.wait(sh)
|
||||
return nil
|
||||
}
|
||||
@@ -353,16 +360,18 @@ func (s *Supervisor) spawn(ctx context.Context, sh *Shard, c engines.Command) (*
|
||||
s.dropTo(proc)
|
||||
proc.Cancel = func() error { return killGroup(proc.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := proc.StdoutPipe()
|
||||
out, err := s.openOutput(sh.ID, c.Name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
proc.Stderr = proc.Stdout.(io.Writer)
|
||||
|
||||
if err := proc.Start(); err != nil {
|
||||
proc.Stdout, proc.Stderr = out, out
|
||||
err = proc.Start()
|
||||
out.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("starting %s: %w", c.Path, err)
|
||||
}
|
||||
go s.pump(sh, stdout, c.Name)
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, c.Name), c.Name)
|
||||
// Reaped so it does not become a zombie; its exit does not end the shard.
|
||||
go func() { _ = proc.Wait() }()
|
||||
return proc, nil
|
||||
@@ -395,24 +404,6 @@ func waitForPort(ctx context.Context, addr string, timeout time.Duration) (err e
|
||||
}
|
||||
|
||||
// pump copies a process's output into the ring buffer and the log.
|
||||
func (s *Supervisor) pump(sh *Shard, r io.Reader, name string) {
|
||||
sc := bufio.NewScanner(r)
|
||||
// Engine tracebacks and progress bars produce long lines; the default
|
||||
// 64 KiB limit truncates them exactly when they matter.
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if name != "" {
|
||||
// Prefixed so the two processes of a Ray shard stay distinguishable
|
||||
// in a single log tail.
|
||||
line = "[" + name + "] " + line
|
||||
}
|
||||
sh.logs.add(line)
|
||||
s.log.Debug("engine", "shard", sh.ID, "line", line)
|
||||
}
|
||||
}
|
||||
|
||||
// wait reaps the process and records why it ended.
|
||||
func (s *Supervisor) wait(sh *Shard) {
|
||||
defer close(sh.done)
|
||||
@@ -440,6 +431,7 @@ func (s *Supervisor) wait(sh *Shard) {
|
||||
final, code := sh.state, sh.exitCode
|
||||
sh.mu.Unlock()
|
||||
|
||||
s.stopOutput(sh)
|
||||
s.removeState(sh)
|
||||
if final == Failed {
|
||||
s.log.Error("shard failed", "shard", sh.ID, "exit_code", code, "err", err,
|
||||
@@ -760,6 +752,7 @@ func (s *Supervisor) writeState(sh *Shard) error {
|
||||
|
||||
func (s *Supervisor) removeState(sh *Shard) {
|
||||
_ = os.Remove(s.statePath(sh.ID))
|
||||
s.removeOutput(sh.ID)
|
||||
}
|
||||
|
||||
// Orphan is an engine process left behind by a previous agent.
|
||||
@@ -838,15 +831,18 @@ func (s *Supervisor) Adopt(o Orphan) (*Shard, error) {
|
||||
logs: newRingBuffer(s.cfg.LogLines),
|
||||
started: o.StartedAt,
|
||||
done: make(chan struct{}),
|
||||
outStop: make(chan struct{}),
|
||||
adopted: true,
|
||||
cmdline: o.Cmdline,
|
||||
}
|
||||
s.shards[o.ShardID] = sh
|
||||
s.mu.Unlock()
|
||||
|
||||
// Output is gone with the previous agent; say so rather than presenting an
|
||||
// empty log as if the engine had been silent.
|
||||
sh.logs.add("[adopted from a previous agent; earlier output is not available]")
|
||||
// Its output file is read from the start: what it wrote under the
|
||||
// previous agent is there, and what it writes now goes on arriving.
|
||||
sh.logs.add("[adopted from a previous agent]")
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, ""), "")
|
||||
|
||||
go s.poll(sh)
|
||||
return sh, nil
|
||||
@@ -874,6 +870,7 @@ func (s *Supervisor) poll(sh *Shard) {
|
||||
}
|
||||
sh.mu.Unlock()
|
||||
|
||||
s.stopOutput(sh)
|
||||
s.removeState(sh)
|
||||
s.log.Info("adopted shard exited", "shard", sh.ID, "expected", expected)
|
||||
return
|
||||
@@ -900,6 +897,7 @@ func (s *Supervisor) KillOrphan(o Orphan) error {
|
||||
}
|
||||
// An already-missing state file is success: the previous agent may have
|
||||
// reaped the process itself between the scan and this call.
|
||||
s.removeOutput(o.ShardID)
|
||||
if err := os.Remove(s.statePath(o.ShardID)); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -905,3 +906,78 @@ func TestAnAdoptedShardsComputeIsKeptWithoutCrashing(t *testing.T) {
|
||||
t.Fatalf("lost: %+v", again)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdoptedEngineGoesOnWritingAndItsEarlierOutputIsKept(t *testing.T) {
|
||||
// An engine's output went to a pipe of the agent's. The agent that
|
||||
// restarted and adopted the engine left that pipe with no reader, and
|
||||
// the engine's next write past the pipe's buffer failed with EPIPE —
|
||||
// which, in a Python engine, raised inside whatever was being reported
|
||||
// on and failed the request. Output goes to a file now: a reader is not
|
||||
// something a file can lose, and the lines from before the restart are
|
||||
// there for the agent after it.
|
||||
s, _ := newSupervisor(t)
|
||||
// 256 KiB at once outgrows a pipe's buffer; a reader gone means EPIPE.
|
||||
sh, err := s.Start(t.Context(), "a", engines.ShardSpec{ReplicaID: "r"},
|
||||
shellCommand(`echo before-restart; sleep 1; head -c 262144 /dev/zero | tr '\0' x; echo; echo after-restart-done; sleep 30`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, 3*time.Second, "the first line was never captured", func() bool {
|
||||
return strings.Contains(strings.Join(sh.Logs(10), "\n"), "before-restart")
|
||||
})
|
||||
// The agent goes away without stopping the engine: nothing reads the
|
||||
// output between here and the adoption.
|
||||
pid := sh.Status().PID
|
||||
sh.outOnce.Do(func() { close(sh.outStop) })
|
||||
sh.outDone.Wait()
|
||||
|
||||
s2, err := New(Config{WorkDir: s.cfg.WorkDir, StopTimeout: 3 * time.Second, LogLines: 64},
|
||||
&scriptedHealth{}, slog.New(slog.DiscardHandler))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
orphans, _ := s2.FindOrphans()
|
||||
adopted, err := s2.Adopt(orphans[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, 5*time.Second, "the engine's output after the restart never arrived: "+strings.Join(adopted.Logs(5), " | "), func() bool {
|
||||
return strings.Contains(strings.Join(adopted.Logs(64), "\n"), "after-restart-done")
|
||||
})
|
||||
if !strings.Contains(strings.Join(adopted.Logs(64), "\n"), "before-restart") {
|
||||
t.Error("the output from before the restart is not in the adopted log")
|
||||
}
|
||||
if !processAlive(pid) {
|
||||
t.Fatal("the engine died writing output nobody was reading")
|
||||
}
|
||||
if err := s2.Stop(t.Context(), "a"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(s2.outputPath("a", "")); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("the output file was not removed with the shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
|
||||
s, _ := newSupervisor(t)
|
||||
path := s.outputPath("a", "")
|
||||
// A file already past the bound, as a long-running engine leaves one.
|
||||
big, _ := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
_, _ = big.Write(bytes.Repeat([]byte("x"), maxOutputBytes+1))
|
||||
big.Close()
|
||||
sh := &Shard{ID: "a", logs: newRingBuffer(8), done: make(chan struct{}), outStop: make(chan struct{})}
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, path, "")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
f, _ := os.OpenFile(path, os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
_, _ = f.WriteString("still here\n")
|
||||
f.Close()
|
||||
s.stopOutput(sh)
|
||||
st, _ := os.Stat(path)
|
||||
if st.Size() > 1024 {
|
||||
t.Errorf("the file was not emptied: %d bytes", st.Size())
|
||||
}
|
||||
if !strings.Contains(strings.Join(sh.Logs(8), "\n"), "still here") {
|
||||
t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
|
||||
}
|
||||
}
|
||||
@@ -178,6 +178,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newJobsCmd(client, &asJSON),
|
||||
newMeshJobsCmd(client, &asJSON),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(client),
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// newMeshJobsCmd is the 3D models clients asked for, as jobs: every owner's,
|
||||
// what each is doing, and a way to stop one. (ctl jobs is the control
|
||||
// plane's own work, serving and updating models.)
|
||||
func newMeshJobsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "3d",
|
||||
Short: "3D models clients asked for: the jobs, how far along each is, and stopping one",
|
||||
}
|
||||
var status string
|
||||
var limit int
|
||||
list := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every owner's 3D jobs, newest first",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
q := url.Values{}
|
||||
if status != "" {
|
||||
q.Set("status", status)
|
||||
}
|
||||
if limit > 0 {
|
||||
q.Set("limit", fmt.Sprint(limit))
|
||||
}
|
||||
var jobs []adminapi.JobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/client-jobs?"+q.Encode(), &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "no 3D jobs")
|
||||
return nil
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "JOB\tMODEL\tSTATUS\tPROGRESS\tOWNER\tASKED")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s ago\n", j.ID, j.Model, j.Status, progressText(j),
|
||||
j.Owner, time.Since(j.CreatedAt).Round(time.Second))
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
list.Flags().StringVar(&status, "status", "", "only jobs in this status: queued, running, completed, failed…")
|
||||
list.Flags().IntVar(&limit, "limit", 0, "the most jobs to show (50)")
|
||||
get := &cobra.Command{
|
||||
Use: "get JOB",
|
||||
Short: "One 3D job: its status, progress, result or error",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var j adminapi.JobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/client-jobs/"+url.PathEscape(args[0]), &j); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(j)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "job %s\nmodel %s\nstatus %s\n", j.ID, j.Model, j.Status)
|
||||
if p := progressText(j); p != "" {
|
||||
fmt.Fprintf(out, "progress %s\n", p)
|
||||
}
|
||||
fmt.Fprintf(out, "owner %s (key %s)\nasked %s\n", j.Owner, j.KeyID, j.CreatedAt.UTC().Format(time.RFC3339))
|
||||
if j.FinishedAt != nil && j.StartedAt != nil {
|
||||
fmt.Fprintf(out, "took %s\n", j.FinishedAt.Sub(*j.StartedAt).Round(time.Second))
|
||||
}
|
||||
if j.OutputFileID != "" {
|
||||
fmt.Fprintf(out, "result %s (%s)\n", j.OutputFileID, j.ContentType)
|
||||
}
|
||||
if j.Error != "" {
|
||||
fmt.Fprintf(out, "error %s\n", j.Error)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cancel := &cobra.Command{
|
||||
Use: "cancel JOB",
|
||||
Short: "Stop a 3D job, in the engine's queue too",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var j adminapi.JobView
|
||||
if err := client().post(cmd.Context(), "/api/v1/client-jobs/"+url.PathEscape(args[0])+"/cancel", nil, &j); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", j.ID, j.Status)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.AddCommand(list, get, cancel)
|
||||
return cmd
|
||||
}
|
||||
|
||||
// progressText is a job's progress as a person reads it.
|
||||
func progressText(j adminapi.JobView) string {
|
||||
if len(j.Progress) == 0 {
|
||||
return ""
|
||||
}
|
||||
var p struct {
|
||||
State string `json:"state"`
|
||||
Ahead int `json:"ahead"`
|
||||
NodeClass string `json:"node_class"`
|
||||
Step int `json:"step"`
|
||||
Steps int `json:"steps"`
|
||||
Percent int `json:"percent"`
|
||||
}
|
||||
if json.Unmarshal(j.Progress, &p) != nil {
|
||||
return ""
|
||||
}
|
||||
switch {
|
||||
case p.State == "queued" && p.Ahead > 0:
|
||||
return fmt.Sprintf("queued, %d ahead", p.Ahead)
|
||||
case p.State == "running" && p.Steps > 0:
|
||||
return fmt.Sprintf("running %d%%, %s step %d/%d", p.Percent, p.NodeClass, p.Step, p.Steps)
|
||||
case p.State == "running" && p.NodeClass != "":
|
||||
return "running " + p.NodeClass
|
||||
}
|
||||
return p.State
|
||||
}
|
||||
@@ -59,6 +59,9 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
if v.Images > 0 {
|
||||
fmt.Fprintf(out, "images %d\n", v.Images)
|
||||
}
|
||||
if v.Meshes > 0 {
|
||||
fmt.Fprintf(out, "meshes %d\n", v.Meshes)
|
||||
}
|
||||
if v.CacheHit {
|
||||
fmt.Fprintf(out, "cache answered from the project's answer cache: nothing generated, nothing charged\n")
|
||||
}
|
||||
|
||||
+11
-8
@@ -80,15 +80,18 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
// resolve is otherwise discovered by a replica failing to start
|
||||
// minutes later on a node — with an error naming the node rather
|
||||
// than the typo.
|
||||
// Every source the model's profiles load, not the model's alone.
|
||||
if !skipCheck {
|
||||
size, err := sourcecheck.Check(cmd.Context(), m.Source)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(m.Source); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(m.Source, size))
|
||||
for _, src := range m.WeightSources() {
|
||||
size, err := sourcecheck.Check(cmd.Context(), src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(src); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(src, size))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+14
-10
@@ -210,6 +210,10 @@ func newServerCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if st := rt.Store(); st != nil {
|
||||
// Each control plane raises the same events; one sends each.
|
||||
notifier.SetClaimer(st)
|
||||
}
|
||||
rt.SetNotifier(notifier)
|
||||
if gate != nil && notifier != nil {
|
||||
gate.SetJudgeAlarm(func(model string, err error) {
|
||||
@@ -1169,18 +1173,18 @@ func servableCheck(llamaVersion string) func(context.Context, catalog.Model) err
|
||||
if llamaVersion == "" {
|
||||
return nil
|
||||
}
|
||||
usesLlama := false
|
||||
// Each source a llama.cpp profile loads: its own, or the model's.
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == "llamacpp" {
|
||||
usesLlama = true
|
||||
if p.Engine != "llamacpp" {
|
||||
continue
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.SourceOf(p))
|
||||
if !ok || !gguf.IsGGUF(url) {
|
||||
continue
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.Source)
|
||||
if !usesLlama || !ok || !gguf.IsGGUF(url) {
|
||||
return nil
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
+207
-19
@@ -21,8 +21,10 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"math/big"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
@@ -98,6 +100,11 @@ type Params struct {
|
||||
Seed int64
|
||||
Steps int
|
||||
Guidance float64
|
||||
// Image is the input picture, by the name Upload gave it in ComfyUI's
|
||||
// input directory: {{image}}.
|
||||
Image string
|
||||
// Extra are a workflow's own placeholders, by name: {{resolution}}.
|
||||
Extra map[string]any
|
||||
}
|
||||
|
||||
func (p Params) value(name string) (any, bool) {
|
||||
@@ -120,8 +127,11 @@ func (p Params) value(name string) (any, bool) {
|
||||
return p.Steps, p.Steps > 0
|
||||
case "guidance":
|
||||
return p.Guidance, p.Guidance > 0
|
||||
case "image":
|
||||
return p.Image, p.Image != ""
|
||||
}
|
||||
return nil, false
|
||||
v, ok := p.Extra[name]
|
||||
return v, ok
|
||||
}
|
||||
|
||||
// Fill is the workflow for one request.
|
||||
@@ -207,10 +217,12 @@ func RandomSeed() int64 {
|
||||
return n.Int64()
|
||||
}
|
||||
|
||||
// Image is one picture made.
|
||||
// Image is one file a workflow made: a picture, a mesh.
|
||||
type Image struct {
|
||||
Data []byte
|
||||
ContentType string
|
||||
// Name is the file's own, as ComfyUI saved it.
|
||||
Name string
|
||||
}
|
||||
|
||||
// Error is a failure, said as what it means for the request.
|
||||
@@ -235,6 +247,17 @@ type Client struct {
|
||||
// workflow out of the queue, or interrupts it if it is running, so a client
|
||||
// that went away does not leave the card working for nobody.
|
||||
func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map[string]any) ([]Image, error) {
|
||||
return c.GenerateWatch(ctx, base, apiKey, workflow, nil)
|
||||
}
|
||||
|
||||
// progressEvery is how often a workflow's progress is asked after, beside
|
||||
// its history: a few requests a second is for the end, not for the news.
|
||||
const progressEvery = time.Second
|
||||
|
||||
// GenerateWatch is Generate, saying how far along the workflow is as it
|
||||
// changes: queued and how many ahead, then the node running and its step.
|
||||
func (c *Client) GenerateWatch(ctx context.Context, base, apiKey string, workflow map[string]any,
|
||||
onProgress func(Progress)) ([]Image, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
client := make([]byte, 8)
|
||||
_, _ = rand.Read(client)
|
||||
@@ -265,6 +288,8 @@ func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map
|
||||
}
|
||||
t := time.NewTicker(poll)
|
||||
defer t.Stop()
|
||||
var last Progress
|
||||
var asked time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
@@ -272,6 +297,13 @@ func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map
|
||||
return nil, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
if onProgress != nil && time.Since(asked) >= progressEvery {
|
||||
asked = time.Now()
|
||||
if p, ok := c.progress(ctx, base, apiKey, id, workflow); ok && p != last {
|
||||
last = p
|
||||
onProgress(p)
|
||||
}
|
||||
}
|
||||
var hist map[string]historyEntry
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/history/"+url.PathEscape(id), apiKey, nil, &hist); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
@@ -299,13 +331,16 @@ type historyEntry struct {
|
||||
Completed bool `json:"completed"`
|
||||
Messages [][]json.RawMessage `json:"messages"`
|
||||
} `json:"status"`
|
||||
Outputs map[string]struct {
|
||||
Images []struct {
|
||||
Filename string `json:"filename"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
Type string `json:"type"`
|
||||
} `json:"images"`
|
||||
} `json:"outputs"`
|
||||
// Outputs are each node's, by kind: "images" for a picture, "3d" for
|
||||
// the mesh a 3D node saves, and whatever else a custom node names.
|
||||
Outputs map[string]map[string]json.RawMessage `json:"outputs"`
|
||||
}
|
||||
|
||||
// fileRef is a file an output names, to be read from /view.
|
||||
type fileRef struct {
|
||||
Filename string `json:"filename"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
Type string `json:"type"`
|
||||
}
|
||||
|
||||
// failure is what ComfyUI said went wrong, from its execution_error.
|
||||
@@ -341,17 +376,38 @@ func (c *Client) fetch(ctx context.Context, base, apiKey string, h historyEntry)
|
||||
})
|
||||
var out []Image
|
||||
for _, n := range nodes {
|
||||
for _, im := range h.Outputs[n].Images {
|
||||
q := url.Values{"filename": {im.Filename}, "subfolder": {im.Subfolder}, "type": {im.Type}}
|
||||
img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
kinds := make([]string, 0, len(h.Outputs[n]))
|
||||
for k := range h.Outputs[n] {
|
||||
kinds = append(kinds, k)
|
||||
}
|
||||
slices.Sort(kinds)
|
||||
for _, k := range kinds {
|
||||
// Any list of files, whatever the node calls it; text and
|
||||
// numbers a node reports beside them are not files.
|
||||
var refs []fileRef
|
||||
if json.Unmarshal(h.Outputs[n][k], &refs) != nil {
|
||||
continue
|
||||
}
|
||||
for _, f := range refs {
|
||||
if f.Filename == "" {
|
||||
continue
|
||||
}
|
||||
q := url.Values{"filename": {f.Filename}, "subfolder": {f.Subfolder}, "type": {f.Type}}
|
||||
img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
img.Name = f.Filename
|
||||
if t := typeByName(f.Filename); t != "" && (img.ContentType == "" ||
|
||||
strings.HasPrefix(img.ContentType, "application/octet-stream") || strings.HasPrefix(img.ContentType, "text/plain")) {
|
||||
img.ContentType = t
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, &Error{Msg: "the workflow finished without a picture: it has no SaveImage or PreviewImage node"}
|
||||
return nil, &Error{Msg: "the workflow finished without a file: it has no node that saves one"}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -434,12 +490,12 @@ func (c *Client) get(ctx context.Context, u, apiKey string) (Image, error) {
|
||||
return Image{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20))
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, maxFile))
|
||||
if err != nil {
|
||||
return Image{}, &Error{Msg: "reading a picture from ComfyUI: " + err.Error(), Retryable: true}
|
||||
return Image{}, &Error{Msg: "reading a file from ComfyUI: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a picture it made", resp.StatusCode)}
|
||||
return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a file it made", resp.StatusCode)}
|
||||
}
|
||||
ct := resp.Header.Get("Content-Type")
|
||||
if ct == "" || ct == "application/octet-stream" {
|
||||
@@ -527,3 +583,135 @@ func snippet(b []byte) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// maxFile bounds one file read from ComfyUI: a mesh with its textures runs
|
||||
// to tens of megabytes, a picture to a few.
|
||||
const maxFile = 256 << 20
|
||||
|
||||
// typeByName is a file's type by its extension, for those a server sends
|
||||
// as octet-stream: the meshes a 3D node saves.
|
||||
func typeByName(name string) string {
|
||||
switch strings.ToLower(path.Ext(name)) {
|
||||
case ".glb":
|
||||
return "model/gltf-binary"
|
||||
case ".gltf":
|
||||
return "model/gltf+json"
|
||||
case ".obj":
|
||||
return "model/obj"
|
||||
case ".stl":
|
||||
return "model/stl"
|
||||
case ".ply":
|
||||
return "application/ply"
|
||||
case ".png":
|
||||
return "image/png"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Upload puts a picture in ComfyUI's input directory, where a LoadImage node
|
||||
// reads it, and returns the name it was given there.
|
||||
func (c *Client) Upload(ctx context.Context, base, apiKey, name string, data []byte) (string, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
var body bytes.Buffer
|
||||
mw := multipart.NewWriter(&body)
|
||||
fw, err := mw.CreateFormFile("image", name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := fw.Write(data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_ = mw.WriteField("type", "input")
|
||||
_ = mw.WriteField("overwrite", "true")
|
||||
if err := mw.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/upload/image", &body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return "", ctx.Err()
|
||||
}
|
||||
return "", &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode >= 500 {
|
||||
return "", &Error{Msg: fmt.Sprintf("ComfyUI answered %d to the picture: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
var out struct {
|
||||
Name string `json:"name"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK || json.Unmarshal(b, &out) != nil || out.Name == "" {
|
||||
return "", &Error{Msg: fmt.Sprintf("ComfyUI did not take the picture (%d): %s", resp.StatusCode, snippet(b))}
|
||||
}
|
||||
if out.Subfolder != "" {
|
||||
return out.Subfolder + "/" + out.Name, nil
|
||||
}
|
||||
return out.Name, nil
|
||||
}
|
||||
|
||||
// progress is a workflow's: its place in the queue, from ComfyUI, and the
|
||||
// node running and its step, from the gate when there is one — a ComfyUI
|
||||
// outside the cluster has no gate, and says only where it is queued.
|
||||
func (c *Client) progress(ctx context.Context, base, apiKey, id string, workflow map[string]any) (Progress, bool) {
|
||||
p := Progress{PromptID: id, State: "queued"}
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
Pending [][]json.RawMessage `json:"queue_pending"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return p, false
|
||||
}
|
||||
quoted := strconv.Quote(id)
|
||||
queued := false
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == quoted {
|
||||
p.State = "running"
|
||||
}
|
||||
}
|
||||
if p.State != "running" {
|
||||
// Pending in the order ComfyUI will take them: by number.
|
||||
mine := -1.0
|
||||
var numbers []float64
|
||||
for _, r := range q.Pending {
|
||||
if len(r) < 2 {
|
||||
continue
|
||||
}
|
||||
var n float64
|
||||
_ = json.Unmarshal(r[0], &n)
|
||||
if string(r[1]) == quoted {
|
||||
mine, queued = n, true
|
||||
continue
|
||||
}
|
||||
numbers = append(numbers, n)
|
||||
}
|
||||
if queued {
|
||||
p.Ahead = len(q.Running)
|
||||
for _, n := range numbers {
|
||||
if n < mine {
|
||||
p.Ahead++
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
var g Progress
|
||||
if st, err := c.do(ctx, http.MethodGet, base+"/progress/"+url.PathEscape(id), apiKey, nil, &g); err == nil &&
|
||||
st == http.StatusOK && g.State != "" {
|
||||
p = g
|
||||
p.PromptID = id
|
||||
if node, ok := workflow[p.Node].(map[string]any); ok {
|
||||
p.NodeClass, _ = node["class_type"].(string)
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
+53
-5
@@ -2,7 +2,10 @@ package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -63,7 +66,10 @@ type GateConfig struct {
|
||||
|
||||
// The requests the router makes of ComfyUI: queueing, waiting, fetching and
|
||||
// cancelling. Nothing that uploads, installs or changes ComfyUI.
|
||||
var gatePaths = []string{"/prompt", "/history/", "/view", "/queue", "/interrupt", "/system_stats", "/object_info"}
|
||||
var gatePaths = []string{"/prompt", "/history/", "/view", "/queue", "/interrupt", "/system_stats", "/object_info",
|
||||
// The input picture of a 3D model: into ComfyUI's input directory,
|
||||
// which is the replica's own.
|
||||
"/upload/image"}
|
||||
|
||||
// modelFolders are the kinds of file ComfyUI's loaders look for. Each is
|
||||
// shown the same directory: a workflow names its files, so a file being
|
||||
@@ -89,6 +95,18 @@ func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
out := cfg.Output
|
||||
if out == nil {
|
||||
out = os.Stdout
|
||||
}
|
||||
nodes, err := parseNodes(os.Getenv(NodesEnv))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
site, err := installNodes(ctx, state, cfg.Python, nodes, findUV(), func(s string) { fmt.Fprintln(out, "[gate] "+s) })
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
models := filepath.Join(state, "models")
|
||||
if err := LinkWeights(cfg.Weights, models); err != nil {
|
||||
return err
|
||||
@@ -109,9 +127,12 @@ func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
child := exec.Command(cfg.Python, args...) //nolint:gosec // the agent's own configuration
|
||||
child.Dir = cfg.Dir
|
||||
child.Env = os.Environ()
|
||||
out := cfg.Output
|
||||
if out == nil {
|
||||
out = os.Stdout
|
||||
if site != "" {
|
||||
pp := site
|
||||
if old := os.Getenv("PYTHONPATH"); old != "" {
|
||||
pp += string(os.PathListSeparator) + old
|
||||
}
|
||||
child.Env = append(child.Env, "PYTHONPATH="+pp)
|
||||
}
|
||||
// ComfyUI writes to a pipe of the gate's, never to what the gate was
|
||||
// given. That is the agent's, and an agent that restarts and adopts the
|
||||
@@ -138,9 +159,16 @@ func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
}()
|
||||
|
||||
upstream := &url.URL{Scheme: "http", Host: net.JoinHostPort("127.0.0.1", strconv.Itoa(inner))}
|
||||
// The workflows' progress, heard on ComfyUI's websocket under an id of
|
||||
// the gate's own.
|
||||
book := newProgressBook()
|
||||
cid := make([]byte, 8)
|
||||
_, _ = rand.Read(cid)
|
||||
clientID := "llmgw-gate-" + hex.EncodeToString(cid)
|
||||
go book.watch(ctx, upstream, clientID, log)
|
||||
srv := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: NewGate(upstream, cfg.Key, cfg.Model),
|
||||
Handler: newGate(upstream, cfg.Key, cfg.Model, book, clientID),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
served := make(chan error, 1)
|
||||
@@ -181,6 +209,12 @@ func shutdown(srv *http.Server) {
|
||||
|
||||
// NewGate is the gate's handler, in front of the ComfyUI at upstream.
|
||||
func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
return newGate(upstream, key, model, newProgressBook(), "")
|
||||
}
|
||||
|
||||
// newGate is the gate with the book of its workflows' progress, kept under
|
||||
// clientID; an empty one leaves the workflows' own client ids alone.
|
||||
func newGate(upstream *url.URL, key, model string, book *progressBook, clientID string) http.Handler {
|
||||
proxy := httputil.NewSingleHostReverseProxy(upstream)
|
||||
base := proxy.Director
|
||||
proxy.Director = func(r *http.Request) {
|
||||
@@ -229,6 +263,20 @@ func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
fmt.Fprintf(w, `{"object":"list","data":[{"id":%q,"object":"model","owned_by":"comfyui"}]}`, model)
|
||||
return
|
||||
}
|
||||
if id, ok := strings.CutPrefix(r.URL.Path, "/progress/"); ok && r.Method == http.MethodGet {
|
||||
p, found := book.get(id)
|
||||
if !found {
|
||||
http.Error(w, "no word of that workflow", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(p)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/prompt" && r.Method == http.MethodPost && clientID != "" {
|
||||
book.promptThroughGate(proxy, clientID)(w, r)
|
||||
return
|
||||
}
|
||||
for _, p := range gatePaths {
|
||||
if r.URL.Path == p || strings.HasSuffix(p, "/") && strings.HasPrefix(r.URL.Path, p) {
|
||||
proxy.ServeHTTP(w, r)
|
||||
|
||||
@@ -45,12 +45,12 @@ func TestTheGateWantsTheKeyAndLetsThroughOnlyTheRoutersRequests(t *testing.T) {
|
||||
if code, _ := do("POST", "/prompt", "wrong"); code != http.StatusUnauthorized {
|
||||
t.Errorf("wrong key: %d", code)
|
||||
}
|
||||
for _, p := range []string{"/upload/image", "/api/manager/queue/install", "/userdata/x", "/internal/logs"} {
|
||||
for _, p := range []string{"/upload/mask", "/api/manager/queue/install", "/userdata/x", "/internal/logs"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusNotFound {
|
||||
t.Errorf("%s: %d, want it refused", p, code)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"/prompt", "/history/p1", "/view?filename=a.png", "/queue", "/interrupt"} {
|
||||
for _, p := range []string{"/prompt", "/history/p1", "/view?filename=a.png", "/queue", "/interrupt", "/upload/image"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusOK {
|
||||
t.Errorf("%s: %d", p, code)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Custom nodes, put in a replica's own base directory at the commit the
|
||||
// catalog names, their packages installed beside the replica rather than
|
||||
// into ComfyUI's environment: one model's node cannot break another
|
||||
// model's replicas, or the pictures every replica makes.
|
||||
|
||||
// Node is a custom node as the catalog gives it.
|
||||
type Node struct {
|
||||
Repo string `json:"repo"`
|
||||
Commit string `json:"commit"`
|
||||
Requirements []string `json:"requirements,omitempty"`
|
||||
}
|
||||
|
||||
// NodesEnv is where the agent hands the gate a model's custom nodes.
|
||||
const NodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
|
||||
var nodeDir = regexp.MustCompile(`[^A-Za-z0-9._-]`)
|
||||
|
||||
// installNodes puts each node in state/custom_nodes at its commit and its
|
||||
// packages in state/site, and returns the PYTHONPATH ComfyUI is started
|
||||
// with: empty when no node wants a package. Run as the engines' user, in
|
||||
// the replica's own directory, before ComfyUI starts.
|
||||
func installNodes(ctx context.Context, state, python string, nodes []Node, uv string, out func(string)) (string, error) {
|
||||
if len(nodes) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
var reqs []string
|
||||
for _, n := range nodes {
|
||||
name := nodeDir.ReplaceAllString(strings.TrimSuffix(path.Base(n.Repo), ".git"), "_")
|
||||
dir := filepath.Join(state, "custom_nodes", name)
|
||||
if head, err := gitOutput(ctx, dir, "rev-parse", "HEAD"); err == nil && head == n.Commit {
|
||||
reqs = append(reqs, n.Requirements...)
|
||||
continue
|
||||
}
|
||||
out(fmt.Sprintf("custom node %s at %s", n.Repo, n.Commit[:12]))
|
||||
_ = os.RemoveAll(dir)
|
||||
for _, args := range [][]string{
|
||||
{"init", "-q", dir},
|
||||
{"-C", dir, "remote", "add", "origin", n.Repo},
|
||||
{"-C", dir, "fetch", "-q", "--depth", "1", "origin", n.Commit},
|
||||
{"-C", dir, "checkout", "-q", "FETCH_HEAD"},
|
||||
} {
|
||||
if b, err := exec.CommandContext(ctx, "git", args...).CombinedOutput(); err != nil { //nolint:gosec // the catalog's repository, checked as https
|
||||
return "", fmt.Errorf("custom node %s at %s: git %s: %v: %s", n.Repo, n.Commit, args[len(args)-2], err, lastLines(b))
|
||||
}
|
||||
}
|
||||
reqs = append(reqs, n.Requirements...)
|
||||
}
|
||||
if len(reqs) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
if uv == "" {
|
||||
return "", fmt.Errorf("custom nodes need %s, and no uv is installed to install them with", strings.Join(reqs, ", "))
|
||||
}
|
||||
site := filepath.Join(state, "site")
|
||||
marker := filepath.Join(site, ".llmgw-requirements")
|
||||
want := strings.Join(reqs, "\n")
|
||||
if b, err := os.ReadFile(marker); err == nil && string(b) == want {
|
||||
return site, nil
|
||||
}
|
||||
out("installing " + strings.Join(reqs, " ") + " for the custom nodes")
|
||||
// What ComfyUI's environment has, as constraints: a package both want is
|
||||
// installed at the version ComfyUI runs, so the replica's copy and
|
||||
// ComfyUI's never disagree.
|
||||
freeze, err := exec.CommandContext(ctx, uv, "pip", "freeze", "--python", python).Output() //nolint:gosec // the agent's own uv
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading ComfyUI's packages: %w", err)
|
||||
}
|
||||
constraints := filepath.Join(state, "constraints.txt")
|
||||
if err := os.WriteFile(constraints, withoutNamed(freeze, reqs), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
_ = os.RemoveAll(site)
|
||||
args := append([]string{"pip", "install", "--python", python, "--target", site, "-c", constraints, "--quiet"}, reqs...)
|
||||
cmd := exec.CommandContext(ctx, uv, args...) //nolint:gosec // requirements checked by the catalog as package specs
|
||||
cmd.Env = append(os.Environ(), "UV_NO_CONFIG=1", "UV_CACHE_DIR="+filepath.Join(state, ".uv-cache"))
|
||||
if b, err := cmd.CombinedOutput(); err != nil {
|
||||
return "", fmt.Errorf("installing the custom nodes' packages (%s): %v: %s", strings.Join(reqs, " "), err, lastLines(b))
|
||||
}
|
||||
// The replica's copies come first on ComfyUI's path, but of anything
|
||||
// both have they are the same version, so which is found is the same.
|
||||
if err := os.WriteFile(marker, []byte(want), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
return site, nil
|
||||
}
|
||||
|
||||
// parseNodes reads the nodes the agent handed over.
|
||||
func parseNodes(s string) ([]Node, error) {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
var nodes []Node
|
||||
if err := json.Unmarshal([]byte(s), &nodes); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", NodesEnv, err)
|
||||
}
|
||||
return nodes, nil
|
||||
}
|
||||
|
||||
// findUV is the uv the agent installed engines with.
|
||||
func findUV() string {
|
||||
if m, _ := filepath.Glob("/opt/llmgw/engines/uv/*/*/uv"); len(m) > 0 {
|
||||
return m[len(m)-1]
|
||||
}
|
||||
if p, err := exec.LookPath("uv"); err == nil {
|
||||
return p
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func gitOutput(ctx context.Context, dir string, args ...string) (string, error) {
|
||||
b, err := exec.CommandContext(ctx, "git", append([]string{"-C", dir}, args...)...).Output() //nolint:gosec // fixed arguments
|
||||
return strings.TrimSpace(string(b)), err
|
||||
}
|
||||
|
||||
func lastLines(b []byte) string {
|
||||
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
|
||||
if len(lines) > 4 {
|
||||
lines = lines[len(lines)-4:]
|
||||
}
|
||||
return strings.Join(lines, " | ")
|
||||
}
|
||||
|
||||
// withoutNamed is ComfyUI's packages less those a node names itself: of a
|
||||
// package a node asks for by name, the node's range decides — TripoSR's
|
||||
// checkpoint loads with transformers 4 and not 5 — and of everything else,
|
||||
// what ComfyUI runs. A node that names a package must name a range ComfyUI
|
||||
// can run with too: the copy it gets is first on ComfyUI's path.
|
||||
func withoutNamed(freeze []byte, reqs []string) []byte {
|
||||
named := map[string]bool{}
|
||||
for _, r := range reqs {
|
||||
named[packageName(r)] = true
|
||||
}
|
||||
var out []string
|
||||
for _, line := range strings.Split(string(freeze), "\n") {
|
||||
if strings.TrimSpace(line) == "" || named[packageName(line)] {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return []byte(strings.Join(out, "\n") + "\n")
|
||||
}
|
||||
|
||||
// packageName is a requirement's package, as PyPI compares them.
|
||||
func packageName(spec string) string {
|
||||
s := strings.TrimSpace(spec)
|
||||
if i := strings.IndexAny(s, "<>=!~;[ @"); i >= 0 {
|
||||
s = s[:i]
|
||||
}
|
||||
return strings.NewReplacer("_", "-", ".", "-").Replace(strings.ToLower(s))
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/net/websocket"
|
||||
)
|
||||
|
||||
// How far along a workflow is, which ComfyUI says only on its websocket and
|
||||
// only to the client that queued the workflow. The gate keeps one websocket
|
||||
// open to its ComfyUI, under a client id of its own, puts that id on every
|
||||
// workflow it lets through, and keeps the last word on each: the node
|
||||
// running and its step out of how many. The router asks at
|
||||
// /progress/{prompt_id} while it waits, and a job says it.
|
||||
|
||||
// Progress is a workflow's state as the gate last heard it.
|
||||
type Progress struct {
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
// State is queued, running, done or failed.
|
||||
State string `json:"state"`
|
||||
// Ahead is how many workflows wait before it, while it is queued.
|
||||
Ahead int `json:"ahead,omitempty"`
|
||||
// Node is the workflow's node running, by its id; NodeClass its kind.
|
||||
Node string `json:"node,omitempty"`
|
||||
NodeClass string `json:"node_class,omitempty"`
|
||||
// Step of Steps is the running node's own count: a sampler's steps.
|
||||
Step int `json:"step,omitempty"`
|
||||
Steps int `json:"steps,omitempty"`
|
||||
// Percent is Step of Steps, where a node counts.
|
||||
Percent int `json:"percent,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// progressBook is the last word on each of the most recent workflows.
|
||||
type progressBook struct {
|
||||
mu sync.Mutex
|
||||
by map[string]*Progress
|
||||
order []string
|
||||
}
|
||||
|
||||
// keptPrompts bounds the book: a gate serves one replica, which runs one
|
||||
// workflow at a time and queues a few.
|
||||
const keptPrompts = 256
|
||||
|
||||
func newProgressBook() *progressBook { return &progressBook{by: map[string]*Progress{}} }
|
||||
|
||||
func (b *progressBook) update(id string, f func(*Progress)) {
|
||||
if id == "" {
|
||||
return
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
p, ok := b.by[id]
|
||||
if !ok {
|
||||
p = &Progress{PromptID: id, State: "queued"}
|
||||
b.by[id] = p
|
||||
b.order = append(b.order, id)
|
||||
for len(b.order) > keptPrompts {
|
||||
delete(b.by, b.order[0])
|
||||
b.order = b.order[1:]
|
||||
}
|
||||
}
|
||||
f(p)
|
||||
}
|
||||
|
||||
func (b *progressBook) get(id string) (Progress, bool) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
p, ok := b.by[id]
|
||||
if !ok {
|
||||
return Progress{}, false
|
||||
}
|
||||
return *p, true
|
||||
}
|
||||
|
||||
// read takes one message off ComfyUI's websocket into the book.
|
||||
func (b *progressBook) read(msg []byte) {
|
||||
var m struct {
|
||||
Type string `json:"type"`
|
||||
Data struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Node json.RawMessage `json:"node"`
|
||||
Value int `json:"value"`
|
||||
Max int `json:"max"`
|
||||
Message string `json:"exception_message"`
|
||||
NodeType string `json:"node_type"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if json.Unmarshal(msg, &m) != nil {
|
||||
return
|
||||
}
|
||||
d := m.Data
|
||||
var node string
|
||||
_ = json.Unmarshal(d.Node, &node)
|
||||
switch m.Type {
|
||||
case "execution_start":
|
||||
b.update(d.PromptID, func(p *Progress) { p.State, p.Ahead = "running", 0 })
|
||||
case "executing":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
if node == "" {
|
||||
// No node: the workflow is through.
|
||||
p.State, p.Node, p.Step, p.Steps, p.Percent = "done", "", 0, 0, 100
|
||||
return
|
||||
}
|
||||
p.State, p.Ahead, p.Node, p.Step, p.Steps, p.Percent = "running", 0, node, 0, 0, 0
|
||||
})
|
||||
case "progress":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
p.State, p.Step, p.Steps = "running", d.Value, d.Max
|
||||
if node != "" {
|
||||
p.Node = node
|
||||
}
|
||||
if d.Max > 0 {
|
||||
p.Percent = min(100, d.Value*100/d.Max)
|
||||
}
|
||||
})
|
||||
case "execution_success":
|
||||
b.update(d.PromptID, func(p *Progress) { p.State, p.Percent = "done", 100 })
|
||||
case "execution_error", "execution_interrupted":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
p.State = "failed"
|
||||
p.Error = strings.TrimSpace(d.NodeType + ": " + d.Message)
|
||||
if m.Type == "execution_interrupted" {
|
||||
p.Error = "interrupted"
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// watch keeps a websocket open to the ComfyUI at upstream under clientID,
|
||||
// reading into the book, until ctx ends: opened again when it closes, as it
|
||||
// does whenever ComfyUI restarts.
|
||||
func (b *progressBook) watch(ctx context.Context, upstream *url.URL, clientID string, log *slog.Logger) {
|
||||
ws := "ws://" + upstream.Host + "/ws?clientId=" + url.QueryEscape(clientID)
|
||||
wait := time.Second
|
||||
for ctx.Err() == nil {
|
||||
conn, err := websocket.Dial(ws, "", "http://"+upstream.Host)
|
||||
if err == nil {
|
||||
wait = time.Second
|
||||
go func() { <-ctx.Done(); conn.Close() }()
|
||||
for {
|
||||
var msg []byte
|
||||
if err := websocket.Message.Receive(conn, &msg); err != nil {
|
||||
break
|
||||
}
|
||||
// Binary frames are previews of pictures; only text is news.
|
||||
if len(msg) > 0 && msg[0] == '{' {
|
||||
b.read(msg)
|
||||
}
|
||||
}
|
||||
conn.Close()
|
||||
} else if log != nil {
|
||||
log.Debug("ComfyUI's websocket is not open yet", "err", err)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(wait):
|
||||
}
|
||||
wait = min(wait*2, 30*time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
// withClient is a /prompt body with the gate's client id in it, so the
|
||||
// progress of the workflow it queues comes to the gate's websocket.
|
||||
func withClient(body []byte, clientID string) []byte {
|
||||
var m map[string]json.RawMessage
|
||||
if json.Unmarshal(body, &m) != nil {
|
||||
return body
|
||||
}
|
||||
id, _ := json.Marshal(clientID)
|
||||
m["client_id"] = id
|
||||
out, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// promptThroughGate forwards a /prompt with the gate's client id, and books
|
||||
// the workflow it queued as queued.
|
||||
func (b *progressBook) promptThroughGate(proxy http.Handler, clientID string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, 64<<20))
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
body = withClient(body, clientID)
|
||||
r.Body = io.NopCloser(bytes.NewReader(body))
|
||||
r.ContentLength = int64(len(body))
|
||||
rec := &captured{ResponseWriter: w}
|
||||
proxy.ServeHTTP(rec, r)
|
||||
var out struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
}
|
||||
if json.Unmarshal(rec.body.Bytes(), &out) == nil && out.PromptID != "" {
|
||||
b.update(out.PromptID, func(*Progress) {})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// captured passes a response on and keeps a copy of the first of it.
|
||||
type captured struct {
|
||||
http.ResponseWriter
|
||||
body bytes.Buffer
|
||||
}
|
||||
|
||||
func (c *captured) Write(p []byte) (int, error) {
|
||||
if c.body.Len() < 64<<10 {
|
||||
c.body.Write(p)
|
||||
}
|
||||
return c.ResponseWriter.Write(p)
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBookReadsComfyUIsWebsocket(t *testing.T) {
|
||||
b := newProgressBook()
|
||||
for _, m := range []string{
|
||||
`{"type":"status","data":{"status":{"exec_info":{"queue_remaining":1}}}}`,
|
||||
`{"type":"execution_start","data":{"prompt_id":"p1"}}`,
|
||||
`{"type":"executing","data":{"node":"5","prompt_id":"p1"}}`,
|
||||
`{"type":"progress","data":{"value":12,"max":30,"prompt_id":"p1","node":"5"}}`,
|
||||
} {
|
||||
b.read([]byte(m))
|
||||
}
|
||||
p, _ := b.get("p1")
|
||||
if p.State != "running" || p.Node != "5" || p.Step != 12 || p.Steps != 30 || p.Percent != 40 {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
b.read([]byte(`{"type":"executing","data":{"node":null,"prompt_id":"p1"}}`))
|
||||
if p, _ := b.get("p1"); p.State != "done" || p.Percent != 100 {
|
||||
t.Errorf("finished: %+v", p)
|
||||
}
|
||||
b.read([]byte(`{"type":"execution_error","data":{"prompt_id":"p2","node_type":"KSampler","exception_message":"CUDA out of memory"}}`))
|
||||
if p, _ := b.get("p2"); p.State != "failed" || p.Error != "KSampler: CUDA out of memory" {
|
||||
t.Errorf("failed: %+v", p)
|
||||
}
|
||||
for i := range keptPrompts + 10 {
|
||||
b.update(fmt.Sprint("x", i), func(*Progress) {})
|
||||
}
|
||||
if _, ok := b.get("p1"); ok || len(b.by) != keptPrompts {
|
||||
t.Errorf("the book kept %d", len(b.by))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGateHearsTheWorkflowsItLetsThrough(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var sent map[string]any
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/prompt" {
|
||||
mu.Lock()
|
||||
_ = json.NewDecoder(r.Body).Decode(&sent)
|
||||
mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"p9"}`)
|
||||
}
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
book := newProgressBook()
|
||||
gate := httptest.NewServer(newGate(u, "", "m", book, "llmgw-gate-1"))
|
||||
defer gate.Close()
|
||||
resp, err := http.Post(gate.URL+"/prompt", "application/json", strings.NewReader(`{"prompt":{},"client_id":"router"}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if !strings.Contains(string(b), "p9") {
|
||||
t.Fatalf("the answer did not come through: %s", b)
|
||||
}
|
||||
mu.Lock()
|
||||
if sent["client_id"] != "llmgw-gate-1" {
|
||||
t.Errorf("ComfyUI was sent client %v: its progress would go to nobody listening", sent["client_id"])
|
||||
}
|
||||
mu.Unlock()
|
||||
book.read([]byte(`{"type":"progress","data":{"value":3,"max":4,"prompt_id":"p9","node":"2"}}`))
|
||||
resp, _ = http.Get(gate.URL + "/progress/p9")
|
||||
var p Progress
|
||||
_ = json.NewDecoder(resp.Body).Decode(&p)
|
||||
resp.Body.Close()
|
||||
if p.State != "running" || p.Percent != 75 {
|
||||
t.Errorf("/progress: %+v", p)
|
||||
}
|
||||
if resp, _ := http.Get(gate.URL + "/progress/unknown"); resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("an unknown workflow: %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWaitingWorkflowSaysHowManyAreAheadAndThenHowFarAlong(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
stage := 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
s := stage
|
||||
mu.Unlock()
|
||||
switch r.URL.Path {
|
||||
case "/prompt":
|
||||
fmt.Fprint(w, `{"prompt_id":"me"}`)
|
||||
case "/queue":
|
||||
if s == 0 {
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"a"]],"queue_pending":[[3,"me"],[2,"b"],[4,"c"]]}`)
|
||||
} else {
|
||||
fmt.Fprint(w, `{"queue_running":[[3,"me"]],"queue_pending":[]}`)
|
||||
}
|
||||
case "/progress/me":
|
||||
fmt.Fprint(w, `{"state":"running","node":"5","step":6,"steps":12,"percent":50}`)
|
||||
case "/history/me":
|
||||
if s < 2 {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"me":{"status":{"status_str":"success","completed":true},"outputs":{"9":{"images":[{"filename":"a.png","type":"temp"}]}}}}`)
|
||||
case "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png")
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
var seen []Progress
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := c.GenerateWatch(t.Context(), srv.URL, "", map[string]any{"5": map[string]any{"class_type": "KSampler"}},
|
||||
func(p Progress) {
|
||||
mu.Lock()
|
||||
seen = append(seen, p)
|
||||
stage++
|
||||
mu.Unlock()
|
||||
})
|
||||
done <- err
|
||||
}()
|
||||
if err := <-done; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(seen) < 2 || seen[0].State != "queued" || seen[0].Ahead != 2 {
|
||||
t.Fatalf("%+v: queued behind the one running and the one numbered before it, wanted first", seen)
|
||||
}
|
||||
if seen[1].State != "running" || seen[1].Percent != 50 || seen[1].NodeClass != "KSampler" {
|
||||
t.Errorf("then %+v", seen[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeIsPutAtItsCommit(t *testing.T) {
|
||||
if _, err := exec.LookPath("git"); err != nil {
|
||||
t.Skip("no git")
|
||||
}
|
||||
repo := t.TempDir()
|
||||
run := func(args ...string) string {
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = repo
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t")
|
||||
b, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v %s", args, err, b)
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
run("init", "-q")
|
||||
run("config", "uploadpack.allowReachableSHA1InWant", "true")
|
||||
_ = os.WriteFile(filepath.Join(repo, "__init__.py"), []byte("v1\n"), 0o644)
|
||||
run("add", ".")
|
||||
run("commit", "-qm", "one")
|
||||
first := run("rev-parse", "HEAD")
|
||||
_ = os.WriteFile(filepath.Join(repo, "__init__.py"), []byte("v2\n"), 0o644)
|
||||
run("commit", "-qam", "two")
|
||||
|
||||
state := t.TempDir()
|
||||
site, err := installNodes(t.Context(), state, "python3", []Node{{Repo: "file://" + repo, Commit: first}}, "", func(string) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "custom_nodes", filepath.Base(repo), "__init__.py"))
|
||||
if string(b) != "v1\n" || site != "" {
|
||||
t.Errorf("the node is %q, site %q: wanted the commit named, not the branch's head", b, site)
|
||||
}
|
||||
if _, err := installNodes(t.Context(), state, "python3", []Node{{Repo: "file://" + repo, Commit: first,
|
||||
Requirements: []string{"trimesh"}}}, "", func(string) {}); err == nil || !strings.Contains(err.Error(), "no uv") {
|
||||
t.Errorf("packages without a uv: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOwnRangeDecidesThePackagesItNames(t *testing.T) {
|
||||
freeze := []byte("numpy==2.3.1\ntransformers==5.19.0\nhuggingface-hub==1.33.0\nPillow==12.3.0\n")
|
||||
got := string(withoutNamed(freeze, []string{"transformers>=4.50.3,<5", "Huggingface_Hub<1", "trimesh"}))
|
||||
if strings.Contains(got, "transformers") || strings.Contains(got, "huggingface") ||
|
||||
!strings.Contains(got, "numpy==2.3.1") || !strings.Contains(got, "Pillow==12.3.0") {
|
||||
t.Errorf("constraints %q", got)
|
||||
}
|
||||
}
|
||||
@@ -496,6 +496,8 @@ type Price struct {
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
// PerMesh is the cost of one 3D model (/v1/3d/generations).
|
||||
PerMesh int64 `koanf:"per_mesh"`
|
||||
}
|
||||
|
||||
// Control configures the control plane.
|
||||
@@ -689,6 +691,10 @@ type Band struct {
|
||||
Name string `koanf:"name"`
|
||||
Priority int `koanf:"priority"`
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL is how long this band's requests may wait for room; the
|
||||
// admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
// had rather wait behind people than be turned away.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Timeouts bounds each phase of an upstream request.
|
||||
|
||||
@@ -203,6 +203,9 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/origin", a.setOrigin)
|
||||
mux.HandleFunc("GET /api/v1/plan", a.planModel)
|
||||
mux.HandleFunc("POST /api/v1/metrics/range", a.metricsBatch)
|
||||
mux.HandleFunc("GET /api/v1/client-jobs", a.listJobs)
|
||||
mux.HandleFunc("GET /api/v1/client-jobs/{id}", a.getJob)
|
||||
mux.HandleFunc("POST /api/v1/client-jobs/{id}/cancel", a.cancelJob)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}/body", a.requestBody)
|
||||
mux.HandleFunc("POST /api/v1/requests/{id}/replay", a.replayRequest)
|
||||
@@ -1612,6 +1615,7 @@ type RequestView struct {
|
||||
PromptTokens int64 `json:"prompt_tokens"`
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
Images int64 `json:"images,omitempty"`
|
||||
Meshes int64 `json:"meshes,omitempty"`
|
||||
CostMicros int64 `json:"cost_micros"`
|
||||
// What the guardrails did, by name.
|
||||
GuardAction string `json:"guard_action,omitempty"`
|
||||
@@ -1645,7 +1649,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
|
||||
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
|
||||
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, Images: u.Images, CostMicros: u.CostMicros,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, Images: u.Images, Meshes: u.Meshes, CostMicros: u.CostMicros,
|
||||
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
|
||||
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
|
||||
})
|
||||
|
||||
@@ -75,11 +75,11 @@ func (a *API) usageExport(w http.ResponseWriter, r *http.Request) {
|
||||
func WriteExportCSV(w io.Writer, rows []store.ExportRow) {
|
||||
c := csv.NewWriter(w)
|
||||
_ = c.Write([]string{"period_start", "org", "team", "project", "user", "key", "model",
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost"})
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost", "images", "meshes"})
|
||||
for _, r := range rows {
|
||||
_ = c.Write([]string{r.Period.Format(time.RFC3339), safe(r.Org), safe(r.Team), safe(r.Project), safe(r.User),
|
||||
r.Key, safe(r.Model), i(r.Requests), i(r.Errors), i(r.Limited), i(r.Prompt), i(r.Completion),
|
||||
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64)})
|
||||
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64), i(r.Images), i(r.Meshes)})
|
||||
}
|
||||
c.Flush()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Jobs — 3D models asked for and asked after — as an operator sees them:
|
||||
// every owner's, which a client sees only of its own.
|
||||
|
||||
// jobStore is what jobs need of the store; asked of the usage source by
|
||||
// type assertion, as requestFinder is.
|
||||
type jobStore interface {
|
||||
AllJobs(ctx context.Context, status string, limit int) ([]store.Job, error)
|
||||
JobByID(ctx context.Context, id string) (store.Job, error)
|
||||
CancelJob(ctx context.Context, id string, now time.Time) (store.Job, error)
|
||||
}
|
||||
|
||||
// JobView is one job.
|
||||
type JobView struct {
|
||||
ID string `json:"id"`
|
||||
Owner string `json:"owner"`
|
||||
KeyID string `json:"key_id"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Model string `json:"model"`
|
||||
Status string `json:"status"`
|
||||
Progress json.RawMessage `json:"progress,omitempty"`
|
||||
OutputFileID string `json:"output_file_id,omitempty"`
|
||||
ContentType string `json:"content_type,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
|
||||
func jobView(j store.Job) JobView {
|
||||
v := JobView{ID: j.ID, Owner: j.Owner, KeyID: j.KeyID, Endpoint: j.Endpoint, Model: j.Model, Status: j.Status,
|
||||
OutputFileID: j.OutputFileID, ContentType: j.ContentType, Error: j.Error, CreatedAt: j.CreatedAt}
|
||||
if j.Progress != "" && json.Valid([]byte(j.Progress)) {
|
||||
v.Progress = json.RawMessage(j.Progress)
|
||||
}
|
||||
if !j.StartedAt.IsZero() {
|
||||
v.StartedAt = &j.StartedAt
|
||||
}
|
||||
if !j.FinishedAt.IsZero() {
|
||||
v.FinishedAt = &j.FinishedAt
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (a *API) jobs() (jobStore, bool) {
|
||||
s, ok := a.usage.(jobStore)
|
||||
return s, ok
|
||||
}
|
||||
|
||||
func (a *API) listJobs(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||
js, err := s.AllJobs(r.Context(), r.URL.Query().Get("status"), limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]JobView, 0, len(js))
|
||||
for _, j := range js {
|
||||
out = append(out, jobView(j))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) getJob(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
j, err := s.JobByID(r.Context(), r.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no job "+r.PathValue("id"))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, jobView(j))
|
||||
}
|
||||
|
||||
func (a *API) cancelJob(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
if _, err := s.JobByID(r.Context(), r.PathValue("id")); errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no job "+r.PathValue("id"))
|
||||
return
|
||||
}
|
||||
j, err := s.CancelJob(r.Context(), r.PathValue("id"), time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, jobView(j))
|
||||
}
|
||||
@@ -62,6 +62,7 @@ var PathParams = map[string]string{
|
||||
"jobs/{id}": "The job's ID, as starting it answered.",
|
||||
"runs/{id}": "The quality run's ID, as starting it answered.",
|
||||
"requests/{id}": "The request's ID: the x-request-id a client was answered with.",
|
||||
"client-jobs/{id}": "The job's ID, job_….",
|
||||
"fabrics/{id}": "The physical fabric's ID, as GET /api/v1/fabrics lists it.",
|
||||
"quotas/{scope}": "The scope: org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT, or their IDs after the colon.",
|
||||
"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
|
||||
@@ -397,6 +398,14 @@ var Routes = []Route{
|
||||
{Name: "format", Description: "json for JSON; CSV otherwise."},
|
||||
},
|
||||
Response: []store.ExportRow{}, CSV: true},
|
||||
{Method: "GET", Path: "/api/v1/client-jobs", Summary: "Jobs clients asked for — 3D models — of every owner, newest first",
|
||||
Query: []Param{
|
||||
{Name: "status", Description: "Only jobs in this status: queued, running, cancelling, completed, failed, cancelled, expired."},
|
||||
{Name: "limit", Type: "integer", Description: "The most jobs to answer with; 50 when absent."},
|
||||
},
|
||||
Response: []JobView{}},
|
||||
{Method: "GET", Path: "/api/v1/client-jobs/{id}", Summary: "One job a client asked for", Response: JobView{}},
|
||||
{Method: "POST", Path: "/api/v1/client-jobs/{id}/cancel", Summary: "Stop a job, in the engine's queue too", Response: JobView{}},
|
||||
{Method: "GET", Path: "/api/v1/requests/{id}", Summary: "What one request went through",
|
||||
Response: RequestView{}},
|
||||
{Method: "GET", Path: "/api/v1/requests/{id}/body", Summary: "A kept request and its answer",
|
||||
|
||||
@@ -30,7 +30,7 @@ type OrphanView struct {
|
||||
func (a *API) Orphans() []OrphanView {
|
||||
var wanted []string
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
wanted = append(wanted, m.Source)
|
||||
wanted = append(wanted, m.WeightSources()...)
|
||||
// A model's adapters are its weights too; collecting them would
|
||||
// leave replicas that cannot restart.
|
||||
for _, ad := range m.Adapters {
|
||||
|
||||
@@ -7,8 +7,10 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/url"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -174,6 +176,12 @@ type Profile struct {
|
||||
// 0.10.2. Empty takes whatever version the node has.
|
||||
EngineVersion string `json:"engine_version,omitempty" koanf:"engine_version"`
|
||||
GPUClass string `json:"gpu_class,omitempty" koanf:"gpu_class"`
|
||||
// Source is the weights this profile loads, when not the model's: an
|
||||
// embedding model served by vLLM from its repository on a card and by
|
||||
// llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
// loads the model's. Updates and canaries move the model's source; a
|
||||
// profile with its own keeps it.
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
|
||||
TP int `json:"tp,omitempty" koanf:"tp"`
|
||||
PP int `json:"pp,omitempty" koanf:"pp"`
|
||||
@@ -326,6 +334,10 @@ type Model struct {
|
||||
// ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
// pictures: the workflow, and the file it loads.
|
||||
ComfyUI *ComfyUI `json:"comfyui,omitempty" koanf:"comfyui"`
|
||||
// License is the terms of what the model makes, for a model whose
|
||||
// output somebody ships: shown with every result and written into a
|
||||
// mesh.
|
||||
License *License `json:"license,omitempty" koanf:"license"`
|
||||
// Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
// for by its own name as though it were a model. Every replica loads
|
||||
// every adapter, applying one only to a request that names it — unless
|
||||
@@ -466,11 +478,52 @@ type Capabilities struct {
|
||||
// /v1/images/generations. Implied by an external model whose API is
|
||||
// ComfyUI's.
|
||||
Images bool `json:"images,omitempty" koanf:"images"`
|
||||
// Meshes is a model that makes 3D models from a picture, asked for as a
|
||||
// job at /v1/3d/generations. Implied by a ComfyUI model whose output is
|
||||
// meshes.
|
||||
Meshes bool `json:"meshes,omitempty" koanf:"meshes"`
|
||||
}
|
||||
|
||||
// The kinds of thing a ComfyUI model makes.
|
||||
const (
|
||||
OutputPictures = "pictures"
|
||||
OutputMeshes = "meshes"
|
||||
)
|
||||
|
||||
// comfyOutput is what a ComfyUI model makes: pictures unless it says.
|
||||
func (m Model) comfyOutput() string {
|
||||
out := ""
|
||||
switch {
|
||||
case m.External != nil:
|
||||
out = m.External.Output
|
||||
case m.ComfyUI != nil:
|
||||
out = m.ComfyUI.Output
|
||||
}
|
||||
if out == "" {
|
||||
return OutputPictures
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MakesImages reports whether a model answers /v1/images/generations.
|
||||
func (m Model) MakesImages() bool {
|
||||
return m.Capabilities.Images || m.ServesComfyUI()
|
||||
return m.Capabilities.Images || m.ServesComfyUI() && m.comfyOutput() == OutputPictures
|
||||
}
|
||||
|
||||
// MakesMeshes reports whether a model answers /v1/3d/generations.
|
||||
func (m Model) MakesMeshes() bool {
|
||||
return m.Capabilities.Meshes || m.ServesComfyUI() && m.comfyOutput() == OutputMeshes
|
||||
}
|
||||
|
||||
// License is the terms what a model makes comes under: said beside every
|
||||
// result, and written into a mesh, which travels without the API.
|
||||
type License struct {
|
||||
// Name is the licence's own: MIT, Apache-2.0.
|
||||
Name string `json:"name" koanf:"name"`
|
||||
URL string `json:"url,omitempty" koanf:"url"`
|
||||
// Notice is what must accompany what was made: attribution, terms on
|
||||
// the output, what the weights were trained on.
|
||||
Notice string `json:"notice,omitempty" koanf:"notice"`
|
||||
}
|
||||
|
||||
// EngineComfyUI is the engine that runs ComfyUI on a node.
|
||||
@@ -495,6 +548,56 @@ type ComfyUI struct {
|
||||
// source's own name unless set, which a source that is a set of files
|
||||
// must.
|
||||
Checkpoint string `json:"checkpoint,omitempty" koanf:"checkpoint"`
|
||||
// Output is what the workflow makes: pictures (the default), asked for
|
||||
// at /v1/images/generations, or meshes, asked for as a job at
|
||||
// /v1/3d/generations from a picture.
|
||||
Output string `json:"output,omitempty" koanf:"output"`
|
||||
// Nodes are custom nodes the workflow needs, put in each replica's own
|
||||
// custom_nodes at the commit named.
|
||||
Nodes []CustomNode `json:"nodes,omitempty" koanf:"nodes"`
|
||||
}
|
||||
|
||||
// CustomNode is a ComfyUI node from a git repository.
|
||||
type CustomNode struct {
|
||||
// Repo is the repository, https only.
|
||||
Repo string `json:"repo" koanf:"repo"`
|
||||
// Commit is the full commit the node is used at: a branch moves, and a
|
||||
// node that changed under a model is a model that changed unasked.
|
||||
Commit string `json:"commit" koanf:"commit"`
|
||||
// Requirements are the PyPI packages the node needs, by name and version
|
||||
// range only. They are installed beside the replica, never into
|
||||
// ComfyUI's environment: a package named here at the range named, and
|
||||
// every other one at the version ComfyUI already runs. One named here
|
||||
// is first on ComfyUI's path, so its range must suit ComfyUI too.
|
||||
Requirements []string `json:"requirements,omitempty" koanf:"requirements"`
|
||||
}
|
||||
|
||||
// NodesEnv carries a ComfyUI model's custom nodes to the agent, which hands
|
||||
// them to the gate.
|
||||
const NodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
|
||||
var (
|
||||
fullCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
|
||||
packageSpec = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*(\[[A-Za-z0-9,._-]+\])?\s*([<>=!~]=?\s*[A-Za-z0-9.*+!_-]+\s*,?\s*)*$`)
|
||||
)
|
||||
|
||||
func (n CustomNode) validate(model string) error {
|
||||
u, err := url.Parse(n.Repo)
|
||||
if err != nil || u.Scheme != "https" || u.Host == "" || u.User != nil {
|
||||
return fmt.Errorf("model %q: custom node %q is not an https repository", model, n.Repo)
|
||||
}
|
||||
if !fullCommit.MatchString(n.Commit) {
|
||||
return fmt.Errorf("model %q: custom node %s needs its full commit (40 hex), not %q: a branch moves",
|
||||
model, n.Repo, n.Commit)
|
||||
}
|
||||
for _, r := range n.Requirements {
|
||||
// A package and a version range: an option or a URL would install
|
||||
// from somewhere other than PyPI.
|
||||
if !packageSpec.MatchString(strings.TrimSpace(r)) {
|
||||
return fmt.Errorf("model %q: custom node %s requirement %q is not a package with a version range", model, n.Repo, r)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ComfyWorkflow is what a ComfyUI model's replicas are sent: the workflow
|
||||
@@ -549,6 +652,19 @@ func (m Model) validateComfyUI() error {
|
||||
if err := checkWorkflow(m.Name, w); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
for _, n := range m.ComfyUI.Nodes {
|
||||
if err := n.validate(m.Name); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if out := m.comfyOutput(); out != OutputPictures && out != OutputMeshes {
|
||||
return fmt.Errorf("model %q: comfyui.output %q is not pictures or meshes", m.Name, out)
|
||||
}
|
||||
if m.comfyOutput() == OutputMeshes && w == "" {
|
||||
return fmt.Errorf("model %q: a model that makes meshes names its workflow: no built-in makes them", m.Name)
|
||||
}
|
||||
if _, ckpt := m.ComfyWorkflow(); ckpt == "" {
|
||||
return fmt.Errorf("model %q: its source is a set of files, so comfyui.checkpoint must name the one the workflow loads",
|
||||
m.Name)
|
||||
@@ -762,9 +878,15 @@ func (m Model) normalize() Model {
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
for i := range m.Profiles {
|
||||
p := &m.Profiles[i]
|
||||
if repo, ok := repositoryPage(p.Source); ok && p.Engine != "llamacpp" {
|
||||
p.Source = repo
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -775,15 +897,26 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a source
|
||||
// llama.cpp loads is left here, which needs a file.
|
||||
if repo, ok := repositoryPage(m.Source); ok && (len(m.Profiles) == 0 ||
|
||||
slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" })) {
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if repo, ok := repositoryPage(p.Source); ok {
|
||||
return fmt.Errorf("model %q profile %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, p.Name, p.Source, repo)
|
||||
}
|
||||
if p.Source != "" && m.External != nil {
|
||||
return fmt.Errorf("model %q profile %q: an external model has no profiles to give a source", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -924,7 +1057,7 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: trtllm runs a replica on one node; "+
|
||||
"use vllm or sglang for a %d-node replica", m.Name, p.Name, p.Nodes)
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.Source, "/")), ".gguf") {
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
|
||||
return fmt.Errorf("model %q profile %q: trtllm loads a Hugging Face checkpoint, not a .gguf; "+
|
||||
"serve a GGUF with llamacpp", m.Name, p.Name)
|
||||
}
|
||||
@@ -1010,20 +1143,53 @@ func (m Model) PlacementProfiles() []placement.Profile {
|
||||
CPUCores: p.CPUCores,
|
||||
PowerWatts: p.PowerWatts,
|
||||
KeepWithin: facility.Level(strings.ToLower(strings.TrimSpace(p.KeepWithin))),
|
||||
// The model's source, so placement can prefer nodes that already
|
||||
// The profile's source, so placement can prefer nodes that already
|
||||
// hold the weights rather than one that must first move them.
|
||||
Source: m.Source,
|
||||
Source: m.SourceOf(p),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SourceOf is the weights a profile loads: its own, or the model's.
|
||||
func (m Model) SourceOf(p Profile) string {
|
||||
if p.Source != "" {
|
||||
return p.Source
|
||||
}
|
||||
return m.Source
|
||||
}
|
||||
|
||||
// WeightSources are every source a model's replicas may load: the model's
|
||||
// and each profile's of its own, once each. What a node must not have
|
||||
// collected from under it.
|
||||
func (m Model) WeightSources() []string {
|
||||
out := []string{}
|
||||
if m.Source != "" {
|
||||
out = append(out, m.Source)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.Source != "" && !slices.Contains(out, p.Source) {
|
||||
out = append(out, p.Source)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EngineProfile converts a catalog profile into the engine adapter's view.
|
||||
func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
for _, p := range m.Profiles {
|
||||
if p.Name != name {
|
||||
continue
|
||||
}
|
||||
env := p.Env
|
||||
if p.Engine == EngineComfyUI && m.ComfyUI != nil && len(m.ComfyUI.Nodes) > 0 {
|
||||
nodes, _ := json.Marshal(m.ComfyUI.Nodes)
|
||||
env = maps.Clone(p.Env)
|
||||
if env == nil {
|
||||
env = map[string]string{}
|
||||
}
|
||||
env[NodesEnv] = string(nodes)
|
||||
}
|
||||
return engines.Profile{
|
||||
Name: p.Name,
|
||||
Engine: p.Engine,
|
||||
@@ -1033,8 +1199,8 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
GPUsPerNode: p.GPUsPerNode,
|
||||
Nodes: p.Nodes,
|
||||
Args: slices.Clone(p.Args),
|
||||
Env: p.Env,
|
||||
ModelPath: m.Source,
|
||||
Env: env,
|
||||
ModelPath: m.SourceOf(p),
|
||||
ServedModelName: m.Name,
|
||||
Adapters: m.engineAdapters(),
|
||||
KVEvents: p.KVEvents,
|
||||
@@ -1210,6 +1376,8 @@ type External struct {
|
||||
// (flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
// plane.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
Output string `json:"output,omitempty" koanf:"output"`
|
||||
}
|
||||
|
||||
// APIComfyUI is External.API for a ComfyUI server.
|
||||
@@ -1231,8 +1399,8 @@ func (e *External) validate(model string, profiles int) error {
|
||||
}
|
||||
switch e.API {
|
||||
case "", "openai":
|
||||
if e.Workflow != "" {
|
||||
return fmt.Errorf("model %q: a workflow is for api: comfyui", model)
|
||||
if e.Workflow != "" || e.Output != "" {
|
||||
return fmt.Errorf("model %q: a workflow and an output are for api: comfyui", model)
|
||||
}
|
||||
case APIComfyUI:
|
||||
if e.Peer != "" {
|
||||
@@ -1241,6 +1409,12 @@ func (e *External) validate(model string, profiles int) error {
|
||||
if err := checkWorkflow(model, e.Workflow); err != nil {
|
||||
return err
|
||||
}
|
||||
if e.Output != "" && e.Output != OutputPictures && e.Output != OutputMeshes {
|
||||
return fmt.Errorf("model %q: external output %q is not pictures or meshes", model, e.Output)
|
||||
}
|
||||
if e.Output == OutputMeshes && e.Workflow == "" {
|
||||
return fmt.Errorf("model %q: a model that makes meshes names its workflow: no built-in makes them", model)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("model %q: external api %q is not openai or comfyui", model, e.API)
|
||||
}
|
||||
|
||||
@@ -49,3 +49,35 @@ func TestAComfyUIModelIsRefusedWhatItCannotDo(t *testing.T) {
|
||||
t.Errorf("a set with its checkpoint named was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACustomNodeIsPinnedAndItsPackagesArePackages(t *testing.T) {
|
||||
node := func(n CustomNode) Model {
|
||||
return comfyModel("hf:stabilityai/TripoSR/model.ckpt", &ComfyUI{Workflow: "w.json", Output: OutputMeshes,
|
||||
Nodes: []CustomNode{n}})
|
||||
}
|
||||
ok := CustomNode{Repo: "https://github.com/flowtyone/ComfyUI-Flowty-TripoSR",
|
||||
Commit: "0123456789abcdef0123456789abcdef01234567", Requirements: []string{"omegaconf", "trimesh>=4,<5", "scikit-image"}}
|
||||
cat, err := New([]Model{node(ok)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
p, _ := m.EngineProfile("gpu")
|
||||
if !strings.Contains(p.Env[NodesEnv], "ComfyUI-Flowty-TripoSR") || !m.MakesMeshes() || m.MakesImages() {
|
||||
t.Errorf("env %v meshes %v pictures %v", p.Env, m.MakesMeshes(), m.MakesImages())
|
||||
}
|
||||
for name, n := range map[string]CustomNode{
|
||||
"a branch": {Repo: ok.Repo, Commit: "main"},
|
||||
"http": {Repo: "http://example.com/n", Commit: ok.Commit},
|
||||
"an option": {Repo: ok.Repo, Commit: ok.Commit, Requirements: []string{"--index-url=https://evil"}},
|
||||
"a url": {Repo: ok.Repo, Commit: ok.Commit, Requirements: []string{"x @ https://evil/x.whl"}},
|
||||
} {
|
||||
if _, err := New([]Model{node(n)}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
noWorkflow := comfyModel("hf:a/b/c.ckpt", &ComfyUI{Output: OutputMeshes})
|
||||
if _, err := New([]Model{noWorkflow}); err == nil || !strings.Contains(err.Error(), "names its workflow") {
|
||||
t.Errorf("a mesh model without a workflow: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// embedModel is an embedding model the way it was wanted on the PoC: vLLM
|
||||
// on the card from the repository, llama.cpp on CPUs from a GGUF.
|
||||
func embedModel() Model {
|
||||
return Model{Name: "qwen3-embed-4b", Source: "Qwen/Qwen3-Embedding-4B",
|
||||
Profiles: []Profile{
|
||||
{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1},
|
||||
{Name: "cpu", Engine: "llamacpp", Source: "hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf"},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestEachProfileLoadsItsOwnSource(t *testing.T) {
|
||||
// The catalog had one source a model, so serving an embedding model by
|
||||
// vLLM on a card and by llama.cpp on CPUs meant swapping the entry by
|
||||
// hand halfway through an import.
|
||||
cat, err := New([]Model{embedModel()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
gpu, _ := m.EngineProfile("gpu")
|
||||
cpu, _ := m.EngineProfile("cpu")
|
||||
if gpu.ModelPath != "Qwen/Qwen3-Embedding-4B" || !strings.HasSuffix(cpu.ModelPath, "Q8_0.gguf") {
|
||||
t.Errorf("gpu loads %q, cpu %q", gpu.ModelPath, cpu.ModelPath)
|
||||
}
|
||||
for _, p := range m.PlacementProfiles() {
|
||||
if want := map[string]string{"gpu": gpu.ModelPath, "cpu": cpu.ModelPath}[p.Name]; p.Source != want {
|
||||
t.Errorf("placement for %s looks for %q, want %q", p.Name, p.Source, want)
|
||||
}
|
||||
}
|
||||
if got := m.WeightSources(); len(got) != 2 || !slices.Contains(got, cpu.ModelPath) {
|
||||
t.Errorf("weight sources %v: a profile's own must not be collected", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileWithoutASourceKeepsItsFingerprint(t *testing.T) {
|
||||
// Replicas are compared with their profile by fingerprint; a field added
|
||||
// to every profile must not make every running replica look stale.
|
||||
p := Profile{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1}
|
||||
before := p.Fingerprint()
|
||||
if strings.Contains(mustJSON(t, p), "source") {
|
||||
t.Error("an empty source is written into the profile")
|
||||
}
|
||||
p.Source = "Qwen/Other"
|
||||
if p.Fingerprint() == before {
|
||||
t.Error("a profile whose source changed reads as unchanged")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileSourceIsCheckedAsTheModelsIs(t *testing.T) {
|
||||
page := embedModel()
|
||||
page.Profiles[1].Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF"
|
||||
if _, err := New([]Model{page}); err == nil || !strings.Contains(err.Error(), `profile "cpu"`) {
|
||||
t.Errorf("a repository page for llama.cpp: %v", err)
|
||||
}
|
||||
// The model's own source may be a page vLLM reads as a repository, now
|
||||
// that the llama.cpp profile loads something else.
|
||||
repo := embedModel()
|
||||
repo.Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B"
|
||||
cat, err := New([]Model{repo})
|
||||
if err != nil {
|
||||
t.Fatalf("a repository page for the vLLM profile beside a llama.cpp one with its own GGUF: %v", err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
if m.Source != "Qwen/Qwen3-Embedding-4B" {
|
||||
t.Errorf("the page was not read as the repository: %q", m.Source)
|
||||
}
|
||||
trt := embedModel()
|
||||
trt.Profiles[0] = Profile{Name: "gpu", Engine: "trtllm", GPUClass: "6000-24g", TP: 1, Source: "hf:a/b/m.gguf"}
|
||||
if _, err := New([]Model{trt}); err == nil || !strings.Contains(err.Error(), "not a .gguf") {
|
||||
t.Errorf("trtllm given a GGUF by its profile: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(t *testing.T, v any) string {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
@@ -87,8 +87,8 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
|
||||
}
|
||||
out := map[string][]catalog.Model{}
|
||||
for _, m := range c.Models() {
|
||||
if m.Source != "" {
|
||||
out[m.Source] = append(out[m.Source], m)
|
||||
for _, src := range m.WeightSources() {
|
||||
out[src] = append(out[src], m)
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
out[ad.Source] = append(out[ad.Source], m)
|
||||
|
||||
@@ -65,10 +65,13 @@ var Reference = []Block{
|
||||
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "images", Type: "bool", Doc: "Images is a model that makes pictures, asked for at\n/v1/images/generations. Implied by an external model whose API is\nComfyUI's.", Nested: ""},
|
||||
{Key: "meshes", Type: "bool", Doc: "Meshes is a model that makes 3D models from a picture, asked for as a\njob at /v1/3d/generations. Implied by a ComfyUI model whose output is\nmeshes.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ComfyUI", Title: "ComfyUI", Doc: "", Fields: []Field{
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is a built-in's name (flux-schnell, the default, or\nflux-dev) or a JSON file on the control plane, in ComfyUI's API\nformat.", Nested: ""},
|
||||
{Key: "checkpoint", Type: "string", Doc: "Checkpoint is the file the workflow's {{checkpoint}} names: the\nsource's own name unless set, which a source that is a set of files\nmust.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what the workflow makes: pictures (the default), asked for\nat /v1/images/generations, or meshes, asked for as a job at\n/v1/3d/generations from a picture.", Nested: ""},
|
||||
{Key: "nodes", Type: "[]CustomNode", Doc: "Nodes are custom nodes the workflow needs, put in each replica's own\ncustom_nodes at the commit named.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "ComfyUIConfig", Title: "ComfyUIConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is ComfyUI's directory, where main.py is. Readable by the user\nengines run as.", Nested: ""},
|
||||
@@ -87,6 +90,11 @@ var Reference = []Block{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
|
||||
{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "CustomNode", Title: "CustomNode", Doc: "", Fields: []Field{
|
||||
{Key: "repo", Type: "string", Doc: "Repo is the repository, https only.", Nested: ""},
|
||||
{Key: "commit", Type: "string", Doc: "Commit is the full commit the node is used at: a branch moves, and a\nnode that changed under a model is a model that changed unasked.", Nested: ""},
|
||||
{Key: "requirements", Type: "[]string", Doc: "Requirements are the PyPI packages the node needs, by name and version\nrange only. They are installed beside the replica, never into\nComfyUI's environment: a package named here at the range named, and\nevery other one at the version ComfyUI already runs. One named here\nis first on ComfyUI's path, so its range must suit ComfyUI too.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
|
||||
{Key: "id", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "", Nested: ""},
|
||||
@@ -116,6 +124,7 @@ var Reference = []Block{
|
||||
{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
|
||||
{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what it makes: pictures, or meshes (see ComfyUI.Output).", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
@@ -161,6 +170,11 @@ var Reference = []Block{
|
||||
{Key: "language", Type: "string", Doc: "Language is what the web portal speaks to someone who has not chosen\na language with the switcher at the foot of every page and whose\nbrowser asks for none it speaks: \"en\" or \"tr\". Empty is English.", Nested: ""},
|
||||
{Key: "docs_sign_in", Type: "bool", Doc: "DocsSignIn serves the docs portal at /docs only to someone signed in.\nOff, the portal is public like this page: its client guide is for the\nperson who has no account yet, and the operator documents describe\nllmgw in general — no address, model or key of this installation —\nwhich the project's repository publishes anyway. On, a visitor is sent\nto sign in, and without accounts the portal needs an API key.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "License", Title: "License", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "Name is the licence's own: MIT, Apache-2.0.", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "notice", Type: "string", Doc: "Notice is what must accompany what was made: attribution, terms on\nthe output, what the weights were trained on.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
|
||||
{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
|
||||
@@ -184,6 +198,7 @@ var Reference = []Block{
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
|
||||
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
|
||||
{Key: "comfyui", Type: "*ComfyUI", Doc: "ComfyUI is how a model whose profiles run engine comfyui makes its\npictures: the workflow, and the file it loads.", Nested: "catalog.ComfyUI"},
|
||||
{Key: "license", Type: "*License", Doc: "License is the terms of what the model makes, for a model whose\noutput somebody ships: shown with every result and written into a\nmesh.", Nested: "catalog.License"},
|
||||
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it — unless\nMaxLoadedAdapters is set.", Nested: ""},
|
||||
{Key: "max_loaded_adapters", Type: "int", Doc: "MaxLoadedAdapters, above zero, has each replica hold at most this many\nadapters at once, loaded when a request first asks for one and unloaded\nleast recently used first to make room. The router sends a request for\nan adapter to a replica that already holds it while that one has room\nto take it. vLLM profiles only: llama-server loads adapters at start\nand cannot load one afterwards. Zero loads every adapter on every\nreplica at start.", Nested: ""},
|
||||
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
|
||||
@@ -284,6 +299,7 @@ var Reference = []Block{
|
||||
{Key: "engine", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "engine_version", Type: "string", Doc: "EngineVersion restricts the profile to nodes whose installed engine's\nversion starts with it: \"0.11\" takes vLLM 0.11.0 and 0.11.1, not\n0.10.2. Empty takes whatever version the node has.", Nested: ""},
|
||||
{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "source", Type: "string", Doc: "Source is the weights this profile loads, when not the model's: an\nembedding model served by vLLM from its repository on a card and by\nllama.cpp from a GGUF on CPUs is one model with two sources. Empty\nloads the model's. Updates and canaries move the model's source; a\nprofile with its own keeps it.", Nested: ""},
|
||||
{Key: "tp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "pp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
|
||||
@@ -459,11 +475,13 @@ var Reference = []Block{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is what one 3D model costs: minutes of a card, which no\ntoken count measures.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL, set, is how long this band's requests may wait instead of\nthe controller's: a band for bulk work — a knowledge-base import, a\nbackfill — whose requests had rather wait ten minutes behind people\nthan be turned away after thirty seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
|
||||
{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
|
||||
@@ -499,11 +517,13 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL is how long this band's requests may wait for room; the\nadmission's queue_ttl unless set. Long for a band of bulk work, which\nhad rather wait behind people than be turned away.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is the cost of one 3D model (/v1/3d/generations).", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
|
||||
{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
|
||||
|
||||
@@ -110,6 +110,23 @@ type Notifier struct {
|
||||
mu sync.Mutex
|
||||
sent map[string]time.Time
|
||||
wg sync.WaitGroup
|
||||
|
||||
claim Claimer
|
||||
}
|
||||
|
||||
// Claimer is where control planes agree which of them sends an event: each
|
||||
// raises the same ones — a budget is spent on every control plane serving
|
||||
// its requests — and only the one that claims an event sends it.
|
||||
type Claimer interface {
|
||||
ClaimNotice(ctx context.Context, key string, at time.Time, window time.Duration) (bool, error)
|
||||
}
|
||||
|
||||
// SetClaimer has events claimed before they are sent. Set before the first
|
||||
// event; nil leaves each process to send its own.
|
||||
func (n *Notifier) SetClaimer(c Claimer) {
|
||||
if n != nil {
|
||||
n.claim = c
|
||||
}
|
||||
}
|
||||
|
||||
type hook struct {
|
||||
@@ -199,6 +216,9 @@ func (n *Notifier) run(ctx context.Context) {
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !n.claimed(ctx, e) {
|
||||
continue
|
||||
}
|
||||
for _, h := range n.hooks {
|
||||
if len(h.Events) > 0 && !slices.Contains(h.Events, e.Kind) {
|
||||
continue
|
||||
@@ -295,3 +315,19 @@ func (n *Notifier) send(ctx context.Context, h hook, e Event, attempts int) erro
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// claimed reports whether this process is the one to send an event. A store
|
||||
// that cannot be asked sends it anyway: told twice beats not told.
|
||||
func (n *Notifier) claimed(ctx context.Context, e Event) bool {
|
||||
if n.claim == nil {
|
||||
return true
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
ok, err := n.claim.ClaimNotice(cctx, e.Kind+"\x00"+e.Subject, e.At, dedupe)
|
||||
if err != nil {
|
||||
n.log.Warn("could not ask the store whether a notification was sent; sending it", "kind", e.Kind, "err", err)
|
||||
return true
|
||||
}
|
||||
return ok
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -167,3 +168,57 @@ func TestTestSaysHowEachAnsweredWithoutTheSecretPath(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sharedClaims is a store two control planes share.
|
||||
type sharedClaims struct {
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *sharedClaims) ClaimNotice(_ context.Context, key string, at time.Time, window time.Duration) (bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.err != nil {
|
||||
return false, s.err
|
||||
}
|
||||
if prev, ok := s.last[key]; ok && at.Sub(prev) < window {
|
||||
return false, nil
|
||||
}
|
||||
s.last[key] = at
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func TestTwoControlPlanesSendAnEventOnce(t *testing.T) {
|
||||
// Each control plane serving a team's requests sees its budget spent,
|
||||
// and each kept its own hour of quiet: two behind a balancer sent every
|
||||
// budget event twice.
|
||||
var g got
|
||||
srv := httptest.NewServer(g.handler(0))
|
||||
defer srv.Close()
|
||||
claims := &sharedClaims{last: map[string]time.Time{}}
|
||||
a := newNotifier(t, Config{Host: "ctl1", Webhooks: []Webhook{{URL: srv.URL}}})
|
||||
b := newNotifier(t, Config{Host: "ctl2", Webhooks: []Webhook{{URL: srv.URL}}})
|
||||
a.SetClaimer(claims)
|
||||
b.SetClaimer(claims)
|
||||
a.Notify(BudgetSpent, "team:t1", "spent", nil)
|
||||
b.Notify(BudgetSpent, "team:t1", "spent", nil)
|
||||
b.Notify(BudgetSpent, "team:t2", "spent", nil)
|
||||
a.Wait()
|
||||
b.Wait()
|
||||
if len(g.calls) != 2 {
|
||||
t.Errorf("%d sent, want one for each team", len(g.calls))
|
||||
}
|
||||
|
||||
// A store that cannot be asked: sent, rather than lost.
|
||||
var g2 got
|
||||
srv2 := httptest.NewServer(g2.handler(0))
|
||||
defer srv2.Close()
|
||||
c := newNotifier(t, Config{Webhooks: []Webhook{{URL: srv2.URL}}})
|
||||
c.SetClaimer(&sharedClaims{err: errors.New("store down")})
|
||||
c.Notify(NodeLost, "node3", "lost", nil)
|
||||
c.Wait()
|
||||
if len(g2.calls) != 1 {
|
||||
t.Errorf("with the store down, %d sent, want 1", len(g2.calls))
|
||||
}
|
||||
}
|
||||
@@ -511,3 +511,35 @@ func TestSaturationWithNoUsableEndpoints(t *testing.T) {
|
||||
t.Errorf("an empty pool should report full saturation, got %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestABandForBulkWorkWaitsLongerThanTheRest(t *testing.T) {
|
||||
// A knowledge-base import sends far more than the cluster can take at
|
||||
// once; turned away after the 30 s every band shared, its requests had
|
||||
// to be sent again. A band of its own may wait as long as it likes.
|
||||
s := &fixedSat{}
|
||||
s.set(1.0)
|
||||
cfg := testConfig()
|
||||
cfg.QueueTTL = 100 * time.Millisecond
|
||||
for i := range cfg.Bands {
|
||||
if cfg.Bands[i].Name == "bronze" {
|
||||
cfg.Bands[i].QueueTTL = 600 * time.Millisecond
|
||||
}
|
||||
}
|
||||
c := newController(t, cfg, s.fn)
|
||||
|
||||
start := time.Now()
|
||||
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"}); !errors.Is(err, ErrQueueTimeout) {
|
||||
t.Fatalf("gold: %v", err)
|
||||
}
|
||||
if d := time.Since(start); d > 400*time.Millisecond {
|
||||
t.Errorf("gold waited %v, past the controller's queue_ttl", d)
|
||||
}
|
||||
start = time.Now()
|
||||
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||
if !errors.Is(err, ErrQueueTimeout) || !strings.Contains(err.Error(), "600ms") {
|
||||
t.Fatalf("bronze: %v", err)
|
||||
}
|
||||
if d := time.Since(start); d < 500*time.Millisecond {
|
||||
t.Errorf("bronze gave up after %v, before its own queue_ttl", d)
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,11 @@ type Band struct {
|
||||
// SaturationCeiling is the pool saturation above which this band waits.
|
||||
// 1.0 means "never hold this band back".
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL, set, is how long this band's requests may wait instead of
|
||||
// the controller's: a band for bulk work — a knowledge-base import, a
|
||||
// backfill — whose requests had rather wait ten minutes behind people
|
||||
// than be turned away after thirty seconds.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Config configures the controller.
|
||||
@@ -86,6 +91,9 @@ func (c *Config) normalize() error {
|
||||
if b.SaturationCeiling <= 0 {
|
||||
b.SaturationCeiling = 1
|
||||
}
|
||||
if b.QueueTTL < 0 {
|
||||
return fmt.Errorf("band %q: queue_ttl %v is negative", b.Name, b.QueueTTL)
|
||||
}
|
||||
}
|
||||
if c.DefaultBand == "" {
|
||||
c.DefaultBand = c.Bands[len(c.Bands)-1].Name
|
||||
@@ -279,12 +287,16 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
ErrQueueFull, c.cfg.QueueCapacityTotal)
|
||||
}
|
||||
|
||||
ttl := c.cfg.QueueTTL
|
||||
if band.QueueTTL > 0 {
|
||||
ttl = band.QueueTTL
|
||||
}
|
||||
w := &waiter{
|
||||
tenant: tenantKey(r),
|
||||
model: r.Model,
|
||||
ready: make(chan struct{}),
|
||||
enqueued: time.Now(),
|
||||
deadline: time.Now().Add(c.cfg.QueueTTL),
|
||||
deadline: time.Now().Add(ttl),
|
||||
}
|
||||
c.queues[idx].push(w)
|
||||
c.mu.Unlock()
|
||||
@@ -312,7 +324,7 @@ func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||
|
||||
case <-time.After(time.Until(w.deadline)):
|
||||
c.remove(idx, w, "ttl")
|
||||
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, c.cfg.QueueTTL)
|
||||
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, ttl)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -148,7 +148,7 @@ func (r *Router) serveLines(ctx context.Context, b *store.Batch, lines []batchLi
|
||||
for _, d := range done {
|
||||
served[d.Line] = true
|
||||
}
|
||||
p, perr := r.batchPrincipalOf(ctx, b.KeyID)
|
||||
p, perr := r.principalOfKey(ctx, b.KeyID, r.batchBand())
|
||||
|
||||
var mu sync.Mutex
|
||||
stopWith := ""
|
||||
@@ -221,9 +221,10 @@ func (r *Router) serveLines(ctx context.Context, b *store.Batch, lines []batchLi
|
||||
return stopWith, nil
|
||||
}
|
||||
|
||||
// batchPrincipalOf is the key a batch was made with, as it stands now: a key
|
||||
// revoked since serves no more of it.
|
||||
func (r *Router) batchPrincipalOf(ctx context.Context, keyID string) (auth.Principal, error) {
|
||||
// principalOfKey is the key a batch or a job was made with, as it stands now:
|
||||
// a key revoked since serves no more of it. band is the admission band the
|
||||
// work goes in; empty is the key's own.
|
||||
func (r *Router) principalOfKey(ctx context.Context, keyID, band string) (auth.Principal, error) {
|
||||
k, err := r.store.KeyByID(ctx, keyID)
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
@@ -238,10 +239,13 @@ func (r *Router) batchPrincipalOf(ctx context.Context, keyID string) (auth.Princ
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
}
|
||||
if band == "" {
|
||||
band = eff.Band
|
||||
}
|
||||
return auth.Principal{
|
||||
KeyID: k.ID, KeyPrefix: k.Prefix, UserID: eff.UserID, ProjectID: eff.ProjectID,
|
||||
TeamID: eff.TeamID, OrgID: eff.OrgID,
|
||||
Band: r.batchBand(), RoutingProfile: eff.RoutingProfile, Limits: eff,
|
||||
Band: band, RoutingProfile: eff.RoutingProfile, Limits: eff,
|
||||
}, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -57,6 +57,7 @@ func (r *Router) registerChat(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
web.ReplyRenderer(r.cfg.MaxBodyBytes)(w, req)
|
||||
}))
|
||||
r.registerImagesPage(mux, pages)
|
||||
r.registerMeshesPage(mux, pages)
|
||||
}
|
||||
|
||||
// chatSend dispatches one turn on behalf of a signed-in person.
|
||||
@@ -257,12 +258,12 @@ func (r *Router) principalFor(req *http.Request, user store.User, asked string)
|
||||
// by name, "big-model" came first and every visitor who pressed Send without
|
||||
// touching the picker got placeholder text.
|
||||
func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
return r.modelsOf(req, user, false)
|
||||
return r.modelsOf(req, user, func(m catalog.Model) bool { return !m.MakesImages() && !m.MakesMeshes() })
|
||||
}
|
||||
|
||||
// modelsOf is the models a person may use from a page: those that make
|
||||
// pictures, or those that write text.
|
||||
func (r *Router) modelsOf(req *http.Request, user store.User, pictures bool) []web.ChatModel {
|
||||
func (r *Router) modelsOf(req *http.Request, user store.User, want func(catalog.Model) bool) []web.ChatModel {
|
||||
principal, err := r.principalFor(req, user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -290,7 +291,7 @@ func (r *Router) modelsOf(req *http.Request, user store.User, pictures bool) []w
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if entry.MakesImages() == pictures && r.permits(principal, name) {
|
||||
if want(entry) && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
|
||||
}
|
||||
}
|
||||
@@ -306,7 +307,7 @@ func (r *Router) modelsOf(req *http.Request, user store.User, pictures bool) []w
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if entry.MakesImages() == pictures && r.permits(principal, name) {
|
||||
if want(entry) && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{
|
||||
Name: name, Simulated: entry.Simulated(), OnDemand: true,
|
||||
})
|
||||
|
||||
@@ -84,10 +84,12 @@ func TestARefusalBeforeDispatchSaysWhetherWaitingHelps(t *testing.T) {
|
||||
}{
|
||||
{"queue full", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueFull, time.Now())
|
||||
}, 503, ""},
|
||||
}, 503, "10"},
|
||||
// Busy, not broken: a client told when to come back comes back,
|
||||
// where one told nothing — llmgw-kb among them — gives up.
|
||||
{"queued too long", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrQueueTimeout, time.Now())
|
||||
}, 504, ""},
|
||||
}, 504, "10"},
|
||||
{"displaced for a higher band", func(w http.ResponseWriter) {
|
||||
rt.rejectAdmission(w, &parsedRequest{Model: "m"}, admission.ErrDisplaced, time.Now())
|
||||
}, 503, "5"},
|
||||
|
||||
@@ -62,7 +62,13 @@ func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err
|
||||
r.m.observeRequest(r.metricModel(req.Model), "", 499, time.Since(start), nil, 0)
|
||||
return
|
||||
case errors.Is(err, admission.ErrQueueTimeout):
|
||||
// The cluster was busy for as long as the band may wait, not broken:
|
||||
// said so a client comes back. Without it llmgw-kb, which retries
|
||||
// only what says when to, counted every such passage as failed.
|
||||
status = http.StatusGatewayTimeout
|
||||
w.Header().Set("Retry-After", "10")
|
||||
case errors.Is(err, admission.ErrQueueFull):
|
||||
w.Header().Set("Retry-After", "10")
|
||||
case errors.Is(err, admission.ErrDisplaced):
|
||||
// Turned away for higher-priority work, not for good: said so the
|
||||
// client backs off and comes back rather than giving up.
|
||||
@@ -123,6 +129,12 @@ func (r *Router) Handler() http.Handler {
|
||||
mux.HandleFunc("GET /v1/batches", r.handleListBatches)
|
||||
mux.HandleFunc("GET /v1/batches/{id}", r.handleGetBatch)
|
||||
mux.HandleFunc("POST /v1/batches/{id}/cancel", r.handleCancelBatch)
|
||||
// Jobs: 3D models, which take minutes and are asked after rather than
|
||||
// waited for. The result is a file.
|
||||
mux.HandleFunc("POST "+meshesPath, r.handleCreateJob)
|
||||
mux.HandleFunc("GET "+meshesPath, r.handleListJobs)
|
||||
mux.HandleFunc("GET "+meshesPath+"/{id}", r.handleGetJob)
|
||||
mux.HandleFunc("POST "+meshesPath+"/{id}/cancel", r.handleCancelJob)
|
||||
// Knowledge bases: documents a person's llmgw-kb read and embedded.
|
||||
r.registerKB(mux)
|
||||
// Liveness: this process is up. It says nothing about whether it can
|
||||
@@ -321,7 +333,7 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
exempt := []string{"/health", "/ready", "/leader", "/", "/about", "/_static/", "/_lang/", "/claim/",
|
||||
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/org", "/org/", "/setup/"}
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/3d", "/3d/", "/org", "/org/", "/setup/"}
|
||||
// The docs are public unless the site said otherwise, and then guarded by
|
||||
// the session check in front of them — or, on a gateway with keys and no
|
||||
// accounts to sign in to, by the key like any other path.
|
||||
@@ -751,6 +763,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
|
||||
u.PromptTokens = int64(req.PromptTokens)
|
||||
}
|
||||
u.Images = int64(res.outcome.Images)
|
||||
u.Meshes = int64(res.outcome.Meshes)
|
||||
r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
|
||||
r.acct.Record(req.reservation, u)
|
||||
}
|
||||
@@ -1130,9 +1143,12 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
|
||||
if m.Description != "" {
|
||||
entry["description"] = m.Description
|
||||
}
|
||||
if m.License != nil {
|
||||
entry["license"] = m.License
|
||||
}
|
||||
entry["capabilities"] = map[string]bool{
|
||||
"tools": m.Capabilities.Tools, "vision": m.Capabilities.Vision,
|
||||
"embeddings": m.Capabilities.Embeddings, "images": m.MakesImages(),
|
||||
"embeddings": m.Capabilities.Embeddings, "images": m.MakesImages(), "meshes": m.MakesMeshes(),
|
||||
}
|
||||
data = append(data, entry)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import (
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
@@ -111,7 +112,22 @@ func parseImageRequest(body []byte) (imageRequest, error) {
|
||||
// endpoint may serve as it likes.
|
||||
func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
|
||||
m, known := r.modelMap()[model]
|
||||
if path == meshesPath {
|
||||
if known && !m.MakesMeshes() {
|
||||
if m.MakesImages() {
|
||||
return fmt.Sprintf("%s makes pictures, not 3D models: ask for them at %s", m.Name, imagesPath)
|
||||
}
|
||||
return fmt.Sprintf("%s does not make 3D models", m.Name)
|
||||
}
|
||||
if _, err := parseMeshRequest(body); err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if path == imagesPath {
|
||||
if known && m.MakesMeshes() && !m.MakesImages() {
|
||||
return fmt.Sprintf("%s makes 3D models, not pictures: ask for one at %s, as a job", m.Name, meshesPath)
|
||||
}
|
||||
if known && !m.MakesImages() {
|
||||
return fmt.Sprintf("%s does not make pictures", m.Name)
|
||||
}
|
||||
@@ -121,6 +137,9 @@ func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
|
||||
return ""
|
||||
}
|
||||
if known && m.ServesComfyUI() {
|
||||
if m.MakesMeshes() {
|
||||
return fmt.Sprintf("%s makes 3D models: ask for one at %s, as a job", m.Name, meshesPath)
|
||||
}
|
||||
return fmt.Sprintf("%s makes pictures: ask for them at %s", m.Name, imagesPath)
|
||||
}
|
||||
return ""
|
||||
@@ -130,8 +149,11 @@ func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
|
||||
// proxy would: a failure of this replica that another may not have is
|
||||
// retryable, and nothing is written until the pictures are all in hand.
|
||||
func (r *Router) generateImages(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
|
||||
if req.Path == meshesPath {
|
||||
return r.generateMeshes(ctx, w, req, ep, body)
|
||||
}
|
||||
if req.Path != imagesPath {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s is a ComfyUI server: it answers %s only", ep.ID, imagesPath)
|
||||
return proxy.Outcome{}, fmt.Errorf("%s is a ComfyUI server: it answers %s and %s only", ep.ID, imagesPath, meshesPath)
|
||||
}
|
||||
in, err := parseImageRequest(body)
|
||||
if err != nil {
|
||||
@@ -227,7 +249,7 @@ func (r *Router) registerImagesPage(mux *http.ServeMux, pages *web.AccountPages)
|
||||
if current == "" && !r.personalOff {
|
||||
current = personalContext
|
||||
}
|
||||
web.Images(user, r.modelsOf(req, user, true), contexts, current, version.Version)(w, req)
|
||||
web.Images(user, r.modelsOf(req, user, catalog.Model.MakesImages), contexts, current, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /images/generate", pages.Guard(r.imagesSend))
|
||||
}
|
||||
|
||||
@@ -19,6 +19,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// imageRouter has a ComfyUI model, flux, beside a text model, and returns
|
||||
@@ -244,7 +245,13 @@ func TestThePicturesPageOffersPictureModelsAndTheChatTheOthers(t *testing.T) {
|
||||
user := store.User{ID: "usr-1", ProjectID: project}
|
||||
names := func(pictures bool) []string {
|
||||
var out []string
|
||||
for _, m := range rt.modelsOf(httptest.NewRequest("GET", "/", nil), user, pictures) {
|
||||
var models []web.ChatModel
|
||||
if pictures {
|
||||
models = rt.modelsOf(httptest.NewRequest("GET", "/", nil), user, catalog.Model.MakesImages)
|
||||
} else {
|
||||
models = rt.modelsFor(httptest.NewRequest("GET", "/", nil), user)
|
||||
}
|
||||
for _, m := range models {
|
||||
out = append(out, m.Name)
|
||||
}
|
||||
return out
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Jobs: a request that takes minutes, answered with an id and asked after.
|
||||
//
|
||||
// A 3D model from a picture takes minutes on a card that first has to be
|
||||
// emptied of whatever else was on it. That is not a request to hold a socket
|
||||
// open for (activate.go says why), so it is a job: queued with 202, its
|
||||
// status and progress read back, its result downloaded as a file. The
|
||||
// request is kept as the client sent it and replayed, by whichever gateway
|
||||
// holds the job's lease, through the same path a client's request takes —
|
||||
// key, allowlist, limits, admission, guardrails, accounting — so a job is
|
||||
// charged and judged like any request, tagged job=<id>.
|
||||
|
||||
// meshesPath is where 3D models are asked for. The synchronous path behind
|
||||
// it is served to the job worker only; clients get a job.
|
||||
const meshesPath = "/v1/3d/generations"
|
||||
|
||||
const (
|
||||
purposeJobOutput = "job_output"
|
||||
// jobWindow is how long a job may take, queue included, before it is
|
||||
// given up on.
|
||||
jobWindow = 24 * time.Hour
|
||||
)
|
||||
|
||||
func (r *Router) jobJSON(ctx context.Context, j store.Job) map[string]any {
|
||||
out := map[string]any{
|
||||
"id": j.ID, "object": "job", "endpoint": j.Endpoint, "model": j.Model, "status": j.Status,
|
||||
"progress": nil, "output_file_id": strOrNil(j.OutputFileID), "content_type": strOrNil(j.ContentType),
|
||||
"error": nil, "created_at": j.CreatedAt.Unix(), "started_at": unixOrNil(j.StartedAt),
|
||||
"finished_at": unixOrNil(j.FinishedAt), "expires_at": unixOrNil(j.ExpiresAt), "usage": nil,
|
||||
}
|
||||
if j.Progress != "" && json.Valid([]byte(j.Progress)) {
|
||||
out["progress"] = json.RawMessage(j.Progress)
|
||||
}
|
||||
if j.Error != "" {
|
||||
out["error"] = map[string]string{"message": j.Error}
|
||||
}
|
||||
// The terms what it makes comes under, beside the result: whoever ships
|
||||
// the mesh answers for them.
|
||||
if m, ok := r.modelMap()[j.Model]; ok && m.License != nil {
|
||||
out["license"] = m.License
|
||||
}
|
||||
// What the work cost, from the usage record the replayed request left
|
||||
// under the job's id.
|
||||
if store.JobDone(j.Status) && r.store != nil {
|
||||
if u, ok, err := r.store.UsageByRequest(ctx, j.ID); err == nil && ok {
|
||||
out["usage"] = map[string]int64{"prompt_tokens": u.PromptTokens, "completion_tokens": u.CompletionTokens,
|
||||
"images": u.Images, "meshes": u.Meshes}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// handleCreateJob queues a request for a 3D model. Refused now: no key, a
|
||||
// model that is not there or not allowed, a request the model cannot serve.
|
||||
// Refused when it is served, and said as the job's error: limits, the
|
||||
// guardrails, a model that will not come up.
|
||||
func (r *Router) handleCreateJob(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", "could not read the request: "+err.Error())
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", "the request is not valid JSON: "+err.Error())
|
||||
return
|
||||
}
|
||||
if in.Model == "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", "model is required")
|
||||
return
|
||||
}
|
||||
if _, known := r.modelMap()[in.Model]; !known || r.hiddenModel(in.Model) {
|
||||
writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found",
|
||||
fmt.Sprintf("the model %q does not exist", in.Model))
|
||||
return
|
||||
}
|
||||
if !r.permits(p, in.Model) {
|
||||
auth.WriteForbiddenModel(w, in.Model)
|
||||
return
|
||||
}
|
||||
if why := r.imageRouteRefusal(meshesPath, in.Model, body); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
j := store.Job{ID: newID("job_"), Owner: ownerOf(p), KeyID: p.KeyID, Endpoint: meshesPath, Model: in.Model,
|
||||
Input: body, Status: store.JobQueued, CreatedAt: now, ExpiresAt: now.Add(jobWindow)}
|
||||
if err := r.store.PutJob(req.Context(), j); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
r.wakeJobs()
|
||||
writeJSON(w, http.StatusAccepted, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
|
||||
// ownedJob is a job the caller owns, or a 404 said.
|
||||
func (r *Router) ownedJob(w http.ResponseWriter, req *http.Request, owner string) (store.Job, bool) {
|
||||
j, err := r.store.JobByID(req.Context(), req.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) || err == nil && j.Owner != owner {
|
||||
writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "job_not_found",
|
||||
fmt.Sprintf("no job %q", req.PathValue("id")))
|
||||
return store.Job{}, false
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return store.Job{}, false
|
||||
}
|
||||
return j, true
|
||||
}
|
||||
|
||||
func (r *Router) handleGetJob(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if j, ok := r.ownedJob(w, req, ownerOf(p)); ok {
|
||||
writeJSON(w, http.StatusOK, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Router) handleCancelJob(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
j, ok := r.ownedJob(w, req, ownerOf(p))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
j, err := r.store.CancelJob(req.Context(), j.ID, time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
r.wakeJobs()
|
||||
writeJSON(w, http.StatusOK, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
|
||||
func (r *Router) handleListJobs(w http.ResponseWriter, req *http.Request) {
|
||||
p, ok := r.batchPrincipal(w, req)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
limit, _ := strconv.Atoi(req.URL.Query().Get("limit"))
|
||||
if limit <= 0 || limit > 100 {
|
||||
limit = 20
|
||||
}
|
||||
js, err := r.store.Jobs(req.Context(), ownerOf(p), req.URL.Query().Get("after"), limit+1)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
more := len(js) > limit
|
||||
if more {
|
||||
js = js[:limit]
|
||||
}
|
||||
data := make([]map[string]any, 0, len(js))
|
||||
for _, j := range js {
|
||||
data = append(data, r.jobJSON(req.Context(), j))
|
||||
}
|
||||
out := map[string]any{"object": "list", "data": data, "has_more": more, "first_id": nil, "last_id": nil}
|
||||
if len(js) > 0 {
|
||||
out["first_id"], out["last_id"] = js[0].ID, js[len(js)-1].ID
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// errorMessage is what an answer said went wrong: the message of an OpenAI
|
||||
// error body, or the body itself, kept short.
|
||||
func errorMessage(body []byte) string {
|
||||
var e struct {
|
||||
Error struct {
|
||||
Message string `json:"message"`
|
||||
} `json:"error"`
|
||||
}
|
||||
if json.Unmarshal(body, &e) == nil && e.Error.Message != "" {
|
||||
return e.Error.Message
|
||||
}
|
||||
s := strings.TrimSpace(string(body))
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
if s == "" {
|
||||
return "no reason given"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,285 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// jobComfy is a ComfyUI that makes one picture a workflow, slowly when
|
||||
// asked: stall holds every workflow until it is cleared, and the paths the
|
||||
// router took to stop one are counted.
|
||||
type jobComfy struct {
|
||||
srv *httptest.Server
|
||||
stall atomic.Bool
|
||||
prompts atomic.Int32
|
||||
cancelled atomic.Int32
|
||||
}
|
||||
|
||||
func newJobComfy(t *testing.T) *jobComfy {
|
||||
t.Helper()
|
||||
c := &jobComfy{}
|
||||
c.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
case r.URL.Path == "/prompt":
|
||||
c.prompts.Add(1)
|
||||
fmt.Fprint(w, `{"prompt_id":"p1"}`)
|
||||
case r.URL.Path == "/history/p1":
|
||||
if c.stall.Load() {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true},"outputs":{"7":{"images":[{"filename":"0.png","type":"temp"}]}}}}`)
|
||||
case r.URL.Path == "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png-bytes")
|
||||
case r.URL.Path == "/queue" && r.Method == http.MethodPost, r.URL.Path == "/interrupt":
|
||||
c.cancelled.Add(1)
|
||||
fmt.Fprint(w, `{}`)
|
||||
case r.URL.Path == "/queue":
|
||||
fmt.Fprint(w, `{"queue_running":[[0,"p1"]],"queue_pending":[]}`)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(c.srv.Close)
|
||||
return c
|
||||
}
|
||||
|
||||
// jobRouter is a gateway with the store, a key, a text model and a ComfyUI
|
||||
// model priced by the picture.
|
||||
func jobRouter(t *testing.T, c *jobComfy) (*Router, string) {
|
||||
t.Helper()
|
||||
cfg := baseConfig()
|
||||
cfg.Models = []catalog.Model{catModel("chat", nil), {Name: "flux", Source: "external",
|
||||
External: &catalog.External{URL: c.srv.URL, Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}}}
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"flux": {PerImage: 5000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:flux", URL: c.srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "platform")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "gateway")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("a", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-1", Hash: sum[:], Prefix: "aaaa", Name: "test",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rt, secret
|
||||
}
|
||||
|
||||
// queueJob puts a job in the store as the create endpoint would, for an
|
||||
// endpoint that serves today: until a model makes 3D models, the worker is
|
||||
// tried with pictures.
|
||||
func queueJob(t *testing.T, rt *Router, id, status string, body string) store.Job {
|
||||
t.Helper()
|
||||
now := time.Now()
|
||||
j := store.Job{ID: id, Owner: "project:" + mustProjectOf(t, rt, "key-1"), KeyID: "key-1", Endpoint: imagesPath,
|
||||
Model: "flux", Input: []byte(body), Status: status, CreatedAt: now, ExpiresAt: now.Add(jobWindow)}
|
||||
if err := rt.store.PutJob(t.Context(), j); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return j
|
||||
}
|
||||
|
||||
func waitJob(t *testing.T, h http.Handler, secret, id string, until func(map[string]any) bool) map[string]any {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for {
|
||||
_, j, raw := call(t, h, secret, http.MethodGet, meshesPath+"/"+id, nil)
|
||||
if until(j) {
|
||||
return j
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the job did not get there: %s", raw)
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func jobDone(j map[string]any) bool { return store.JobDone(fmt.Sprint(j["status"])) }
|
||||
|
||||
func TestAJobIsServedItsResultKeptAndItsWorkCharged(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
j := queueJob(t, rt, "job_1", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
rt.wakeJobs()
|
||||
|
||||
got := waitJob(t, h, secret, j.ID, jobDone)
|
||||
if got["status"] != "completed" || got["content_type"] != "application/json" || got["error"] != nil {
|
||||
t.Fatalf("%v", got)
|
||||
}
|
||||
if got["started_at"] == nil || got["finished_at"] == nil {
|
||||
t.Errorf("times: %v", got)
|
||||
}
|
||||
code, _, out := call(t, h, secret, http.MethodGet, "/v1/files/"+got["output_file_id"].(string)+"/content", nil)
|
||||
if code != 200 || !strings.Contains(out, `"b64_json":"cG5nLWJ5dGVz"`) {
|
||||
t.Errorf("result %d: %s", code, out)
|
||||
}
|
||||
_, f, _ := call(t, h, secret, http.MethodGet, "/v1/files/"+got["output_file_id"].(string), nil)
|
||||
if f["purpose"] != purposeJobOutput || f["filename"] != "job_1.json" || f["expires_at"] == nil {
|
||||
t.Errorf("file %v", f)
|
||||
}
|
||||
|
||||
// Charged like any request, by the picture, and findable by the job.
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, _ := rt.store.UsageByRequest(t.Context(), j.ID)
|
||||
if ok {
|
||||
if u.Tags["job"] != j.ID || u.Images != 1 || u.CostMicros != 5000 {
|
||||
t.Errorf("usage %+v", u)
|
||||
}
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the job left no usage")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
got = waitJob(t, h, secret, j.ID, func(j map[string]any) bool { return j["usage"] != nil })
|
||||
if u := got["usage"].(map[string]any); u["images"] != 1.0 {
|
||||
t.Errorf("usage shown %v", u)
|
||||
}
|
||||
_, list, _ := call(t, h, secret, http.MethodGet, meshesPath, nil)
|
||||
if d := list["data"].([]any); len(d) != 1 {
|
||||
t.Errorf("listed %v", list)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobIsRefusedBeforeItIsQueuedWhenItCannotBeServed(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
code, _, raw := call(t, h, secret, http.MethodPost, meshesPath, map[string]any{"model": "chat", "prompt": "a cube"})
|
||||
if code != 400 || !strings.Contains(raw, "does not make 3D models") {
|
||||
t.Errorf("text model: %d %s", code, raw)
|
||||
}
|
||||
code, _, raw = call(t, h, secret, http.MethodPost, meshesPath, map[string]any{"model": "nope", "prompt": "a cube"})
|
||||
if code != 404 || !strings.Contains(raw, "model_not_found") {
|
||||
t.Errorf("unknown model: %d %s", code, raw)
|
||||
}
|
||||
code, _, raw = call(t, h, secret, http.MethodPost, meshesPath, map[string]any{"prompt": "a cube"})
|
||||
if code != 400 || !strings.Contains(raw, "model is required") {
|
||||
t.Errorf("no model: %d %s", code, raw)
|
||||
}
|
||||
if code, _, _ := call(t, h, "", http.MethodPost, meshesPath, map[string]any{"model": "chat"}); code != 401 {
|
||||
t.Errorf("no key: %d", code)
|
||||
}
|
||||
if _, list, _ := call(t, h, secret, http.MethodGet, meshesPath, nil); len(list["data"].([]any)) != 0 {
|
||||
t.Errorf("a refused job was queued: %v", list)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARunningJobIsCancelledAndTheEngineToldToStop(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
c.stall.Store(true)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
j := queueJob(t, rt, "job_slow", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
rt.wakeJobs()
|
||||
waitJob(t, h, secret, j.ID, func(j map[string]any) bool { return j["status"] == "running" })
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for c.prompts.Load() == 0 && time.Now().Before(deadline) {
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
|
||||
code, got, raw := call(t, h, secret, http.MethodPost, meshesPath+"/"+j.ID+"/cancel", nil)
|
||||
if code != 200 || got["status"] != "cancelling" {
|
||||
t.Fatalf("cancel: %d %s", code, raw)
|
||||
}
|
||||
got = waitJob(t, h, secret, j.ID, jobDone)
|
||||
if got["status"] != "cancelled" || got["output_file_id"] != nil {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
if c.cancelled.Load() == 0 {
|
||||
t.Error("ComfyUI was not told to stop")
|
||||
}
|
||||
|
||||
// One not yet taken ends at once, and is never sent.
|
||||
before := c.prompts.Load()
|
||||
j2 := queueJob(t, rt, "job_never", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
_, got, _ = call(t, h, secret, http.MethodPost, meshesPath+"/"+j2.ID+"/cancel", nil)
|
||||
if got["status"] != "cancelled" {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
rt.wakeJobs()
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
if c.prompts.Load() != before {
|
||||
t.Error("a cancelled job was sent")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobWhoseGatewayDiedIsTakenByAnother(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
// As a gateway that died mid-job left it: running, its lease run out.
|
||||
j := queueJob(t, rt, "job_orphan", store.JobRunning, `{"model":"flux","prompt":"a cube"}`)
|
||||
j.StartedAt = time.Now().Add(-time.Minute)
|
||||
_ = rt.store.UpdateJob(t.Context(), j)
|
||||
rt.wakeJobs()
|
||||
if got := waitJob(t, h, secret, j.ID, jobDone); got["status"] != "completed" {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
|
||||
// A refusal of the request is the job's end, said.
|
||||
bad := queueJob(t, rt, "job_bad", store.JobQueued, `{"model":"flux","prompt":"a cube","n":99}`)
|
||||
rt.wakeJobs()
|
||||
got := waitJob(t, h, secret, bad.ID, jobDone)
|
||||
if got["status"] != "failed" || !strings.Contains(fmt.Sprint(got["error"]), "n is between 1 and 4") {
|
||||
t.Errorf("%v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobsAreTheirOwners(t *testing.T) {
|
||||
c := newJobComfy(t)
|
||||
rt, secret := jobRouter(t, c)
|
||||
h := rt.Handler()
|
||||
j := queueJob(t, rt, "job_mine", store.JobQueued, `{"model":"flux","prompt":"a cube"}`)
|
||||
rt.wakeJobs()
|
||||
waitJob(t, h, secret, j.ID, jobDone)
|
||||
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "other")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "t")
|
||||
project, _ := rt.store.CreateProject(t.Context(), team, "p")
|
||||
other := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(other))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "other",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if code, _, _ := call(t, h, other, http.MethodGet, meshesPath+"/"+j.ID, nil); code != 404 {
|
||||
t.Errorf("another project read a job: %d", code)
|
||||
}
|
||||
if code, _, _ := call(t, h, other, http.MethodPost, meshesPath+"/"+j.ID+"/cancel", nil); code != 404 {
|
||||
t.Errorf("another project cancelled a job: %d", code)
|
||||
}
|
||||
_, mine, _ := call(t, h, secret, http.MethodGet, meshesPath+"/"+j.ID, nil)
|
||||
if code, _, _ := call(t, h, other, http.MethodGet, "/v1/files/"+mine["output_file_id"].(string)+"/content", nil); code != 404 {
|
||||
t.Errorf("another project read a job's result: %d", code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,264 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The worker that serves jobs. Any gateway on the store may run one: a job
|
||||
// is worked on by whichever holds its lease, and one whose gateway died is
|
||||
// taken by another when the lease runs out.
|
||||
|
||||
const (
|
||||
jobLease = 2 * time.Minute
|
||||
jobPoll = 5 * time.Second
|
||||
// jobCancelCheck is how often a running job looks for a cancel.
|
||||
jobCancelCheck = 2 * time.Second
|
||||
// jobMaxResult bounds what one job may answer: a mesh, not a film.
|
||||
jobMaxResult = 256 << 20
|
||||
)
|
||||
|
||||
// wakeJobs has the worker look for work now rather than at its next poll.
|
||||
func (r *Router) wakeJobs() {
|
||||
select {
|
||||
case r.jobWake <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Router) runJobs(ctx context.Context) {
|
||||
host, _ := os.Hostname()
|
||||
owner := fmt.Sprintf("%s:%d:%s", host, os.Getpid(), newID(""))
|
||||
t := time.NewTicker(jobPoll)
|
||||
defer t.Stop()
|
||||
for {
|
||||
for {
|
||||
now := time.Now()
|
||||
j, ok, err := r.store.ClaimJob(ctx, owner, now, now.Add(jobLease))
|
||||
if err != nil {
|
||||
if ctx.Err() == nil {
|
||||
r.log.Warn("could not look for jobs to serve", "err", err)
|
||||
}
|
||||
break
|
||||
}
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
r.workJob(ctx, owner, j)
|
||||
}
|
||||
// A job's record outlives its file by nothing: both go at the
|
||||
// retention.
|
||||
_, _, _, days := r.batchCfg()
|
||||
if n, err := r.store.DeleteJobsBefore(ctx, time.Now().AddDate(0, 0, -days)); err == nil && n > 0 {
|
||||
r.log.Info("jobs past their retention forgotten", "jobs", n)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-r.jobWake:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// workJob takes one job as far as it goes, holding its lease the while.
|
||||
func (r *Router) workJob(ctx context.Context, owner string, j store.Job) {
|
||||
jctx, stop := context.WithCancel(ctx)
|
||||
defer stop()
|
||||
defer func() { _ = r.store.ReleaseJob(context.WithoutCancel(ctx), j.ID, owner) }()
|
||||
// The lease kept while working; lost, the work stops and another
|
||||
// gateway has it.
|
||||
go func() {
|
||||
t := time.NewTicker(jobLease / 4)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-jctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if held, err := r.store.RenewJobLease(jctx, j.ID, owner, time.Now().Add(jobLease)); err == nil && !held {
|
||||
stop()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
if err := r.serveJob(jctx, &j); err != nil && jctx.Err() == nil {
|
||||
r.log.Warn("a job stopped with an error; it is picked up again", "job", j.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// serveJob replays the job's request and keeps what came back. A failure
|
||||
// of the gateway's own (the store unreachable) is returned, and the job is
|
||||
// picked up again; a refusal of the request is the job's end.
|
||||
func (r *Router) serveJob(ctx context.Context, j *store.Job) error {
|
||||
now := time.Now()
|
||||
switch {
|
||||
case j.Status == store.JobCancelling:
|
||||
j.Status, j.FinishedAt = store.JobCancelled, now
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
case now.After(j.ExpiresAt):
|
||||
j.Status, j.FinishedAt = store.JobExpired, now
|
||||
j.Error = fmt.Sprintf("not finished within %s", jobWindow)
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
if j.Status == store.JobQueued {
|
||||
j.Status, j.StartedAt = store.JobRunning, now
|
||||
if err := r.store.UpdateJob(ctx, *j); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
// A job found running was another gateway's, which died with it. It is
|
||||
// run again from the start: the engine's queue does not outlive the
|
||||
// request that was waiting on it.
|
||||
fail := func(msg string) error {
|
||||
j.Status, j.Error, j.FinishedAt = store.JobFailed, msg, time.Now()
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
p, err := r.principalOfJob(ctx, *j)
|
||||
if err != nil {
|
||||
return fail("the key or account this job was made with can no longer be used: " + err.Error())
|
||||
}
|
||||
|
||||
// A cancel asked for while it runs ends the request, which takes the
|
||||
// work out of the engine's queue.
|
||||
rctx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
go func() {
|
||||
t := time.NewTicker(jobCancelCheck)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-rctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if cur, err := r.store.JobByID(rctx, j.ID); err == nil && cur.Status == store.JobCancelling {
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
|
||||
handler := r.handleInference(j.Endpoint)
|
||||
for attempt := 1; ; attempt++ {
|
||||
hreq, err := http.NewRequestWithContext(auth.WithPrincipal(rctx, p), http.MethodPost, j.Endpoint, bytes.NewReader(j.Input))
|
||||
if err != nil {
|
||||
return fail(err.Error())
|
||||
}
|
||||
hreq.Header.Set("Content-Type", "application/json")
|
||||
hreq.Header.Set("X-Request-Id", j.ID)
|
||||
hreq.Header.Set(tagsHeader, "job="+j.ID)
|
||||
w := &bufferWriter{header: http.Header{}, max: jobMaxResult}
|
||||
handler(w, hreq)
|
||||
if rctx.Err() != nil {
|
||||
if ctx.Err() != nil {
|
||||
// The lease was lost or the gateway is stopping: another
|
||||
// gateway's to pick up.
|
||||
return ctx.Err()
|
||||
}
|
||||
j.Status, j.FinishedAt = store.JobCancelled, time.Now()
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
status := w.status
|
||||
if status == 0 {
|
||||
status = http.StatusOK
|
||||
}
|
||||
// Waited out, for as long as the job has: a model loading, a rate
|
||||
// limit, a queue full. Nobody is holding a connection, which is the
|
||||
// point of a job. Not a spent budget, which says no Retry-After
|
||||
// because waiting does not help.
|
||||
if wait := w.header.Get("Retry-After"); wait != "" && (status == http.StatusTooManyRequests || status >= 500) &&
|
||||
time.Now().Before(j.ExpiresAt) {
|
||||
secs, _ := strconv.Atoi(wait)
|
||||
select {
|
||||
case <-rctx.Done():
|
||||
continue // said above
|
||||
case <-time.After(time.Duration(max(secs, 1)) * time.Second):
|
||||
}
|
||||
continue
|
||||
}
|
||||
if status >= 300 {
|
||||
return fail(errorMessage(w.buf.Bytes()))
|
||||
}
|
||||
ct := w.header.Get("Content-Type")
|
||||
if ct == "" {
|
||||
ct = http.DetectContentType(w.buf.Bytes())
|
||||
}
|
||||
_, _, _, days := r.batchCfg()
|
||||
done := time.Now()
|
||||
f := store.File{ID: newID("file-"), Owner: j.Owner, Purpose: purposeJobOutput,
|
||||
Filename: j.ID + extensionFor(ct), Bytes: int64(w.buf.Len()), CreatedAt: done, ExpiresAt: done.AddDate(0, 0, days)}
|
||||
if err := r.store.PutFile(ctx, f, w.buf.Bytes()); err != nil {
|
||||
return err
|
||||
}
|
||||
// Said done while it may still be said: the last word heard was the
|
||||
// node running, and a finished job saying it is running reads as
|
||||
// one that stopped halfway.
|
||||
_ = r.store.SetJobProgress(ctx, j.ID, `{"state":"done","percent":100}`, "")
|
||||
j.Status, j.OutputFileID, j.ContentType, j.FinishedAt = store.JobCompleted, f.ID, ct, done
|
||||
return r.store.UpdateJob(ctx, *j)
|
||||
}
|
||||
}
|
||||
|
||||
// extensionFor names a result's file by what it is.
|
||||
func extensionFor(contentType string) string {
|
||||
ct, _, _ := strings.Cut(contentType, ";")
|
||||
switch strings.TrimSpace(ct) {
|
||||
case "application/json":
|
||||
return ".json"
|
||||
case "model/gltf-binary":
|
||||
return ".glb"
|
||||
case "model/gltf+json":
|
||||
return ".gltf"
|
||||
case "model/obj":
|
||||
return ".obj"
|
||||
case "image/png":
|
||||
return ".png"
|
||||
case "image/jpeg":
|
||||
return ".jpg"
|
||||
case "image/webp":
|
||||
return ".webp"
|
||||
}
|
||||
return ".bin"
|
||||
}
|
||||
|
||||
// principalOfJob is who a job acts for, as they stand now: the key it was
|
||||
// made with, or — for one made from the portal — the person whose session
|
||||
// made it, while they may still use the gateway and the project the job is
|
||||
// charged to.
|
||||
func (r *Router) principalOfJob(ctx context.Context, j store.Job) (auth.Principal, error) {
|
||||
uid, session := strings.CutPrefix(j.KeyID, "session:")
|
||||
if !session {
|
||||
return r.principalOfKey(ctx, j.KeyID, "")
|
||||
}
|
||||
u, err := r.store.UserByID(ctx, uid)
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
}
|
||||
if u.State != store.UserActive {
|
||||
return auth.Principal{}, fmt.Errorf("the account that made it is %s", u.State)
|
||||
}
|
||||
project, _ := strings.CutPrefix(j.Owner, "project:")
|
||||
if !strings.HasPrefix(j.Owner, "project:") {
|
||||
project = ""
|
||||
} else if ok, err := r.store.CanUseProject(ctx, uid, project); err != nil || !ok {
|
||||
return auth.Principal{}, errors.New("its account is no longer a member of the project it is charged to")
|
||||
}
|
||||
eff, err := r.store.ResolveFor(ctx, j.KeyID, uid, project)
|
||||
if err != nil {
|
||||
return auth.Principal{}, err
|
||||
}
|
||||
return auth.Principal{KeyID: j.KeyID, KeyPrefix: "session", UserID: uid, ProjectID: eff.ProjectID,
|
||||
TeamID: eff.TeamID, OrgID: eff.OrgID, Band: eff.Band, RoutingProfile: eff.RoutingProfile, Limits: eff}, nil
|
||||
}
|
||||
@@ -3,7 +3,6 @@ package router
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
|
||||
@@ -60,6 +59,9 @@ type bufferWriter struct {
|
||||
header http.Header
|
||||
status int
|
||||
buf bytes.Buffer
|
||||
// max bounds what is kept; 1 MiB when unset, which is a judge's answer
|
||||
// many times over. A job's result is a file, and sets more.
|
||||
max int
|
||||
}
|
||||
|
||||
func (b *bufferWriter) Header() http.Header { return b.header }
|
||||
@@ -72,8 +74,12 @@ func (b *bufferWriter) Write(p []byte) (int, error) {
|
||||
if b.status == 0 {
|
||||
b.status = http.StatusOK
|
||||
}
|
||||
if b.buf.Len()+len(p) > 1<<20 {
|
||||
return 0, errors.New("a judge's answer over 1 MiB")
|
||||
limit := b.max
|
||||
if limit <= 0 {
|
||||
limit = 1 << 20
|
||||
}
|
||||
if b.buf.Len()+len(p) > limit {
|
||||
return 0, fmt.Errorf("an answer over %d bytes", limit)
|
||||
}
|
||||
return b.buf.Write(p)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,383 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// 3D models from a picture, made by ComfyUI. Clients ask for them as a job
|
||||
// (jobs.go); the job worker replays the request through the ordinary path,
|
||||
// and at a ComfyUI replica it arrives here: the picture is put in ComfyUI's
|
||||
// input directory, the model's workflow is filled in with its name and run,
|
||||
// and the mesh it saves comes back as the answer — the file the job keeps.
|
||||
|
||||
// meshRequest is what a request for a 3D model asks. The picture is the
|
||||
// whole of the prompt: base64, or a data URL.
|
||||
type meshRequest struct {
|
||||
Model string `json:"model"`
|
||||
Image string `json:"image"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
// Resolution is the model's own grid, where it has a choice: TripoSR's
|
||||
// marching cubes take 256 by default, TRELLIS.2 512 to 1536. Zero leaves
|
||||
// it to the workflow.
|
||||
Resolution int `json:"resolution"`
|
||||
|
||||
picture []byte
|
||||
ext string
|
||||
}
|
||||
|
||||
// maxPicture bounds the picture a mesh is made from.
|
||||
const maxPicture = 16 << 20
|
||||
|
||||
func parseMeshRequest(body []byte) (meshRequest, error) {
|
||||
var in meshRequest
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
return in, err
|
||||
}
|
||||
s := strings.TrimSpace(in.Image)
|
||||
if s == "" {
|
||||
return in, errors.New("image is required: the picture to make a 3D model of, base64 or a data URL")
|
||||
}
|
||||
if strings.HasPrefix(s, "data:") {
|
||||
_, after, ok := strings.Cut(s, ";base64,")
|
||||
if !ok {
|
||||
return in, errors.New("image is a data URL without base64 in it")
|
||||
}
|
||||
s = after
|
||||
}
|
||||
if base64.StdEncoding.DecodedLen(len(s)) > maxPicture+3 {
|
||||
return in, fmt.Errorf("image is past %d MB", maxPicture>>20)
|
||||
}
|
||||
b, err := base64.StdEncoding.DecodeString(s)
|
||||
if err != nil {
|
||||
return in, errors.New("image is not base64: " + err.Error())
|
||||
}
|
||||
switch http.DetectContentType(b) {
|
||||
case "image/png":
|
||||
in.ext = ".png"
|
||||
case "image/jpeg":
|
||||
in.ext = ".jpg"
|
||||
case "image/webp":
|
||||
in.ext = ".webp"
|
||||
default:
|
||||
return in, errors.New("image is not a PNG, JPEG or WebP picture")
|
||||
}
|
||||
in.picture = b
|
||||
if in.Steps < 0 || in.Steps > maxSteps {
|
||||
return in, fmt.Errorf("steps is between 1 and %d", maxSteps)
|
||||
}
|
||||
if r := in.Resolution; r != 0 && (r < 64 || r > 2048 || r%32 != 0) {
|
||||
return in, fmt.Errorf("resolution %d is not a multiple of 32 from 64 to 2048", r)
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// generateMeshes answers a request for a 3D model at a ComfyUI replica. It
|
||||
// writes the mesh itself, with its type, which is what the job keeps.
|
||||
func (r *Router) generateMeshes(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
|
||||
in, err := parseMeshRequest(body)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
tpl, err := comfy.Load(ep.Workflow)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
if t := r.cfg.Timeouts.Total; t > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, t)
|
||||
defer cancel()
|
||||
}
|
||||
start := time.Now()
|
||||
fail := func(err error) (proxy.Outcome, error) {
|
||||
var ce *comfy.Error
|
||||
switch {
|
||||
case errors.As(err, &ce):
|
||||
return proxy.Outcome{Retryable: ce.Retryable}, err
|
||||
case errors.Is(err, context.Canceled) && ctxCauseIsClient(ctx):
|
||||
return proxy.Outcome{}, proxy.ErrClientGone
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
return proxy.Outcome{}, fmt.Errorf("%s made no 3D model within %s", ep.ID, r.cfg.Timeouts.Total)
|
||||
}
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
// A name of its own: two requests' pictures must not overwrite each
|
||||
// other in the replica's input directory.
|
||||
rnd := make([]byte, 8)
|
||||
_, _ = rand.Read(rnd)
|
||||
name, err := r.comfy.Upload(ctx, ep.URL, ep.APIKey, "llmgw-"+hex.EncodeToString(rnd)+in.ext, in.picture)
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
seed := comfy.RandomSeed()
|
||||
if in.Seed != nil {
|
||||
seed = *in.Seed
|
||||
}
|
||||
var upstream struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &upstream)
|
||||
p := comfy.Params{Checkpoint: upstream.Model, Image: name, Seed: seed, Steps: in.Steps, Extra: map[string]any{}}
|
||||
if in.Resolution > 0 {
|
||||
p.Extra["resolution"] = in.Resolution
|
||||
}
|
||||
workflow, err := tpl.Fill(p)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
files, err := r.comfy.GenerateWatch(ctx, ep.URL, ep.APIKey, workflow, r.jobProgress(ctx, req))
|
||||
if err != nil {
|
||||
return fail(err)
|
||||
}
|
||||
var mesh *comfy.Image
|
||||
for i := range files {
|
||||
if isMeshType(files[i].ContentType) {
|
||||
mesh = &files[i]
|
||||
break
|
||||
}
|
||||
}
|
||||
if mesh == nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: the workflow saved no mesh (it saved %d other files): "+
|
||||
"its last node should save a .glb", ep.ID, len(files))
|
||||
}
|
||||
data := mesh.Data
|
||||
if mesh.ContentType == "model/obj" {
|
||||
glb, err := objToGLB(data)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: the mesh it saved: %w", ep.ID, err)
|
||||
}
|
||||
data, mesh.ContentType = glb, "model/gltf-binary"
|
||||
}
|
||||
m := r.modelMap()[req.Model]
|
||||
if m.License != nil && mesh.ContentType == "model/gltf-binary" {
|
||||
if with, err := glbWithLicense(data, *m.License); err == nil {
|
||||
data = with
|
||||
} else {
|
||||
// Refused rather than sent without it: a mesh travels without
|
||||
// the API, and the notice is what the terms ask to travel too.
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: the licence could not be written into the mesh: %w", ep.ID, err)
|
||||
}
|
||||
}
|
||||
w.Header().Set("x-llmgw-seed", strconv.FormatInt(seed, 10))
|
||||
if m.License != nil {
|
||||
w.Header().Set("x-llmgw-license", m.License.Name)
|
||||
}
|
||||
w.Header().Set("Content-Type", mesh.ContentType)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(data)
|
||||
took := time.Since(start)
|
||||
return proxy.Outcome{Wrote: true, Status: http.StatusOK, Bytes: int64(len(data)), TTFT: took, Decode: took,
|
||||
Meshes: 1}, nil
|
||||
}
|
||||
|
||||
func isMeshType(ct string) bool {
|
||||
return strings.HasPrefix(ct, "model/") || ct == "application/ply"
|
||||
}
|
||||
|
||||
// glbWithLicense is a GLB with the licence in asset.extras.license, where a
|
||||
// file's terms travel with it: glTF's own place for what an application
|
||||
// adds, kept by every tool that loads and saves the file.
|
||||
func glbWithLicense(glb []byte, lic catalog.License) ([]byte, error) {
|
||||
if len(glb) < 20 || string(glb[:4]) != "glTF" || binary.LittleEndian.Uint32(glb[4:]) != 2 {
|
||||
return nil, errors.New("not a glTF 2 binary")
|
||||
}
|
||||
n := int(binary.LittleEndian.Uint32(glb[12:]))
|
||||
if string(glb[16:20]) != "JSON" || 20+n > len(glb) {
|
||||
return nil, errors.New("its first chunk is not its JSON")
|
||||
}
|
||||
var doc map[string]any
|
||||
d := json.NewDecoder(bytes.NewReader(glb[20 : 20+n]))
|
||||
d.UseNumber() // every number as it was written
|
||||
if err := d.Decode(&doc); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
asset, _ := doc["asset"].(map[string]any)
|
||||
if asset == nil {
|
||||
asset = map[string]any{"version": "2.0"}
|
||||
}
|
||||
extras, _ := asset["extras"].(map[string]any)
|
||||
if extras == nil {
|
||||
extras = map[string]any{}
|
||||
}
|
||||
l := map[string]any{"name": lic.Name}
|
||||
if lic.URL != "" {
|
||||
l["url"] = lic.URL
|
||||
}
|
||||
if lic.Notice != "" {
|
||||
l["notice"] = lic.Notice
|
||||
}
|
||||
extras["license"] = l
|
||||
asset["extras"] = extras
|
||||
doc["asset"] = asset
|
||||
js, err := json.Marshal(doc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// A chunk is padded to four bytes, the JSON one with spaces.
|
||||
for len(js)%4 != 0 {
|
||||
js = append(js, ' ')
|
||||
}
|
||||
rest := glb[20+n:]
|
||||
out := make([]byte, 0, 20+len(js)+len(rest))
|
||||
out = append(out, "glTF"...)
|
||||
out = binary.LittleEndian.AppendUint32(out, 2)
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(20+len(js)+len(rest))) //nolint:gosec // bounded by the file read
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(len(js))) //nolint:gosec // bounded by the file read
|
||||
out = append(out, "JSON"...)
|
||||
out = append(out, js...)
|
||||
return append(out, rest...), nil
|
||||
}
|
||||
|
||||
// jobProgress is where a job's request says how far along it is: into the
|
||||
// job, for whoever asks after it. Nil for a request that is no job's — and
|
||||
// for one that only says it is: the job must be the same key's, so nobody
|
||||
// writes into another's progress by naming it in a tag.
|
||||
func (r *Router) jobProgress(ctx context.Context, req *parsedRequest) func(comfy.Progress) {
|
||||
id := req.Tags["job"]
|
||||
if id == "" || r.store == nil || id != req.RequestID {
|
||||
return nil
|
||||
}
|
||||
j, err := r.store.JobByID(ctx, id)
|
||||
if err != nil || j.KeyID != req.KeyID {
|
||||
return nil
|
||||
}
|
||||
return func(p comfy.Progress) {
|
||||
b, err := json.Marshal(p)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if err := r.store.SetJobProgress(context.WithoutCancel(ctx), id, string(b), p.PromptID); err != nil {
|
||||
r.log.Warn("a job's progress could not be kept", "job", id, "err", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// registerMeshesPage mounts the page that makes 3D models, beside the
|
||||
// pictures page and for the same reasons: a session, never a key, and the
|
||||
// request sent down the path an API client's takes — here, a job.
|
||||
func (r *Router) registerMeshesPage(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
pages.SetMeshes(func() bool {
|
||||
for _, m := range r.modelMap() {
|
||||
if m.MakesMeshes() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})
|
||||
mux.HandleFunc("GET /3d", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
contexts, _ := r.chatContexts(req.Context(), user)
|
||||
current, err := r.pickContext(req.Context(), user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
current = ""
|
||||
}
|
||||
if current == "" && !r.personalOff {
|
||||
current = personalContext
|
||||
}
|
||||
web.Meshes(user, r.modelsOf(req, user, catalog.Model.MakesMeshes), contexts, current, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /3d/generate", pages.Guard(r.meshesSend))
|
||||
mux.HandleFunc("GET /3d/jobs/{id}", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
if next, ok := r.asPerson(w, req, user, req.URL.Query().Get("context")); ok {
|
||||
r.handleGetJob(w, next)
|
||||
}
|
||||
}))
|
||||
// The mesh itself, from where the page is, for its viewer to load.
|
||||
mux.HandleFunc("GET /3d/jobs/{id}/file", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
next, ok := r.asPerson(w, req, user, req.URL.Query().Get("context"))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
p, _ := auth.FromContext(next.Context())
|
||||
j, ok := r.ownedJob(w, next, ownerOf(p))
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if j.OutputFileID == "" {
|
||||
http.Error(w, "the job has made nothing yet", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
b, err := r.store.FileContent(req.Context(), j.OutputFileID)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", j.ContentType)
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf("inline; filename=%q", j.ID+".glb"))
|
||||
_, _ = w.Write(b)
|
||||
}))
|
||||
}
|
||||
|
||||
// asPerson is a page's request as the signed-in person's, in the place they
|
||||
// chose: what the API handlers read a key's principal from.
|
||||
func (r *Router) asPerson(w http.ResponseWriter, req *http.Request, user store.User, context string) (*http.Request, bool) {
|
||||
p, err := r.principalFor(req, user, context)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return nil, false
|
||||
}
|
||||
return req.WithContext(auth.WithPrincipal(req.Context(), p)), true
|
||||
}
|
||||
|
||||
// meshesSend asks for a 3D model on behalf of a signed-in person.
|
||||
func (r *Router) meshesSend(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
http.Error(w, "could not read the request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
Image string `json:"image"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Resolution int `json:"resolution"`
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil || in.Model == "" {
|
||||
http.Error(w, "a model and a picture are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
// Rebuilt from the fields the page has, so it can set nothing else.
|
||||
out := map[string]any{"model": in.Model, "image": in.Image}
|
||||
if in.Seed != nil {
|
||||
out["seed"] = *in.Seed
|
||||
}
|
||||
if in.Resolution > 0 {
|
||||
out["resolution"] = in.Resolution
|
||||
}
|
||||
upstream, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
http.Error(w, "could not build the request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
next, ok := r.asPerson(w, req, user, in.Context)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
next = next.Clone(next.Context())
|
||||
next.Body = io.NopCloser(bytes.NewReader(upstream))
|
||||
next.ContentLength = int64(len(upstream))
|
||||
next.Header.Set("Content-Type", "application/json")
|
||||
r.handleCreateJob(w, next)
|
||||
}
|
||||
@@ -0,0 +1,363 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"image"
|
||||
"image/png"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// testGLB is the smallest glTF binary: an asset, and a binary chunk after
|
||||
// its JSON that must come through untouched.
|
||||
func testGLB(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
js := []byte(`{"asset":{"version":"2.0","generator":"test"},"buffers":[{"byteLength":4}]}`)
|
||||
for len(js)%4 != 0 {
|
||||
js = append(js, ' ')
|
||||
}
|
||||
bin := []byte{1, 2, 3, 4}
|
||||
var b []byte
|
||||
b = append(b, "glTF"...)
|
||||
b = binary.LittleEndian.AppendUint32(b, 2)
|
||||
b = binary.LittleEndian.AppendUint32(b, uint32(12+8+len(js)+8+len(bin)))
|
||||
b = binary.LittleEndian.AppendUint32(b, uint32(len(js)))
|
||||
b = append(b, "JSON"...)
|
||||
b = append(b, js...)
|
||||
b = binary.LittleEndian.AppendUint32(b, uint32(len(bin)))
|
||||
b = append(b, "BIN\x00"...)
|
||||
return append(b, bin...)
|
||||
}
|
||||
|
||||
// meshComfy is a ComfyUI that takes a picture and saves a mesh, as a 3D
|
||||
// node's SaveGLB does: under "3d", not "images".
|
||||
type meshComfy struct {
|
||||
srv *httptest.Server
|
||||
mu sync.Mutex
|
||||
uploaded []string
|
||||
workflow map[string]any
|
||||
}
|
||||
|
||||
func newMeshComfy(t *testing.T) *meshComfy {
|
||||
glb := testGLB(t)
|
||||
c := &meshComfy{}
|
||||
c.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/upload/image":
|
||||
f, h, err := r.FormFile("image")
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 400)
|
||||
return
|
||||
}
|
||||
b, _ := io.ReadAll(f)
|
||||
c.mu.Lock()
|
||||
c.uploaded = append(c.uploaded, h.Filename+":"+http.DetectContentType(b))
|
||||
c.mu.Unlock()
|
||||
fmt.Fprintf(w, `{"name":%q,"subfolder":"","type":"input"}`, h.Filename)
|
||||
case "/prompt":
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
c.mu.Lock()
|
||||
c.workflow = b.Prompt
|
||||
c.mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"m1"}`)
|
||||
case "/history/m1":
|
||||
fmt.Fprint(w, `{"m1":{"status":{"status_str":"success","completed":true},"outputs":{
|
||||
"3":{"text":["a preview"]},"4":{"3d":[{"filename":"mesh_00001_.glb","subfolder":"3d","type":"output"}]}}}}`)
|
||||
case "/view":
|
||||
// Served as ComfyUI serves a file it has no type for.
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
_, _ = w.Write(glb)
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(c.srv.Close)
|
||||
return c
|
||||
}
|
||||
|
||||
const meshWorkflow = `{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"2": {"class_type": "Trellis2ImageToMesh", "inputs": {"image": ["1", 0], "seed": "{{seed}}", "resolution": "{{resolution|512}}"}},
|
||||
"4": {"class_type": "SaveGLB", "inputs": {"mesh": ["2", 0], "filename_prefix": "3d/mesh"}}
|
||||
}`
|
||||
|
||||
func testPicture(t *testing.T) string {
|
||||
var b bytes.Buffer
|
||||
_ = png.Encode(&b, image.NewRGBA(image.Rect(0, 0, 8, 8)))
|
||||
return base64.StdEncoding.EncodeToString(b.Bytes())
|
||||
}
|
||||
|
||||
func meshRouter(t *testing.T, c *meshComfy) (*Router, string) {
|
||||
t.Helper()
|
||||
cfg := baseConfig()
|
||||
mit := &catalog.License{Name: "MIT", URL: "https://github.com/microsoft/TRELLIS.2/blob/main/LICENSE",
|
||||
Notice: "Made with TRELLIS.2 (MIT)."}
|
||||
cfg.Models = []catalog.Model{
|
||||
{Name: "trellis2", Source: "external", License: mit, External: &catalog.External{URL: c.srv.URL,
|
||||
Model: "trellis2", API: catalog.APIComfyUI, Output: catalog.OutputMeshes, Workflow: "w.json"}},
|
||||
{Name: "flux", Source: "external", External: &catalog.External{URL: c.srv.URL,
|
||||
Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}},
|
||||
}
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"trellis2": {PerMesh: 250000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:trellis2", URL: c.srv.URL, Models: []string{"trellis2"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "trellis2", Workflow: meshWorkflow},
|
||||
{ID: "external:flux", URL: c.srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "studio")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "art")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "props")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("m", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-m", Hash: sum[:], Prefix: "mmmm", Name: "art",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rt, secret
|
||||
}
|
||||
|
||||
func TestA3DModelIsMadeFromAPictureWithItsLicenceInside(t *testing.T) {
|
||||
c := newMeshComfy(t)
|
||||
rt, secret := meshRouter(t, c)
|
||||
h := rt.Handler()
|
||||
|
||||
code, j, raw := call(t, h, secret, http.MethodPost, meshesPath,
|
||||
map[string]any{"model": "trellis2", "image": "data:image/png;base64," + testPicture(t), "seed": 11, "resolution": 1024})
|
||||
if code != http.StatusAccepted {
|
||||
t.Fatalf("%d %s", code, raw)
|
||||
}
|
||||
got := waitJob(t, h, secret, j["id"].(string), jobDone)
|
||||
if got["status"] != "completed" || got["content_type"] != "model/gltf-binary" {
|
||||
t.Fatalf("job %v", got)
|
||||
}
|
||||
if p, _ := got["progress"].(map[string]any); p["state"] != "done" {
|
||||
t.Errorf("a finished job says %v", got["progress"])
|
||||
}
|
||||
if lic, _ := got["license"].(map[string]any); lic["name"] != "MIT" {
|
||||
t.Errorf("the job does not say the licence: %v", got["license"])
|
||||
}
|
||||
code, _, file := call(t, h, secret, http.MethodGet, "/v1/files/"+got["output_file_id"].(string)+"/content", nil)
|
||||
if code != http.StatusOK || !strings.HasPrefix(file, "glTF") {
|
||||
t.Fatalf("the result is not the mesh: %d %q", code, file[:min(len(file), 40)])
|
||||
}
|
||||
b := []byte(file)
|
||||
n := binary.LittleEndian.Uint32(b[12:])
|
||||
var doc struct {
|
||||
Asset struct {
|
||||
Generator string `json:"generator"`
|
||||
Extras struct {
|
||||
License catalog.License `json:"license"`
|
||||
} `json:"extras"`
|
||||
} `json:"asset"`
|
||||
}
|
||||
if err := json.Unmarshal(b[20:20+n], &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if doc.Asset.Extras.License.Notice != "Made with TRELLIS.2 (MIT)." || doc.Asset.Generator != "test" {
|
||||
t.Errorf("asset %+v: the notice is not in the mesh, or what was there is lost", doc.Asset)
|
||||
}
|
||||
if int(binary.LittleEndian.Uint32(b[8:])) != len(b) || !bytes.HasSuffix(b, []byte{1, 2, 3, 4}) {
|
||||
t.Error("the mesh's length or its binary chunk did not survive")
|
||||
}
|
||||
|
||||
c.mu.Lock()
|
||||
wf, _ := json.Marshal(c.workflow)
|
||||
uploaded := strings.Join(c.uploaded, ",")
|
||||
c.mu.Unlock()
|
||||
if !strings.HasSuffix(uploaded, ".png:image/png") || !strings.HasPrefix(uploaded, "llmgw-") {
|
||||
t.Errorf("uploaded %q", uploaded)
|
||||
}
|
||||
name := strings.TrimSuffix(uploaded, ":image/png")
|
||||
for _, want := range []string{`"image":"` + name + `"`, `"seed":11`, `"resolution":1024`} {
|
||||
if !strings.Contains(string(wf), want) {
|
||||
t.Errorf("%s not in the workflow sent: %s", want, wf)
|
||||
}
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, _ := rt.store.UsageByRequest(t.Context(), j["id"].(string))
|
||||
if ok {
|
||||
if u.Meshes != 1 || u.CostMicros < 250000 {
|
||||
t.Errorf("recorded %d meshes, %d micros; want one at its price", u.Meshes, u.CostMicros)
|
||||
}
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("no usage recorded")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestA3DModelIsAskedOfAModelThatMakesThem(t *testing.T) {
|
||||
c := newMeshComfy(t)
|
||||
rt, secret := meshRouter(t, c)
|
||||
h := rt.Handler()
|
||||
for name, c := range map[string]struct {
|
||||
path string
|
||||
body map[string]any
|
||||
want string
|
||||
}{
|
||||
"a mesh of flux": {meshesPath, map[string]any{"model": "flux", "image": testPicture(t)}, "makes pictures, not 3D models"},
|
||||
"no picture": {meshesPath, map[string]any{"model": "trellis2"}, "image is required"},
|
||||
"not a picture": {meshesPath, map[string]any{"model": "trellis2", "image": base64.StdEncoding.EncodeToString([]byte("hello"))}, "not a PNG"},
|
||||
"an odd resolution": {meshesPath, map[string]any{"model": "trellis2", "image": testPicture(t), "resolution": 700}, "multiple of 32"},
|
||||
"a picture of trellis2": {imagesPath, map[string]any{"model": "trellis2", "prompt": "a cube"}, "makes 3D models, not pictures"},
|
||||
"a chat with trellis2": {"/v1/chat/completions", map[string]any{"model": "trellis2", "messages": []any{}}, "makes 3D models"},
|
||||
} {
|
||||
code, _, raw := call(t, h, secret, http.MethodPost, c.path, c.body)
|
||||
if code != http.StatusBadRequest || !strings.Contains(raw, c.want) {
|
||||
t.Errorf("%s: %d %s", name, code, raw)
|
||||
}
|
||||
}
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
if !strings.Contains(rec.Body.String(), `"meshes":true`) || !strings.Contains(rec.Body.String(), `"license":{"name":"MIT"`) {
|
||||
t.Errorf("/v1/models: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOBJLeavesTheGatewayAsAGLB(t *testing.T) {
|
||||
// TripoSR saves an OBJ with a colour after each position, and a quad
|
||||
// here stands for the polygons other nodes write.
|
||||
obj := "# made by a test\nv 0 0 0 1 0 0\nv 1 0 0 0 1 0\nv 1 1 0 0 0 1\nv 0 1 0 1 1 1\nf 1 2 3 4\nf -4 -2 -1\n"
|
||||
glb, err := objToGLB([]byte(obj))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(glb[:4]) != "glTF" || int(binary.LittleEndian.Uint32(glb[8:])) != len(glb) {
|
||||
t.Fatal("not a glTF binary of its own length")
|
||||
}
|
||||
n := binary.LittleEndian.Uint32(glb[12:])
|
||||
var doc struct {
|
||||
Accessors []struct {
|
||||
Count int `json:"count"`
|
||||
Max []float64 `json:"max"`
|
||||
} `json:"accessors"`
|
||||
Meshes []struct {
|
||||
Primitives []struct {
|
||||
Attributes map[string]int `json:"attributes"`
|
||||
} `json:"primitives"`
|
||||
} `json:"meshes"`
|
||||
}
|
||||
if err := json.Unmarshal(glb[20:20+n], &doc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if doc.Accessors[0].Count != 4 || doc.Accessors[1].Count != 9 || doc.Accessors[0].Max[1] != 1 {
|
||||
t.Errorf("accessors %+v: four vertices and three triangles wanted", doc.Accessors)
|
||||
}
|
||||
if _, ok := doc.Meshes[0].Primitives[0].Attributes["COLOR_0"]; !ok {
|
||||
t.Error("the vertices' colours were lost")
|
||||
}
|
||||
if _, err := glbWithLicense(glb, catalog.License{Name: "MIT"}); err != nil {
|
||||
t.Errorf("the licence cannot be written into it: %v", err)
|
||||
}
|
||||
for _, bad := range []string{"v 0 0 0\n", "v 0 0 0\nf 1 2 3\n", "f x y z\n"} {
|
||||
if _, err := objToGLB([]byte(bad)); err == nil {
|
||||
t.Errorf("%q made a GLB", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestThe3DPageMakesAJobAndShowsItsMeshToItsOwnerOnly(t *testing.T) {
|
||||
c := newMeshComfy(t)
|
||||
rt, _ := meshRouter(t, c)
|
||||
project := mustProjectOf(t, rt, "key-m")
|
||||
user := store.User{ID: "usr-1", Email: "ann@studio.example", State: store.UserActive, CreatedAt: time.Now()}
|
||||
if err := rt.store.CreateUser(t.Context(), user); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := rt.store.AddMembership(t.Context(), user.ID, store.ScopeProject, project, "member", "test", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "other")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "t")
|
||||
otherProject, _ := rt.store.CreateProject(t.Context(), team, "p")
|
||||
stranger := store.User{ID: "usr-2", Email: "bob@else.example", State: store.UserActive, CreatedAt: time.Now()}
|
||||
_ = rt.store.CreateUser(t.Context(), stranger)
|
||||
_ = rt.store.AddMembership(t.Context(), stranger.ID, store.ScopeProject, otherProject, "member", "test", time.Now())
|
||||
|
||||
body := `{"model":"trellis2","image":"` + testPicture(t) + `","resolution":256,"seed":5}`
|
||||
rec := httptest.NewRecorder()
|
||||
rt.meshesSend(rec, httptest.NewRequest("POST", "/3d/generate", strings.NewReader(body)), user)
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var j map[string]any
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &j)
|
||||
id := j["id"].(string)
|
||||
rt.wakeJobs()
|
||||
|
||||
get := func(u store.User, path string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest("GET", path, nil)
|
||||
req.SetPathValue("id", id)
|
||||
rec := httptest.NewRecorder()
|
||||
next, ok := rt.asPerson(rec, req, u, "")
|
||||
if !ok {
|
||||
return rec
|
||||
}
|
||||
if strings.HasSuffix(path, "/file") {
|
||||
p, _ := auth.FromContext(next.Context())
|
||||
jj, ok := rt.ownedJob(rec, next, ownerOf(p))
|
||||
if ok {
|
||||
b, _ := rt.store.FileContent(next.Context(), jj.OutputFileID)
|
||||
rec.Header().Set("Content-Type", jj.ContentType)
|
||||
_, _ = rec.Write(b)
|
||||
}
|
||||
return rec
|
||||
}
|
||||
rt.handleGetJob(rec, next)
|
||||
return rec
|
||||
}
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for {
|
||||
rec := get(user, "/3d/jobs/"+id)
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &j)
|
||||
if j["status"] == "completed" {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("not made: %s", rec.Body.String())
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if rec := get(user, "/3d/jobs/"+id+"/file"); !strings.HasPrefix(rec.Body.String(), "glTF") {
|
||||
t.Errorf("the mesh: %q", rec.Body.String()[:min(20, rec.Body.Len())])
|
||||
}
|
||||
if rec := get(stranger, "/3d/jobs/"+id); rec.Code != http.StatusNotFound {
|
||||
t.Errorf("another project's person read the job: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// objToGLB turns a Wavefront OBJ — what a 3D node saves when it saves no
|
||||
// GLB — into a glTF binary: one mesh of its vertices, their colours where
|
||||
// the file gives them (TripoSR's do, after each position), and its faces
|
||||
// as triangles. A GLB is the one file a game engine, a browser and the
|
||||
// licence notice all take, so every mesh leaves the gateway as one.
|
||||
func objToGLB(obj []byte) ([]byte, error) {
|
||||
var pos, col []float32
|
||||
var idx []uint32
|
||||
colours := true
|
||||
sc := bufio.NewScanner(bytes.NewReader(obj))
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
f := strings.Fields(sc.Text())
|
||||
if len(f) == 0 {
|
||||
continue
|
||||
}
|
||||
switch f[0] {
|
||||
case "v":
|
||||
if len(f) < 4 {
|
||||
return nil, errors.New("a vertex with fewer than three coordinates")
|
||||
}
|
||||
for _, s := range f[1:4] {
|
||||
v, err := strconv.ParseFloat(s, 32)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("a vertex coordinate %q: %w", s, err)
|
||||
}
|
||||
pos = append(pos, float32(v))
|
||||
}
|
||||
if len(f) >= 7 {
|
||||
for _, s := range f[4:7] {
|
||||
v, err := strconv.ParseFloat(s, 32)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("a vertex colour %q: %w", s, err)
|
||||
}
|
||||
col = append(col, float32(v))
|
||||
}
|
||||
} else {
|
||||
colours = false
|
||||
}
|
||||
case "f":
|
||||
n := uint32(len(pos) / 3) //nolint:gosec // bounded by the file read
|
||||
var face []uint32
|
||||
for _, s := range f[1:] {
|
||||
ref, _, _ := strings.Cut(s, "/")
|
||||
i, err := strconv.Atoi(ref)
|
||||
if err != nil || i == 0 {
|
||||
return nil, fmt.Errorf("a face index %q", s)
|
||||
}
|
||||
if i < 0 {
|
||||
i = int(n) + i + 1
|
||||
}
|
||||
if i < 1 || uint32(i) > n { //nolint:gosec // checked positive
|
||||
return nil, fmt.Errorf("a face index %d past the %d vertices", i, n)
|
||||
}
|
||||
face = append(face, uint32(i-1)) //nolint:gosec // checked positive
|
||||
}
|
||||
// A polygon as a fan of triangles.
|
||||
for k := 1; k+1 < len(face); k++ {
|
||||
idx = append(idx, face[0], face[k], face[k+1])
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(pos) == 0 || len(idx) == 0 {
|
||||
return nil, errors.New("the OBJ has no faces")
|
||||
}
|
||||
if !colours || len(col) != len(pos) {
|
||||
col = nil
|
||||
}
|
||||
|
||||
var bin bytes.Buffer
|
||||
view := func(data any) (offset, length int) {
|
||||
offset = bin.Len()
|
||||
_ = binary.Write(&bin, binary.LittleEndian, data)
|
||||
length = bin.Len() - offset
|
||||
for bin.Len()%4 != 0 {
|
||||
bin.WriteByte(0)
|
||||
}
|
||||
return offset, length
|
||||
}
|
||||
lo := []float32{math.MaxFloat32, math.MaxFloat32, math.MaxFloat32}
|
||||
hi := []float32{-math.MaxFloat32, -math.MaxFloat32, -math.MaxFloat32}
|
||||
for i, v := range pos {
|
||||
lo[i%3], hi[i%3] = min(lo[i%3], v), max(hi[i%3], v)
|
||||
}
|
||||
po, pl := view(pos)
|
||||
io, il := view(idx)
|
||||
views := []map[string]any{
|
||||
{"buffer": 0, "byteOffset": po, "byteLength": pl, "target": 34962},
|
||||
{"buffer": 0, "byteOffset": io, "byteLength": il, "target": 34963},
|
||||
}
|
||||
accessors := []map[string]any{
|
||||
{"bufferView": 0, "componentType": 5126, "count": len(pos) / 3, "type": "VEC3", "min": lo, "max": hi},
|
||||
{"bufferView": 1, "componentType": 5125, "count": len(idx), "type": "SCALAR"},
|
||||
}
|
||||
attrs := map[string]any{"POSITION": 0}
|
||||
if col != nil {
|
||||
co, cl := view(col)
|
||||
views = append(views, map[string]any{"buffer": 0, "byteOffset": co, "byteLength": cl, "target": 34962})
|
||||
accessors = append(accessors, map[string]any{"bufferView": 2, "componentType": 5126, "count": len(col) / 3, "type": "VEC3"})
|
||||
attrs["COLOR_0"] = 2
|
||||
}
|
||||
doc := map[string]any{
|
||||
"asset": map[string]any{"version": "2.0", "generator": "llmgw"},
|
||||
"scene": 0,
|
||||
"scenes": []any{map[string]any{"nodes": []int{0}}},
|
||||
"nodes": []any{map[string]any{"mesh": 0}},
|
||||
"meshes": []any{map[string]any{"primitives": []any{map[string]any{"attributes": attrs, "indices": 1}}}},
|
||||
"accessors": accessors,
|
||||
"bufferViews": views,
|
||||
"buffers": []any{map[string]any{"byteLength": bin.Len()}},
|
||||
}
|
||||
js, err := json.Marshal(doc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for len(js)%4 != 0 {
|
||||
js = append(js, ' ')
|
||||
}
|
||||
out := make([]byte, 0, 28+len(js)+bin.Len())
|
||||
out = append(out, "glTF"...)
|
||||
out = binary.LittleEndian.AppendUint32(out, 2)
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(28+len(js)+bin.Len())) //nolint:gosec // bounded by the file read
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(len(js))) //nolint:gosec // bounded by the file read
|
||||
out = append(out, "JSON"...)
|
||||
out = append(out, js...)
|
||||
out = binary.LittleEndian.AppendUint32(out, uint32(bin.Len())) //nolint:gosec // bounded by the file read
|
||||
out = append(out, "BIN\x00"...)
|
||||
return append(out, bin.Bytes()...), nil
|
||||
}
|
||||
@@ -17,7 +17,7 @@ func TestSessionPagesAreNotAskedForAnAPIKey(t *testing.T) {
|
||||
h := rt.Handler()
|
||||
// The /admin pages are mounted only with a cluster view, which this
|
||||
// router has none of; these are the session pages it does mount.
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/org", "/org/org-x"} {
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/3d", "/org", "/org/org-x"} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
if strings.Contains(w.Body.String(), "api key") {
|
||||
|
||||
@@ -226,6 +226,8 @@ type Outcome struct {
|
||||
Usage *Usage
|
||||
// Images is how many pictures were made, by an endpoint that makes them.
|
||||
Images int
|
||||
// Meshes is how many 3D models were made.
|
||||
Meshes int
|
||||
// Bytes is how many bytes were relayed to the client.
|
||||
Bytes int64
|
||||
// Reply is the assistant's text, when CaptureReply asked for it. Truncated
|
||||
|
||||
@@ -77,8 +77,10 @@ type Router struct {
|
||||
canaries canaryStats
|
||||
// slo keeps each model's recent token timings, for admission.
|
||||
slo sloTracker
|
||||
// batchWake has the batch worker look for work at once.
|
||||
// batchWake has the batch worker look for work at once; jobWake the
|
||||
// job worker.
|
||||
batchWake chan struct{}
|
||||
jobWake chan struct{}
|
||||
// reqLogs is each project's keeping of requests, as last read.
|
||||
reqLogs requestLogs
|
||||
// answers are the answers projects reuse; nil reuses none.
|
||||
@@ -304,6 +306,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
|
||||
r := &Router{
|
||||
batchWake: make(chan struct{}, 1),
|
||||
jobWake: make(chan struct{}, 1),
|
||||
cfg: cfg.Router,
|
||||
landing: cfg.Landing,
|
||||
log: log,
|
||||
@@ -374,7 +377,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
prices := make(map[string]accounting.Price, len(cfg.Auth.Pricing))
|
||||
for name, p := range cfg.Auth.Pricing {
|
||||
prices[name] = accounting.Price{
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, PerImage: p.PerImage,
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, PerImage: p.PerImage, PerMesh: p.PerMesh,
|
||||
}
|
||||
}
|
||||
r.recorder = accounting.NewRecorder(st, r.auth, accounting.RecorderConfig{
|
||||
@@ -391,6 +394,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
PromptPer1k: cfg.Auth.DefaultPrice.PromptPer1k,
|
||||
CompletionPer1k: cfg.Auth.DefaultPrice.CompletionPer1k,
|
||||
PerImage: cfg.Auth.DefaultPrice.PerImage,
|
||||
PerMesh: cfg.Auth.DefaultPrice.PerMesh,
|
||||
}),
|
||||
r.recorder, log)
|
||||
r.threads = r.store
|
||||
@@ -432,6 +436,7 @@ func admissionConfig(rc config.Router) admission.Config {
|
||||
Name: b.Name,
|
||||
Priority: b.Priority,
|
||||
SaturationCeiling: b.SaturationCeiling,
|
||||
QueueTTL: b.QueueTTL,
|
||||
})
|
||||
}
|
||||
return admission.Config{
|
||||
@@ -528,6 +533,7 @@ func (r *Router) Start(ctx context.Context) {
|
||||
}
|
||||
if r.store != nil {
|
||||
go r.runBatches(ctx)
|
||||
go r.runJobs(ctx)
|
||||
go r.pruneRequestBodies(ctx)
|
||||
go r.refreshTenants(ctx)
|
||||
go r.purgeSessions(ctx)
|
||||
|
||||
@@ -68,6 +68,8 @@ var copyOrder = []string{
|
||||
"files",
|
||||
"batches",
|
||||
"batch_results",
|
||||
// After files, which a finished job's result is one of.
|
||||
"jobs",
|
||||
"request_bodies",
|
||||
// Runs before their cases, which point at them.
|
||||
"eval_runs",
|
||||
@@ -88,6 +90,7 @@ var copyOrder = []string{
|
||||
var notCopied = map[string]string{
|
||||
"leadership": "a lease on a running process, meaningless in a new store",
|
||||
"cluster_state": "worked out again by the leader from the settings and the machines, within minutes",
|
||||
"notify_sent": "when notifications were last sent; at worst one is sent again an hour early",
|
||||
}
|
||||
|
||||
// CopyResult is how many rows each table received.
|
||||
|
||||
@@ -25,6 +25,8 @@ type ExportRow struct {
|
||||
Prompt int64
|
||||
Completion int64
|
||||
CostMicros int64
|
||||
Images int64
|
||||
Meshes int64
|
||||
}
|
||||
|
||||
// ExportDims are what an export can be broken down by.
|
||||
@@ -51,16 +53,18 @@ func (s *DB) UsageExport(ctx context.Context, from, to time.Time, period string,
|
||||
bucket int64
|
||||
key, user, project, model string
|
||||
requests, errors, limited, prompt, completion, cost int64
|
||||
images, meshes int64
|
||||
}
|
||||
//nolint:gosec // table is one of two literals above
|
||||
rows, err := queryAll(ctx, s.db, `SELECT bucket, key_id, user_id, project_id, served_model,
|
||||
SUM(requests), SUM(errors), SUM(limited), SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros)
|
||||
SUM(requests), SUM(errors), SUM(limited), SUM(prompt_tokens), SUM(completion_tokens), SUM(cost_micros),
|
||||
SUM(images), SUM(meshes)
|
||||
FROM `+table+` WHERE bucket >= ? AND bucket < ?
|
||||
GROUP BY bucket, key_id, user_id, project_id, served_model`,
|
||||
[]any{millis(from), millis(to)}, func(r *sql.Rows) (raw, error) {
|
||||
var x raw
|
||||
err := r.Scan(&x.bucket, &x.key, &x.user, &x.project, &x.model,
|
||||
&x.requests, &x.errors, &x.limited, &x.prompt, &x.completion, &x.cost)
|
||||
&x.requests, &x.errors, &x.limited, &x.prompt, &x.completion, &x.cost, &x.images, &x.meshes)
|
||||
return x, err
|
||||
})
|
||||
if err != nil {
|
||||
@@ -137,6 +141,8 @@ func (s *DB) UsageExport(ctx context.Context, from, to time.Time, period string,
|
||||
t.Prompt += x.prompt
|
||||
t.Completion += x.completion
|
||||
t.CostMicros += x.cost
|
||||
t.Images += x.images
|
||||
t.Meshes += x.meshes
|
||||
}
|
||||
out := make([]ExportRow, 0, len(order))
|
||||
for _, k := range order {
|
||||
|
||||
@@ -46,3 +46,36 @@ func TestUsageIsExportedByPeriodAndWhatWasAskedFor(t *testing.T) {
|
||||
t.Error("an unknown dimension was taken")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAreCountedInTheSumsAndTheExport(t *testing.T) {
|
||||
// The request rows said how many pictures each made, and the sums did
|
||||
// not: once the rows were past their retention, an export said what the
|
||||
// pictures cost and not how many there were.
|
||||
db := capacityStore(t)
|
||||
ctx := t.Context()
|
||||
at := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
|
||||
if err := db.WriteUsage(ctx, []Usage{
|
||||
{At: at, KeyID: "k", Model: "flux", ServedModel: "flux", PromptTokens: 9, Images: 2, CostMicros: 40000, Status: 200},
|
||||
{At: at.Add(time.Minute), KeyID: "k", Model: "flux", ServedModel: "flux", PromptTokens: 9, Images: 3, CostMicros: 60000, Status: 200},
|
||||
{At: at.Add(2 * time.Minute), KeyID: "k", Model: "chat", ServedModel: "chat", PromptTokens: 9, Status: 200},
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Past their retention: only the sums are left.
|
||||
if _, err := db.PruneUsage(ctx, at.Add(time.Hour)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, period := range []string{"hour", "day"} {
|
||||
rows, err := db.UsageExport(ctx, at.Add(-24*time.Hour), at.Add(48*time.Hour), period, []string{"model"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := map[string]int64{}
|
||||
for _, r := range rows {
|
||||
got[r.Model] += r.Images
|
||||
}
|
||||
if got["flux"] != 5 || got["chat"] != 0 {
|
||||
t.Errorf("%s: pictures by model %v, want flux 5", period, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Job is one request that is answered with an id and asked after: work that
|
||||
// takes minutes, which no client should hold a connection for. The request
|
||||
// is kept as it was sent and replayed by whichever gateway holds the lease.
|
||||
type Job struct {
|
||||
ID string
|
||||
Owner string
|
||||
KeyID string
|
||||
Endpoint string
|
||||
Model string
|
||||
// Input is the request's body, as the client sent it.
|
||||
Input []byte
|
||||
Status string
|
||||
// Progress is what the engine said of the work so far, as JSON; empty
|
||||
// until it said anything.
|
||||
Progress string
|
||||
// PromptID is the engine's own id for the work, so a gateway that takes
|
||||
// over a running job asks after the same work rather than queueing it
|
||||
// again.
|
||||
PromptID string
|
||||
OutputFileID string
|
||||
ContentType string
|
||||
Error string
|
||||
CreatedAt time.Time
|
||||
StartedAt time.Time
|
||||
FinishedAt time.Time
|
||||
// ExpiresAt is when a job not yet finished is given up on.
|
||||
ExpiresAt time.Time
|
||||
}
|
||||
|
||||
// Job states.
|
||||
const (
|
||||
JobQueued = "queued"
|
||||
JobRunning = "running"
|
||||
JobCompleted = "completed"
|
||||
JobFailed = "failed"
|
||||
JobCancelling = "cancelling"
|
||||
JobCancelled = "cancelled"
|
||||
JobExpired = "expired"
|
||||
)
|
||||
|
||||
// JobDone is whether a state is an end.
|
||||
func JobDone(status string) bool {
|
||||
switch status {
|
||||
case JobCompleted, JobFailed, JobCancelled, JobExpired:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
const jobCols = `id, owner, key_id, endpoint, model, input, status, progress, prompt_id, output_file_id,
|
||||
content_type, error, created_at, started_at, finished_at, expires_at`
|
||||
|
||||
func scanJob(sc interface{ Scan(...any) error }) (Job, error) {
|
||||
var j Job
|
||||
var created, started, finished, expires int64
|
||||
err := sc.Scan(&j.ID, &j.Owner, &j.KeyID, &j.Endpoint, &j.Model, &j.Input, &j.Status, &j.Progress, &j.PromptID,
|
||||
&j.OutputFileID, &j.ContentType, &j.Error, &created, &started, &finished, &expires)
|
||||
if err != nil {
|
||||
return Job{}, err
|
||||
}
|
||||
at := func(v int64) time.Time {
|
||||
if v == 0 {
|
||||
return time.Time{}
|
||||
}
|
||||
return time.UnixMilli(v)
|
||||
}
|
||||
j.CreatedAt, j.StartedAt, j.FinishedAt, j.ExpiresAt = at(created), at(started), at(finished), at(expires)
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// PutJob records a new job.
|
||||
func (s *DB) PutJob(ctx context.Context, j Job) error {
|
||||
if j.Input == nil {
|
||||
j.Input = []byte{}
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx, `INSERT INTO jobs (id, owner, key_id, endpoint, model, input, status, created_at, expires_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
j.ID, j.Owner, j.KeyID, j.Endpoint, j.Model, j.Input, j.Status, millis(j.CreatedAt), millis(j.ExpiresAt))
|
||||
return err
|
||||
}
|
||||
|
||||
// UpdateJob writes how a job stands: its state, result, error and times.
|
||||
// Its progress and prompt id are written by SetJobProgress, which the work
|
||||
// calls often and which must not undo an end written here.
|
||||
func (s *DB) UpdateJob(ctx context.Context, j Job) error {
|
||||
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, output_file_id = ?, content_type = ?, error = ?,
|
||||
started_at = ?, finished_at = ? WHERE id = ?`,
|
||||
j.Status, j.OutputFileID, j.ContentType, j.Error, optMillis(j.StartedAt), optMillis(j.FinishedAt), j.ID)
|
||||
return err
|
||||
}
|
||||
|
||||
// SetJobProgress writes what the engine said of a running job, and its id
|
||||
// for the work when it is known. A job that has ended keeps its end.
|
||||
func (s *DB) SetJobProgress(ctx context.Context, id, progress, promptID string) error {
|
||||
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET progress = ?,
|
||||
prompt_id = CASE WHEN ? = '' THEN prompt_id ELSE ? END
|
||||
WHERE id = ? AND status IN (?, ?)`, progress, promptID, promptID, id, JobRunning, JobCancelling)
|
||||
return err
|
||||
}
|
||||
|
||||
// CancelJob asks for a job to stop: cancelling, unless it has ended already.
|
||||
// It says the job as it stands.
|
||||
func (s *DB) CancelJob(ctx context.Context, id string, now time.Time) (Job, error) {
|
||||
// One not yet taken ends at once: there is nothing to stop.
|
||||
if _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ?, finished_at = ?
|
||||
WHERE id = ? AND status = ? AND lease_until < ?`, JobCancelled, millis(now), id, JobQueued, millis(now)); err != nil {
|
||||
return Job{}, err
|
||||
}
|
||||
if _, err := s.db.ExecContext(ctx, `UPDATE jobs SET status = ? WHERE id = ? AND status IN (?, ?)`,
|
||||
JobCancelling, id, JobQueued, JobRunning); err != nil {
|
||||
return Job{}, err
|
||||
}
|
||||
return s.JobByID(ctx, id)
|
||||
}
|
||||
|
||||
// JobByID finds a job, ErrNotFound when there is none.
|
||||
func (s *DB) JobByID(ctx context.Context, id string) (Job, error) {
|
||||
j, err := scanJob(s.db.QueryRowContext(ctx, `SELECT `+jobCols+` FROM jobs WHERE id = ?`, id))
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Job{}, ErrNotFound
|
||||
}
|
||||
return j, err
|
||||
}
|
||||
|
||||
// Jobs lists an owner's jobs, the newest first, those made before the job
|
||||
// after when it is given.
|
||||
func (s *DB) Jobs(ctx context.Context, owner, after string, limit int) ([]Job, error) {
|
||||
q := `SELECT ` + jobCols + ` FROM jobs WHERE owner = ?`
|
||||
args := []any{owner}
|
||||
if after != "" {
|
||||
q += ` AND created_at < (SELECT created_at FROM jobs WHERE id = ?)`
|
||||
args = append(args, after)
|
||||
}
|
||||
if limit <= 0 || limit > 100 {
|
||||
limit = 20
|
||||
}
|
||||
q += ` ORDER BY created_at DESC LIMIT ?`
|
||||
args = append(args, limit)
|
||||
rows, err := s.db.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []Job
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, j)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// AllJobs is every owner's jobs, newest first, those in one status when it
|
||||
// is named: what an operator looks over.
|
||||
func (s *DB) AllJobs(ctx context.Context, status string, limit int) ([]Job, error) {
|
||||
q := `SELECT ` + jobCols + ` FROM jobs`
|
||||
var args []any
|
||||
if status != "" {
|
||||
q += ` WHERE status = ?`
|
||||
args = append(args, status)
|
||||
}
|
||||
if limit <= 0 || limit > 500 {
|
||||
limit = 50
|
||||
}
|
||||
q += ` ORDER BY created_at DESC LIMIT ?`
|
||||
args = append(args, limit)
|
||||
rows, err := s.db.QueryContext(ctx, q, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []Job
|
||||
for rows.Next() {
|
||||
j, err := scanJob(rows)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, j)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// ClaimJob takes the lease on the oldest job with work left that no gateway
|
||||
// holds, for owner until the time given. ok is false when there is none.
|
||||
func (s *DB) ClaimJob(ctx context.Context, owner string, now, until time.Time) (Job, bool, error) {
|
||||
var id string
|
||||
err := s.db.QueryRowContext(ctx, `UPDATE jobs SET lease_owner = ?, lease_until = ?
|
||||
WHERE id = (SELECT id FROM jobs WHERE status IN (?, ?, ?) AND lease_until < ?
|
||||
ORDER BY created_at LIMIT 1)
|
||||
AND lease_until < ?
|
||||
RETURNING id`, owner, millis(until), JobQueued, JobRunning, JobCancelling, millis(now), millis(now)).Scan(&id)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Job{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return Job{}, false, err
|
||||
}
|
||||
j, err := s.JobByID(ctx, id)
|
||||
return j, err == nil, err
|
||||
}
|
||||
|
||||
// RenewJobLease keeps a lease held; false when it is held no longer.
|
||||
func (s *DB) RenewJobLease(ctx context.Context, id, owner string, until time.Time) (bool, error) {
|
||||
res, err := s.db.ExecContext(ctx, `UPDATE jobs SET lease_until = ? WHERE id = ? AND lease_owner = ?`,
|
||||
millis(until), id, owner)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// ReleaseJob gives a lease up.
|
||||
func (s *DB) ReleaseJob(ctx context.Context, id, owner string) error {
|
||||
_, err := s.db.ExecContext(ctx, `UPDATE jobs SET lease_until = 0, lease_owner = '' WHERE id = ? AND lease_owner = ?`, id, owner)
|
||||
return err
|
||||
}
|
||||
|
||||
// DeleteJobsBefore forgets the jobs that ended before the time given, whose
|
||||
// result files are gone by then too.
|
||||
func (s *DB) DeleteJobsBefore(ctx context.Context, before time.Time) (int64, error) {
|
||||
res, err := s.db.ExecContext(ctx, `DELETE FROM jobs WHERE finished_at > 0 AND finished_at < ?`, millis(before))
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return res.RowsAffected()
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestJobs(t *testing.T) {
|
||||
checkJobs(t, capacityStore(t))
|
||||
}
|
||||
|
||||
func TestPostgresJobs(t *testing.T) {
|
||||
checkJobs(t, openPG(t))
|
||||
}
|
||||
|
||||
func checkJobs(t *testing.T, db *DB) {
|
||||
ctx := t.Context()
|
||||
now := time.Now()
|
||||
j := Job{ID: "job-1", Owner: "p1", KeyID: "k", Endpoint: "/v1/3d/generations", Model: "trellis2",
|
||||
Input: []byte(`{"model":"trellis2"}`), Status: JobQueued, CreatedAt: now, ExpiresAt: now.Add(24 * time.Hour)}
|
||||
if err := db.PutJob(ctx, j); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, ok, err := db.ClaimJob(ctx, "gw-a", now, now.Add(time.Minute))
|
||||
if err != nil || !ok || got.ID != "job-1" || string(got.Input) != `{"model":"trellis2"}` {
|
||||
t.Fatalf("claim: %+v %v %v", got, ok, err)
|
||||
}
|
||||
if _, ok, _ := db.ClaimJob(ctx, "gw-b", now, now.Add(time.Minute)); ok {
|
||||
t.Error("a leased job was claimed twice")
|
||||
}
|
||||
if held, _ := db.RenewJobLease(ctx, "job-1", "gw-b", now.Add(time.Hour)); held {
|
||||
t.Error("another gateway renewed the lease")
|
||||
}
|
||||
if held, _ := db.RenewJobLease(ctx, "job-1", "gw-a", now.Add(time.Hour)); !held {
|
||||
t.Error("the holder could not renew")
|
||||
}
|
||||
|
||||
// Progress is written while it runs, and never over an end.
|
||||
got.Status, got.StartedAt = JobRunning, now
|
||||
if err := db.UpdateJob(ctx, got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetJobProgress(ctx, "job-1", `{"percent":40}`, "p-7"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := db.SetJobProgress(ctx, "job-1", `{"percent":60}`, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cur, _ := db.JobByID(ctx, "job-1")
|
||||
if cur.Progress != `{"percent":60}` || cur.PromptID != "p-7" || cur.Status != JobRunning {
|
||||
t.Errorf("%+v", cur)
|
||||
}
|
||||
|
||||
// A running job asked to stop is cancelling, for the worker to stop it.
|
||||
cur, err = db.CancelJob(ctx, "job-1", now)
|
||||
if err != nil || cur.Status != JobCancelling {
|
||||
t.Fatalf("%+v %v", cur, err)
|
||||
}
|
||||
cur.Status, cur.FinishedAt = JobCancelled, now
|
||||
_ = db.UpdateJob(ctx, cur)
|
||||
if err := db.SetJobProgress(ctx, "job-1", `{"percent":99}`, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cur, _ = db.JobByID(ctx, "job-1"); cur.Progress != `{"percent":60}` || cur.Status != JobCancelled {
|
||||
t.Errorf("progress written over an end: %+v", cur)
|
||||
}
|
||||
_ = db.ReleaseJob(ctx, "job-1", "gw-a")
|
||||
if _, ok, _ := db.ClaimJob(ctx, "gw-b", now.Add(2*time.Hour), now.Add(3*time.Hour)); ok {
|
||||
t.Error("an ended job was claimed")
|
||||
}
|
||||
|
||||
// One not yet taken ends at once when cancelled.
|
||||
j2 := j
|
||||
j2.ID, j2.CreatedAt = "job-2", now.Add(time.Second)
|
||||
_ = db.PutJob(ctx, j2)
|
||||
if cur, _ = db.CancelJob(ctx, "job-2", now.Add(2*time.Second)); cur.Status != JobCancelled || cur.FinishedAt.IsZero() {
|
||||
t.Errorf("%+v", cur)
|
||||
}
|
||||
|
||||
// Listed newest first, another owner's not at all.
|
||||
j3 := j
|
||||
j3.ID, j3.Owner, j3.CreatedAt = "job-3", "p2", now.Add(3*time.Second)
|
||||
_ = db.PutJob(ctx, j3)
|
||||
js, _ := db.Jobs(ctx, "p1", "", 10)
|
||||
if len(js) != 2 || js[0].ID != "job-2" || js[1].ID != "job-1" {
|
||||
t.Errorf("listed %+v", js)
|
||||
}
|
||||
if js, _ = db.Jobs(ctx, "p1", "job-2", 10); len(js) != 1 || js[0].ID != "job-1" {
|
||||
t.Errorf("after: %+v", js)
|
||||
}
|
||||
if _, err := db.JobByID(ctx, "nope"); err != ErrNotFound {
|
||||
t.Errorf("err %v", err)
|
||||
}
|
||||
if n, _ := db.DeleteJobsBefore(ctx, now.Add(time.Hour)); n != 2 {
|
||||
t.Errorf("deleted %d", n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOperatorSeesEveryOwnersJobs(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
now := time.Now()
|
||||
for i, owner := range []string{"project:a", "project:b", "user:c"} {
|
||||
status := JobQueued
|
||||
if i == 2 {
|
||||
status = JobCompleted
|
||||
}
|
||||
if err := db.PutJob(t.Context(), Job{ID: fmt.Sprint("job_", i), Owner: owner, KeyID: "k", Endpoint: "/v1/3d/generations",
|
||||
Model: "m", Input: []byte("{}"), Status: status, CreatedAt: now.Add(time.Duration(i) * time.Second),
|
||||
ExpiresAt: now.Add(time.Hour)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
all, err := db.AllJobs(t.Context(), "", 0)
|
||||
if err != nil || len(all) != 3 || all[0].ID != "job_2" {
|
||||
t.Fatalf("%v %v: every owner's, newest first", all, err)
|
||||
}
|
||||
queued, _ := db.AllJobs(t.Context(), JobQueued, 0)
|
||||
if len(queued) != 2 {
|
||||
t.Errorf("%d queued", len(queued))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- How many pictures each slice's requests made, beside their tokens and
|
||||
-- cost, so the sums and exports say it after the request rows are gone.
|
||||
-- Pictures made before this were counted only in those rows: added from
|
||||
-- them, for the slices since the first.
|
||||
ALTER TABLE usage_5m ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN images BIGINT NOT NULL DEFAULT 0;
|
||||
UPDATE usage_5m SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_5m.bucket AND u.ts < usage_5m.bucket + 300000
|
||||
AND u.key_id = usage_5m.key_id AND u.user_id = usage_5m.user_id AND u.project_id = usage_5m.project_id
|
||||
AND u.model = usage_5m.model AND u.served_model = usage_5m.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 300000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1h SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1h.bucket AND u.ts < usage_1h.bucket + 3600000
|
||||
AND u.key_id = usage_1h.key_id AND u.user_id = usage_1h.user_id AND u.project_id = usage_1h.project_id
|
||||
AND u.model = usage_1h.model AND u.served_model = usage_1h.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 3600000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1d SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1d.bucket AND u.ts < usage_1d.bucket + 86400000
|
||||
AND u.key_id = usage_1d.key_id AND u.user_id = usage_1d.user_id AND u.project_id = usage_1d.project_id
|
||||
AND u.model = usage_1d.model AND u.served_model = usage_1d.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 86400000 FROM usage WHERE images > 0);
|
||||
@@ -0,0 +1,8 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- When each notification was last sent, by its kind and subject: every
|
||||
-- control plane raises the same events — a budget spent is seen by each one
|
||||
-- serving its requests — and the one that claims an event here sends it.
|
||||
CREATE TABLE notify_sent (
|
||||
key TEXT PRIMARY KEY,
|
||||
sent_at BIGINT NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,34 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- Jobs (BACKLOG G1): a request that takes minutes — a 3D model from a
|
||||
-- picture — answered with an id and asked after, rather than held open.
|
||||
-- The request is kept as it was sent, so whichever gateway holds the lease
|
||||
-- replays it through the same path a client's request takes; the result is
|
||||
-- a file in the files table, kept for the batch files' retention.
|
||||
CREATE TABLE jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL,
|
||||
key_id TEXT NOT NULL,
|
||||
endpoint TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
input BYTEA NOT NULL,
|
||||
-- status is queued, running, completed, failed, cancelling, cancelled or
|
||||
-- expired.
|
||||
status TEXT NOT NULL,
|
||||
-- progress is what the engine said of the work so far, as JSON: queue
|
||||
-- position, percent, step. Empty until it says anything.
|
||||
progress TEXT NOT NULL DEFAULT '',
|
||||
-- prompt_id is the engine's own id for the work (ComfyUI's prompt_id),
|
||||
-- so a gateway that takes over a running job asks after the same work.
|
||||
prompt_id TEXT NOT NULL DEFAULT '',
|
||||
output_file_id TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL DEFAULT '',
|
||||
error TEXT NOT NULL DEFAULT '',
|
||||
created_at BIGINT NOT NULL,
|
||||
started_at BIGINT NOT NULL DEFAULT 0,
|
||||
finished_at BIGINT NOT NULL DEFAULT 0,
|
||||
expires_at BIGINT NOT NULL,
|
||||
lease_owner TEXT NOT NULL DEFAULT '',
|
||||
lease_until BIGINT NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE INDEX jobs_owner ON jobs (owner, created_at);
|
||||
CREATE INDEX jobs_status ON jobs (status, lease_until);
|
||||
@@ -0,0 +1,7 @@
|
||||
-- Ported from the SQLite migration of the same number.
|
||||
-- How many 3D models a request made, as images says how many pictures: what
|
||||
-- per_mesh is charged by, in the request rows and in the sums.
|
||||
ALTER TABLE usage ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_5m ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN meshes BIGINT NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,22 @@
|
||||
-- How many pictures each slice's requests made, beside their tokens and
|
||||
-- cost, so the sums and exports say it after the request rows are gone.
|
||||
-- Pictures made before this were counted only in those rows: added from
|
||||
-- them, for the slices since the first.
|
||||
ALTER TABLE usage_5m ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN images INTEGER NOT NULL DEFAULT 0;
|
||||
UPDATE usage_5m SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_5m.bucket AND u.ts < usage_5m.bucket + 300000
|
||||
AND u.key_id = usage_5m.key_id AND u.user_id = usage_5m.user_id AND u.project_id = usage_5m.project_id
|
||||
AND u.model = usage_5m.model AND u.served_model = usage_5m.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 300000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1h SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1h.bucket AND u.ts < usage_1h.bucket + 3600000
|
||||
AND u.key_id = usage_1h.key_id AND u.user_id = usage_1h.user_id AND u.project_id = usage_1h.project_id
|
||||
AND u.model = usage_1h.model AND u.served_model = usage_1h.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 3600000 FROM usage WHERE images > 0);
|
||||
UPDATE usage_1d SET images = COALESCE((SELECT SUM(u.images) FROM usage u
|
||||
WHERE u.images > 0 AND u.ts >= usage_1d.bucket AND u.ts < usage_1d.bucket + 86400000
|
||||
AND u.key_id = usage_1d.key_id AND u.user_id = usage_1d.user_id AND u.project_id = usage_1d.project_id
|
||||
AND u.model = usage_1d.model AND u.served_model = usage_1d.served_model), 0)
|
||||
WHERE bucket >= (SELECT COALESCE(MIN(ts), 9000000000000000000) - 86400000 FROM usage WHERE images > 0);
|
||||
@@ -0,0 +1,7 @@
|
||||
-- When each notification was last sent, by its kind and subject: every
|
||||
-- control plane raises the same events — a budget spent is seen by each one
|
||||
-- serving its requests — and the one that claims an event here sends it.
|
||||
CREATE TABLE notify_sent (
|
||||
key TEXT PRIMARY KEY,
|
||||
sent_at BIGINT NOT NULL
|
||||
);
|
||||
@@ -0,0 +1,33 @@
|
||||
-- Jobs (BACKLOG G1): a request that takes minutes — a 3D model from a
|
||||
-- picture — answered with an id and asked after, rather than held open.
|
||||
-- The request is kept as it was sent, so whichever gateway holds the lease
|
||||
-- replays it through the same path a client's request takes; the result is
|
||||
-- a file in the files table, kept for the batch files' retention.
|
||||
CREATE TABLE jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
owner TEXT NOT NULL,
|
||||
key_id TEXT NOT NULL,
|
||||
endpoint TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
input BLOB NOT NULL,
|
||||
-- status is queued, running, completed, failed, cancelling, cancelled or
|
||||
-- expired.
|
||||
status TEXT NOT NULL,
|
||||
-- progress is what the engine said of the work so far, as JSON: queue
|
||||
-- position, percent, step. Empty until it says anything.
|
||||
progress TEXT NOT NULL DEFAULT '',
|
||||
-- prompt_id is the engine's own id for the work (ComfyUI's prompt_id),
|
||||
-- so a gateway that takes over a running job asks after the same work.
|
||||
prompt_id TEXT NOT NULL DEFAULT '',
|
||||
output_file_id TEXT NOT NULL DEFAULT '',
|
||||
content_type TEXT NOT NULL DEFAULT '',
|
||||
error TEXT NOT NULL DEFAULT '',
|
||||
created_at INTEGER NOT NULL,
|
||||
started_at INTEGER NOT NULL DEFAULT 0,
|
||||
finished_at INTEGER NOT NULL DEFAULT 0,
|
||||
expires_at INTEGER NOT NULL,
|
||||
lease_owner TEXT NOT NULL DEFAULT '',
|
||||
lease_until INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE INDEX jobs_owner ON jobs (owner, created_at);
|
||||
CREATE INDEX jobs_status ON jobs (status, lease_until);
|
||||
@@ -0,0 +1,6 @@
|
||||
-- How many 3D models a request made, as images says how many pictures: what
|
||||
-- per_mesh is charged by, in the request rows and in the sums.
|
||||
ALTER TABLE usage ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_5m ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1h ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE usage_1d ADD COLUMN meshes INTEGER NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,23 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ClaimNotice claims the sending of a notification: true when nobody has
|
||||
// sent the same one within the window, and from now on, nobody else may
|
||||
// until it has passed. One statement, so two control planes asking at once
|
||||
// cannot both be told yes.
|
||||
func (s *DB) ClaimNotice(ctx context.Context, key string, at time.Time, window time.Duration) (bool, error) {
|
||||
now := at.UnixMilli()
|
||||
res, err := s.db.ExecContext(ctx,
|
||||
`INSERT INTO notify_sent (key, sent_at) VALUES (?, ?)
|
||||
ON CONFLICT (key) DO UPDATE SET sent_at = excluded.sent_at WHERE notify_sent.sent_at < ?`,
|
||||
key, now, now-window.Milliseconds())
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, err := res.RowsAffected()
|
||||
return n == 1, err
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestANoticeIsClaimedByOneOfThoseAskingAtOnce(t *testing.T) {
|
||||
db := capacityStore(t)
|
||||
at := time.Date(2026, 10, 10, 9, 0, 0, 0, time.UTC)
|
||||
var yes atomic.Int64
|
||||
var wg sync.WaitGroup
|
||||
for range 8 {
|
||||
wg.Go(func() {
|
||||
ok, err := db.ClaimNotice(t.Context(), "budget_spent\x00team:t1", at, time.Hour)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if ok {
|
||||
yes.Add(1)
|
||||
}
|
||||
})
|
||||
}
|
||||
wg.Wait()
|
||||
if yes.Load() != 1 {
|
||||
t.Fatalf("%d of eight claimed the same notice", yes.Load())
|
||||
}
|
||||
if ok, _ := db.ClaimNotice(t.Context(), "budget_spent\x00team:t1", at.Add(59*time.Minute), time.Hour); ok {
|
||||
t.Error("claimed again within the hour")
|
||||
}
|
||||
if ok, _ := db.ClaimNotice(t.Context(), "budget_spent\x00team:t2", at, time.Hour); !ok {
|
||||
t.Error("another subject could not be claimed")
|
||||
}
|
||||
if ok, _ := db.ClaimNotice(t.Context(), "budget_spent\x00team:t1", at.Add(61*time.Minute), time.Hour); !ok {
|
||||
t.Error("not claimable once the hour had passed")
|
||||
}
|
||||
}
|
||||
@@ -69,7 +69,7 @@ type rollupKey struct {
|
||||
// rollupRow is what a batch adds to one row.
|
||||
type rollupRow struct {
|
||||
requests, errors, limited, prompt int64
|
||||
completion, cost int64
|
||||
completion, cost, images, meshes int64
|
||||
latency, queue, ttft, ttftN, last int64
|
||||
lat, ttftH []int64
|
||||
}
|
||||
@@ -98,6 +98,8 @@ func rollupBatch(t tier, records []Usage) (map[rollupKey]*rollupRow, []rollupKey
|
||||
row.prompt += r.PromptTokens
|
||||
row.completion += r.CompletionTokens
|
||||
row.cost += r.CostMicros
|
||||
row.images += r.Images
|
||||
row.meshes += r.Meshes
|
||||
row.latency += r.LatencyMs
|
||||
row.queue += r.QueueMs
|
||||
row.lat[latencyBucket(r.LatencyMs)]++
|
||||
@@ -116,7 +118,7 @@ var rollupColumns = func() []string {
|
||||
n := len(LatencyBounds) + 1
|
||||
cols := []string{"bucket", "key_id", "user_id", "project_id", "model", "served_model",
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros",
|
||||
"latency_ms_sum", "queue_ms_sum", "ttft_ms_sum", "ttft_n", "last_ts"}
|
||||
"latency_ms_sum", "queue_ms_sum", "ttft_ms_sum", "ttft_n", "last_ts", "images", "meshes"}
|
||||
for i := range n {
|
||||
cols = append(cols, fmt.Sprintf("lat_%d", i))
|
||||
}
|
||||
@@ -156,7 +158,7 @@ func writeRollup(ctx context.Context, tx conn, records []Usage) error {
|
||||
r := rows[k]
|
||||
args := []any{k.bucket, k.keyID, k.userID, k.projectID, k.model, k.servedModel,
|
||||
r.requests, r.errors, r.limited, r.prompt, r.completion, r.cost,
|
||||
r.latency, r.queue, r.ttft, r.ttftN, r.last}
|
||||
r.latency, r.queue, r.ttft, r.ttftN, r.last, r.images, r.meshes}
|
||||
for _, v := range r.lat {
|
||||
args = append(args, v)
|
||||
}
|
||||
@@ -187,11 +189,11 @@ func usageSince(since time.Time) (string, []any) {
|
||||
from := millis(since)
|
||||
to5m, to1h, to1d := tier5m.ceil(since), tier1h.ceil(since), tier1d.ceil(since)
|
||||
const cols = `key_id, user_id, project_id, model, served_model, requests, errors, limited,
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms_sum, last_ts`
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms_sum, last_ts, images, meshes`
|
||||
return `(SELECT key_id, user_id, project_id, model, served_model, 1 AS requests,
|
||||
CASE WHEN status >= 400 THEN 1 ELSE 0 END AS errors,
|
||||
CASE WHEN status = 429 THEN 1 ELSE 0 END AS limited,
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms AS latency_ms_sum, ts AS last_ts
|
||||
prompt_tokens, completion_tokens, cost_micros, latency_ms AS latency_ms_sum, ts AS last_ts, images, meshes
|
||||
FROM usage WHERE ts >= ? AND ts < ?
|
||||
UNION ALL SELECT ` + cols + ` FROM usage_5m WHERE bucket >= ? AND bucket < ?
|
||||
UNION ALL SELECT ` + cols + ` FROM usage_1h WHERE bucket >= ? AND bucket < ?
|
||||
|
||||
@@ -442,8 +442,8 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
|
||||
`INSERT INTO usage (ts, key_id, project_id, model, served_model,
|
||||
prompt_tokens, completion_tokens, cost_micros, status, latency_ms, user_id,
|
||||
request_id, replica, queue_ms, ttft_ms,
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit, images)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit, images, meshes)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -457,7 +457,7 @@ func (s *DB) WriteUsage(ctx context.Context, records []Usage) error {
|
||||
r.Model, r.ServedModel, r.PromptTokens, r.CompletionTokens,
|
||||
r.CostMicros, r.Status, r.LatencyMs, r.UserID,
|
||||
r.RequestID, r.Replica, r.QueueMs, r.TTFTMs,
|
||||
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardRule, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit, r.Images); err != nil {
|
||||
r.GuardAction, r.GuardPolicy, r.GuardCategory, r.GuardRule, r.GuardMs, r.EndUser, tagsJSON(r.Tags), hit, r.Images, r.Meshes); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -871,12 +871,12 @@ func (s *DB) UsageByRequest(ctx context.Context, requestID string) (Usage, bool,
|
||||
err := s.db.QueryRowContext(ctx,
|
||||
`SELECT ts, key_id, project_id, COALESCE(user_id, ''), model, served_model, prompt_tokens,
|
||||
completion_tokens, cost_micros, status, latency_ms, request_id, replica, queue_ms, ttft_ms,
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit, images
|
||||
guard_action, guard_policy, guard_category, guard_rule, guard_ms, end_user, tags, cache_hit, images, meshes
|
||||
FROM usage WHERE request_id = ? ORDER BY ts DESC LIMIT 1`, requestID).
|
||||
Scan(&ts, &u.KeyID, &u.ProjectID, &u.UserID, &u.Model, &u.ServedModel, &u.PromptTokens,
|
||||
&u.CompletionTokens, &u.CostMicros, &u.Status, &u.LatencyMs, &u.RequestID, &u.Replica,
|
||||
&u.QueueMs, &u.TTFTMs, &u.GuardAction, &u.GuardPolicy, &u.GuardCategory, &u.GuardRule, &u.GuardMs,
|
||||
&u.EndUser, &tags, &hit, &u.Images)
|
||||
&u.EndUser, &tags, &hit, &u.Images, &u.Meshes)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return Usage{}, false, nil
|
||||
}
|
||||
|
||||
@@ -243,6 +243,8 @@ type Usage struct {
|
||||
CompletionTokens int64
|
||||
// Images is how many pictures were made, for a request for them.
|
||||
Images int64
|
||||
// Meshes is how many 3D models were made.
|
||||
Meshes int64
|
||||
// CostMicros is what this request spent against the budget.
|
||||
CostMicros int64
|
||||
Status int
|
||||
|
||||
@@ -96,6 +96,7 @@ type AccountPages struct {
|
||||
// images says whether some model makes pictures; nil when the
|
||||
// pictures page is not mounted.
|
||||
images func() bool
|
||||
meshes func() bool
|
||||
guardLibrary func() guard.Config
|
||||
// guardFor names the guardrail policies in force for someone, and
|
||||
// whether they keep nothing; nil when there are none.
|
||||
|
||||
+3
-1
@@ -40,7 +40,9 @@ func contentSecurityPolicy(page []byte) string {
|
||||
"script-src 'self'" + prefixSpace(strings.Join(hashes, " ")) + "; " +
|
||||
"style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; " +
|
||||
"font-src 'self' https://fonts.gstatic.com; " +
|
||||
"img-src 'self' data: https:; " + // model cards show their images
|
||||
// Model cards show their images; a mesh's viewer hands its textures
|
||||
// to the browser as blob: URLs.
|
||||
"img-src 'self' data: blob: https:; " +
|
||||
"connect-src 'self'; " +
|
||||
"object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'"
|
||||
}
|
||||
|
||||
@@ -35,3 +35,19 @@ func Images(user store.User, models []ChatModel, contexts []ChatContext, current
|
||||
// SetImages has the menu offer the pictures page while some model makes
|
||||
// them. Asked on every page, so it must be cheap: the catalog, not a request.
|
||||
func (p *AccountPages) SetImages(available func() bool) { p.images = available }
|
||||
|
||||
// Meshes renders the page that makes 3D models from a picture.
|
||||
func Meshes(user store.User, models []ChatModel, contexts []ChatContext, current, version string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
noStore(w)
|
||||
render(w, r, "meshes", imagesData{
|
||||
Nav: navFrom(r, "meshes"),
|
||||
Contexts: contexts, Context: current,
|
||||
User: user, Models: models, CSRF: csrfFor(r),
|
||||
Version: version, Now: time.Now().UTC().Format(time.RFC3339),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// SetMeshes has the menu offer the 3D page while some model makes meshes.
|
||||
func (p *AccountPages) SetMeshes(available func() bool) { p.meshes = available }
|
||||
@@ -34,3 +34,28 @@ func TestThePicturesPageDrawsInEveryLanguage(t *testing.T) {
|
||||
t.Error("no models, and the page does not say so")
|
||||
}
|
||||
}
|
||||
|
||||
func TestThe3DPageDrawsInEveryLanguageWithItsViewer(t *testing.T) {
|
||||
for _, lang := range []string{"en", "tr"} {
|
||||
req := httptest.NewRequest("GET", "/3d", nil)
|
||||
req.Header.Set("Accept-Language", lang)
|
||||
rec := httptest.NewRecorder()
|
||||
Meshes(store.User{Email: "a@b.c"}, []ChatModel{{Name: "triposr"}},
|
||||
[]ChatContext{{Value: "personal", Label: "Personal"}}, "personal", "v0")(rec, req)
|
||||
page := rec.Body.String()
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("%s: %d %s", lang, rec.Code, page)
|
||||
}
|
||||
for _, want := range []string{`<option value="triposr">`, `id="picture"`, `/3d/generate`, `model-viewer.min.js?v=`} {
|
||||
if !strings.Contains(page, want) {
|
||||
t.Errorf("%s: %s missing", lang, want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(page, "meshes.") {
|
||||
t.Errorf("%s: an untranslated key is on the page", lang)
|
||||
}
|
||||
if csp := rec.Header().Get("Content-Security-Policy"); !strings.Contains(csp, "script-src 'self'") || !strings.Contains(csp, "blob:") {
|
||||
t.Errorf("policy %q", csp)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"meshes.title": "3D models",
|
||||
"meshes.picture": "Picture",
|
||||
"meshes.picture_hint": "one object, on a plain background; a PNG with a transparent background is best",
|
||||
"meshes.resolution": "Detail",
|
||||
"meshes.resolution_hint": "the model's grid: higher is finer and slower",
|
||||
"meshes.seed": "Seed",
|
||||
"meshes.make": "Make",
|
||||
"meshes.kept": "A 3D model is made as a job: it waits for the card, then takes a minute or more. It is kept for the files' retention, and you can leave this page and come back to the job by its id through the API.",
|
||||
"meshes.empty": "No model you may use makes 3D models right now.",
|
||||
"meshes.empty_why": "A model that makes them is a ComfyUI model in the catalog with <code>output: meshes</code>.",
|
||||
"meshes.js.queued": "Waiting for the card… {ahead} ahead",
|
||||
"meshes.js.queued_none": "Waiting for the card…",
|
||||
"meshes.js.running": "Making it… {node} {percent}%",
|
||||
"meshes.js.done": "Made in {s} s.",
|
||||
"meshes.js.failed": "Not made: {why}",
|
||||
"meshes.js.download": "download the .glb",
|
||||
"meshes.js.license": "Licence",
|
||||
"meshes.js.no_picture": "Choose a picture first."
|
||||
}
|
||||
@@ -12,5 +12,6 @@
|
||||
"nav.sign_out": "Sign out",
|
||||
"nav.docs": "Docs",
|
||||
"nav.settings": "Settings",
|
||||
"nav.images": "Pictures"
|
||||
"nav.images": "Pictures",
|
||||
"nav.meshes": "3D"
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"meshes.title": "3D modeller",
|
||||
"meshes.picture": "Resim",
|
||||
"meshes.picture_hint": "düz bir arka plan önünde tek bir nesne; arka planı saydam bir PNG en iyisi",
|
||||
"meshes.resolution": "Ayrıntı",
|
||||
"meshes.resolution_hint": "modelin ızgarası: yüksek değer daha ince ve daha yavaş",
|
||||
"meshes.seed": "Tohum",
|
||||
"meshes.make": "Üret",
|
||||
"meshes.kept": "3D model bir iş olarak üretilir: önce kartı bekler, sonra bir dakika ya da daha fazla sürer. Dosyaların saklama süresi boyunca tutulur; sayfadan ayrılıp işe API üzerinden kimliğiyle dönebilirsiniz.",
|
||||
"meshes.empty": "Şu anda kullanabileceğiniz, 3D model üreten bir model yok.",
|
||||
"meshes.empty_why": "3D model üreten model, katalogda <code>output: meshes</code> olan bir ComfyUI modelidir.",
|
||||
"meshes.js.queued": "Kart bekleniyor… önünde {ahead} iş var",
|
||||
"meshes.js.queued_none": "Kart bekleniyor…",
|
||||
"meshes.js.running": "Üretiliyor… {node} %{percent}",
|
||||
"meshes.js.done": "{s} sn'de üretildi.",
|
||||
"meshes.js.failed": "Üretilemedi: {why}",
|
||||
"meshes.js.download": ".glb dosyasını indir",
|
||||
"meshes.js.license": "Lisans",
|
||||
"meshes.js.no_picture": "Önce bir resim seçin."
|
||||
}
|
||||
@@ -12,5 +12,6 @@
|
||||
"nav.sign_out": "Çıkış yap",
|
||||
"nav.docs": "Belgeler",
|
||||
"nav.settings": "Ayarlar",
|
||||
"nav.images": "Görseller"
|
||||
"nav.images": "Görseller",
|
||||
"nav.meshes": "3D"
|
||||
}
|
||||
+4
-2
@@ -35,8 +35,9 @@ type Nav struct {
|
||||
// Org offers the organization page: to an administrator, and to anyone
|
||||
// who manages an organization.
|
||||
Org bool
|
||||
// Images offers the pictures page, while some model makes them.
|
||||
Images bool
|
||||
// Images offers the pictures page, while some model makes them; Meshes
|
||||
// the 3D page.
|
||||
Images, Meshes bool
|
||||
}
|
||||
|
||||
type navKey struct{}
|
||||
@@ -53,6 +54,7 @@ func (p *AccountPages) navFor(r *http.Request, user store.User, current string)
|
||||
Settings: p.hasSettings && user.IsAdmin,
|
||||
CSRF: csrfFor(r),
|
||||
Images: p.images != nil && p.images(),
|
||||
Meshes: p.meshes != nil && p.meshes(),
|
||||
}
|
||||
// Only when there is an organization to show: a link that leads to "there
|
||||
// are none" on every page is noise.
|
||||
|
||||
+10
-4
@@ -33,6 +33,8 @@ type OIDCStore interface {
|
||||
AllOrgs(ctx context.Context) ([]store.Org, error)
|
||||
AllTeams(ctx context.Context) ([]store.TeamPath, error)
|
||||
AllProjects(ctx context.Context) ([]store.ProjectPath, error)
|
||||
// ClusterSecret is a secret every control plane reads alike.
|
||||
ClusterSecret(ctx context.Context, name string) (string, error)
|
||||
}
|
||||
|
||||
// oidcLogin is sign-in through an identity provider.
|
||||
@@ -59,11 +61,15 @@ func (p *AccountPages) RegisterOIDC(mux *http.ServeMux, cfg config.OIDC, st OIDC
|
||||
}
|
||||
secret = strings.TrimSpace(string(b))
|
||||
}
|
||||
key := make([]byte, 32)
|
||||
if _, err := rand.Read(key); err != nil {
|
||||
return err
|
||||
// From the store, not made here: a sign-in begun on one control plane
|
||||
// comes back from the provider to whichever one the balancer picks, and
|
||||
// a key of this process's own made that one refuse it — with two behind
|
||||
// a balancer, half of all sign-ins.
|
||||
key, err := st.ClusterSecret(context.Background(), "oidc-state")
|
||||
if err != nil {
|
||||
return fmt.Errorf("the key that carries a sign-in across the provider: %w", err)
|
||||
}
|
||||
o := &oidcLogin{cfg: cfg, secret: secret, st: st, key: key}
|
||||
o := &oidcLogin{cfg: cfg, secret: secret, st: st, key: []byte(key)}
|
||||
p.oidc = o
|
||||
mux.HandleFunc("GET /login/oidc", p.oidcStart)
|
||||
mux.HandleFunc("GET /login/oidc/callback", p.oidcCallback)
|
||||
|
||||
@@ -98,6 +98,9 @@ func (m *oidcStoreMem) AllTeams(context.Context) ([]store.TeamPath, error) {
|
||||
return []store.TeamPath{{ID: "t1", Name: "ml", OrgName: "Acme"}}, nil
|
||||
}
|
||||
func (m *oidcStoreMem) AllProjects(context.Context) ([]store.ProjectPath, error) { return nil, nil }
|
||||
func (m *oidcStoreMem) ClusterSecret(_ context.Context, name string) (string, error) {
|
||||
return "the store's " + name, nil
|
||||
}
|
||||
|
||||
type sessionAccounts struct{ fakeAccounts }
|
||||
|
||||
@@ -107,6 +110,13 @@ func (sessionAccounts) OpenSession(context.Context, string) (string, string, err
|
||||
|
||||
// signIn goes through /login/oidc and back, as a browser would.
|
||||
func signIn(t *testing.T, mux http.Handler, idp *fakeIdP, tamper bool) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
return signInAcross(t, mux, mux, idp, tamper)
|
||||
}
|
||||
|
||||
// signInAcross begins a sign-in on one gateway and comes back from the
|
||||
// provider to another, as a balancer in front of two may send it.
|
||||
func signInAcross(t *testing.T, mux, back http.Handler, idp *fakeIdP, tamper bool) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
start := httptest.NewRecorder()
|
||||
mux.ServeHTTP(start, httptest.NewRequest("GET", "/login/oidc", nil))
|
||||
@@ -127,7 +137,7 @@ func signIn(t *testing.T, mux http.Handler, idp *fakeIdP, tamper bool) *httptest
|
||||
cb.AddCookie(c)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, cb)
|
||||
back.ServeHTTP(rec, cb)
|
||||
return rec
|
||||
}
|
||||
|
||||
@@ -172,3 +182,28 @@ func TestSigningInThroughAProvider(t *testing.T) {
|
||||
t.Errorf("unknown without create_accounts: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestASignInBegunOnOneControlPlaneEndsOnTheOther(t *testing.T) {
|
||||
// The cookie carrying a sign-in across the provider was signed with a
|
||||
// key each process made for itself, so with two control planes behind a
|
||||
// balancer a callback reaching the other one was refused as tampered.
|
||||
idp := newFakeIdP(t)
|
||||
st := &oidcStoreMem{users: map[string]store.User{}, roles: map[string]string{}}
|
||||
cfg := config.OIDC{Issuer: idp.srv.URL, ClientID: "llmgw", CreateAccounts: true}
|
||||
var muxes []*http.ServeMux
|
||||
for range 2 {
|
||||
p := NewAccountPages(sessionAccounts{}, nil, nil, "v0", slog.New(slog.NewTextHandler(io.Discard, nil)))
|
||||
mux := http.NewServeMux()
|
||||
if err := p.RegisterOIDC(mux, cfg, st); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
muxes = append(muxes, mux)
|
||||
}
|
||||
idp.claims = map[string]any{"email": "dan@example.com", "email_verified": true}
|
||||
if rec := signInAcross(t, muxes[0], muxes[1], idp, false); rec.Code != http.StatusSeeOther {
|
||||
t.Fatalf("a callback on the other control plane: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if rec := signInAcross(t, muxes[0], muxes[1], idp, true); rec.Code != http.StatusForbidden {
|
||||
t.Errorf("a tampered state on the other control plane: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -927,3 +927,9 @@ a.sel-link.small { font-size: .78rem; color: var(--muted); margin-left: .4rem; }
|
||||
.images-gallery img { display: block; width: 100%; height: auto; }
|
||||
.images-gallery figcaption { font-size: .8rem; color: var(--muted); padding: .45rem .6rem 0; }
|
||||
.images-prompt { font-size: .82rem; margin: .25rem .6rem .55rem; overflow-wrap: anywhere; }
|
||||
|
||||
/* The 3D page: the picture asked about, and each mesh in a viewer. */
|
||||
.meshes-picture { display: block; margin: .6rem 0; font-size: .85rem; font-weight: 600; }
|
||||
.meshes-picture input { display: block; margin-top: .3rem; font-weight: 400; }
|
||||
.meshes-preview { max-width: min(100%, 16rem); max-height: 16rem; border: 1px solid var(--line); border-radius: 6px; }
|
||||
.meshes-viewer { display: block; width: 100%; height: 22rem; background: var(--card); }
|
||||
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+1118
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,178 @@
|
||||
{{define "meshes"}}<!doctype html>
|
||||
<html lang="{{lang}}"><head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>{{t "meshes.title"}} · llmgw</title><link rel="stylesheet" href="{{stylesheet}}">
|
||||
<script defer src="{{asset "pages.js"}}"></script>
|
||||
<script type="module" src="{{asset "model-viewer.min.js"}}"></script>
|
||||
<meta name="csrf" content="{{.CSRF}}">
|
||||
</head><body><div class="wrap">
|
||||
<header>
|
||||
<h1>{{t "meshes.title"}}</h1>
|
||||
<p class="sub">{{.User.Email}}</p>
|
||||
{{template "nav" .Nav}}
|
||||
</header>
|
||||
|
||||
{{if gt (len .Contexts) 1}}
|
||||
<form method="get" action="/3d" class="chat-context">
|
||||
<label class="inline-label">{{t "chat.context.using"}}
|
||||
<select id="context" name="context" data-autosubmit>
|
||||
{{range .Contexts}}<option value="{{.Value}}"{{if eq .Value $.Context}} selected{{end}}>{{.Label}}</option>{{end}}
|
||||
</select></label>
|
||||
<noscript translate="no"><button type="submit" class="link">{{t "chat.context.switch"}}</button></noscript>
|
||||
</form>
|
||||
{{else if .Contexts}}{{with index .Contexts 0}}<input type="hidden" id="context" value="{{.Value}}">{{end}}{{end}}
|
||||
|
||||
{{if .Models}}
|
||||
<form id="make" class="images-form">
|
||||
<div class="chat-bar">
|
||||
<label class="inline-label">{{t "chat.model"}}
|
||||
<select id="model">{{range .Models}}<option value="{{.Name}}">{{.Name}}{{with .Runs}} · {{.}}{{end}}</option>{{end}}</select>
|
||||
</label>
|
||||
<label class="inline-label">{{t "meshes.resolution"}}
|
||||
<select id="resolution" title="{{t "meshes.resolution_hint"}}">
|
||||
<option value="">—</option><option>192</option><option>256</option><option>384</option><option>512</option>
|
||||
</select></label>
|
||||
<label class="inline-label">{{t "meshes.seed"}}
|
||||
<input type="number" id="seed" min="0" step="1" class="images-num"></label>
|
||||
</div>
|
||||
<label class="meshes-picture">{{t "meshes.picture"}} <span class="opt">{{t "meshes.picture_hint"}}</span>
|
||||
<input type="file" id="picture" accept="image/png,image/jpeg,image/webp"></label>
|
||||
<img id="preview" class="meshes-preview" alt="" hidden>
|
||||
<div class="chat-composer"><button type="submit" id="go">{{t "meshes.make"}}</button></div>
|
||||
</form>
|
||||
<p id="status" class="note" aria-live="polite"></p>
|
||||
<div id="gallery" class="images-gallery"></div>
|
||||
<p class="note">{{t "meshes.kept"}}</p>
|
||||
{{else}}
|
||||
<p class="empty">{{t "meshes.empty"}}</p>
|
||||
<p class="note">{{th "meshes.empty_why"}}</p>
|
||||
{{end}}
|
||||
|
||||
<footer><span>llmgw {{.Version}}</span><span>{{.Now}}</span>{{template "langs"}}</footer>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const say = {
|
||||
queued: {{t "meshes.js.queued"}},
|
||||
queuedNone: {{t "meshes.js.queued_none"}},
|
||||
running: {{t "meshes.js.running"}},
|
||||
done: {{t "meshes.js.done"}},
|
||||
failed: {{t "meshes.js.failed"}},
|
||||
download: {{t "meshes.js.download"}},
|
||||
license: {{t "meshes.js.license"}},
|
||||
noPicture: {{t "meshes.js.no_picture"}},
|
||||
decimal: {{t "_meta.decimal"}},
|
||||
};
|
||||
function fill(s, vals) { return s.replace(/\{(\w+)\}/g, (m, k) => (k in vals ? String(vals[k]) : m)); }
|
||||
function fixed(n, places) { return n.toFixed(places).replace(".", say.decimal); }
|
||||
// A 3D model is a job: asked for, then asked after until it is made. The
|
||||
// mesh is shown from the gateway itself, which the page's policy allows.
|
||||
(function () {
|
||||
const form = document.getElementById("make");
|
||||
if (!form) return;
|
||||
const csrf = document.querySelector('meta[name="csrf"]').content;
|
||||
const $ = (id) => document.getElementById(id);
|
||||
const status = $("status"), gallery = $("gallery"), go = $("go"), preview = $("preview");
|
||||
let picture = "";
|
||||
const ctx = () => ($("context") ? $("context").value : "");
|
||||
|
||||
$("picture").addEventListener("change", () => {
|
||||
const f = $("picture").files[0];
|
||||
if (!f) return;
|
||||
const rd = new FileReader();
|
||||
rd.addEventListener("load", () => {
|
||||
picture = rd.result;
|
||||
preview.src = picture;
|
||||
preview.hidden = false;
|
||||
});
|
||||
rd.readAsDataURL(f);
|
||||
});
|
||||
|
||||
function describe(j) {
|
||||
const p = j.progress || {};
|
||||
if (j.status === "queued" || p.state === "queued" || !p.state) {
|
||||
return p.ahead ? fill(say.queued, { ahead: p.ahead }) : say.queuedNone;
|
||||
}
|
||||
return fill(say.running, { node: p.node_class || "", percent: p.percent || 0 });
|
||||
}
|
||||
|
||||
async function follow(id, started) {
|
||||
const q = ctx() ? "?context=" + encodeURIComponent(ctx()) : "";
|
||||
for (;;) {
|
||||
const resp = await fetch("/3d/jobs/" + encodeURIComponent(id) + q);
|
||||
const j = await resp.json().catch(() => ({}));
|
||||
if (!resp.ok) throw new Error((j.error && j.error.message) || String(resp.status));
|
||||
if (j.status === "completed") return j;
|
||||
if (["failed", "cancelled", "expired"].includes(j.status)) {
|
||||
throw new Error((j.error && j.error.message) || j.status);
|
||||
}
|
||||
status.textContent = describe(j);
|
||||
await new Promise((r) => setTimeout(r, 2000));
|
||||
}
|
||||
}
|
||||
|
||||
function show(j, took) {
|
||||
const q = ctx() ? "?context=" + encodeURIComponent(ctx()) : "";
|
||||
const src = "/3d/jobs/" + encodeURIComponent(j.id) + "/file" + q;
|
||||
const fig = document.createElement("figure");
|
||||
const mv = document.createElement("model-viewer");
|
||||
mv.setAttribute("src", src);
|
||||
mv.setAttribute("camera-controls", "");
|
||||
mv.setAttribute("auto-rotate", "");
|
||||
mv.setAttribute("shadow-intensity", "0.6");
|
||||
mv.className = "meshes-viewer";
|
||||
const cap = document.createElement("figcaption");
|
||||
const line = document.createElement("span");
|
||||
line.textContent = j.model + " · " + j.id + " · " + fill(say.done, { s: fixed(took, 0) });
|
||||
const dl = document.createElement("a");
|
||||
dl.href = src;
|
||||
dl.download = j.id + ".glb";
|
||||
dl.className = "link";
|
||||
dl.textContent = say.download;
|
||||
cap.append(line, " ", dl);
|
||||
fig.append(mv, cap);
|
||||
if (j.license) {
|
||||
const lic = document.createElement("p");
|
||||
lic.className = "images-prompt";
|
||||
lic.textContent = say.license + ": " + j.license.name + (j.license.notice ? " — " + j.license.notice : "");
|
||||
fig.append(lic);
|
||||
}
|
||||
gallery.prepend(fig);
|
||||
}
|
||||
|
||||
form.addEventListener("submit", async (e) => {
|
||||
e.preventDefault();
|
||||
if (go.disabled) return;
|
||||
if (!picture) { status.textContent = say.noPicture; return; }
|
||||
const body = { model: $("model").value, image: picture, context: ctx() };
|
||||
if ($("seed").value !== "") body.seed = Number($("seed").value);
|
||||
if ($("resolution").value !== "") body.resolution = Number($("resolution").value);
|
||||
go.disabled = true;
|
||||
status.classList.remove("error");
|
||||
status.textContent = say.queuedNone;
|
||||
const start = performance.now();
|
||||
try {
|
||||
const resp = await fetch("/3d/generate", {
|
||||
method: "POST", headers: { "Content-Type": "application/json", "X-CSRF-Token": csrf },
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
const text = await resp.text();
|
||||
let j = null;
|
||||
try { j = JSON.parse(text); } catch (_) {}
|
||||
if (!resp.ok || !j || !j.id) {
|
||||
throw new Error(String((j && j.error && j.error.message) || text.trim() || resp.status));
|
||||
}
|
||||
const done = await follow(j.id, start);
|
||||
const took = (performance.now() - start) / 1000;
|
||||
status.textContent = fill(say.done, { s: fixed(took, 0) });
|
||||
show(done, took);
|
||||
} catch (err) {
|
||||
status.textContent = fill(say.failed, { why: err.message });
|
||||
status.classList.add("error");
|
||||
} finally {
|
||||
go.disabled = false;
|
||||
}
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body></html>{{end}}
|
||||
@@ -31,6 +31,9 @@
|
||||
{{- if .Images}}
|
||||
<a href="/images"{{if eq .Current "images"}} aria-current="page"{{end}}>{{t "nav.images"}}</a>
|
||||
{{- end}}
|
||||
{{- if .Meshes}}
|
||||
<a href="/3d"{{if eq .Current "meshes"}} aria-current="page"{{end}}>{{t "nav.meshes"}}</a>
|
||||
{{- end}}
|
||||
<a href="/profile"{{if eq .Current "keys"}} aria-current="page"{{end}}>{{t "nav.account"}}</a>
|
||||
<a href="/docs"{{if eq .Current "docs"}} aria-current="page"{{end}}>{{t "nav.docs"}}</a>
|
||||
<form method="post" action="/logout" class="inline">
|
||||
|
||||
Reference in new issue
Block a user