Compare commits
51
Commits
No files matched your search
+249
-13
@@ -386,10 +386,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
tenant-admin / billing; `ctl audit`. Depends on A5 for a real actor.
|
||||
2. **Done (branch `features-c`):** OIDC sign-in (`auth.oidc`, PKCE,
|
||||
verified email, roles and memberships from groups, `no_passwords`) and
|
||||
SCIM 2.0 (`/scim/v2` Users and Groups). Left: the sign-in cookie is
|
||||
signed with a per-process key, so with two control planes behind a
|
||||
balancer the callback must reach the one that started it (sticky
|
||||
sessions) until the key comes from the store. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||||
SCIM 2.0 (`/scim/v2` Users and Groups). The sign-in cookie's key comes
|
||||
from the store (2026-10-10), so a callback may reach either control
|
||||
plane behind a balancer. **OIDC/SSO login (M), then SCIM (M).** `internal/account/service.go`,
|
||||
`internal/web/account.go`; map IdP groups to org/team/admin.
|
||||
3. **Done (branch `features-c`):** key lifetime limits (gateway, org, team,
|
||||
project), `keys rotate --overlap`, kinds and expiry in `keys list`.
|
||||
@@ -405,8 +404,9 @@ portal is not done — the page shows; the console and ctl change.
|
||||
lost, budget spent, update available; signed JSON or Slack; retries,
|
||||
hourly de-duplication), `ctl notify test`, Alertmanager receiver
|
||||
templates. Left: `server init` and the console do not yet ask for a
|
||||
receiver — the templates are filled in by hand; each control plane sends
|
||||
budget events on its own, so two behind a balancer may both send one.
|
||||
receiver — the templates are filled in by hand. Events are claimed in
|
||||
the store before they are sent (2026-10-10), so two control planes send
|
||||
each once.
|
||||
**Alerting out of the box (S).** `deploy/monitoring/alertmanager.yml`
|
||||
ships `receiver: none`; add Slack/email/webhook/PagerDuty receiver
|
||||
templates driven by `server init` or the console; llmgw-native webhooks
|
||||
@@ -569,9 +569,12 @@ portal is not done — the page shows; the console and ctl change.
|
||||
client tests against a fake admin API and the OpenAPI document;
|
||||
acceptance tests with Terraform and OpenTofu against the fake; one
|
||||
apply, empty plan, in-place change and destroy against a real `llmgw
|
||||
server` run by hand. Left: organizations, teams, projects, people and
|
||||
keys are not in the admin API (they are `llmgw org`/`users`/`keys`
|
||||
against the store), so not in the provider; a model's definition beyond
|
||||
server` run by hand. Organizations, teams, projects, keys and
|
||||
memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
|
||||
`teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
|
||||
provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
|
||||
`llmgw_membership`); accounts are still made by sign-up, invitation or
|
||||
OIDC/SCIM. Left: a model's definition beyond
|
||||
scaling, class, owner and adapters is changed by replacing it (the API
|
||||
has no edit, and `resources` is not used yet); `canary`, `slo`,
|
||||
`scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
|
||||
@@ -793,10 +796,19 @@ leader 30s later with /health answering throughout; rollback did the same.
|
||||
facility state from the store (it kept what it read at start).
|
||||
- Retires `appendGuardIfMissing` (server init): the import does it.
|
||||
|
||||
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
|
||||
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
|
||||
`firewall.enabled` in the cluster's settings reaches nodes over the agent
|
||||
stream and control planes through llmgw-firewall-sync.path. Left: the mirror
|
||||
machine and store members, the allowed set from membership, the other files.
|
||||
stream and control planes through llmgw-firewall-sync.path. Since
|
||||
2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
|
||||
writes `.firewall.json`, the same path unit applies it) and store members
|
||||
that run no llmgw (the leader keeps it in `cluster_state`, a root timer
|
||||
reads the member's own copy with psql); `firewall_networks` reach all of
|
||||
them, and the mirror lets in the network of every node and control plane
|
||||
the leader knows. Tried on the PoC: off and on, and a network added, on
|
||||
control planes, mirror, store member and nodes. Left: the allowed set
|
||||
replaced by membership rather than added to (a network taken out of the
|
||||
settings stays let in), the other files (the mirror's unit flags,
|
||||
keepalived, etcd and Patroni configuration).
|
||||
|
||||
Every machine runs the agent (root) with roles — node, control plane, mirror,
|
||||
store member — and the agent renders and applies what was written once by
|
||||
@@ -863,6 +875,12 @@ Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
|
||||
partial HNSW index per collection on its organization's partition
|
||||
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
|
||||
20k rows: partition pruned, that index used, policy applied.
|
||||
Later (scale, 2026-10-08): a partial GIN per collection too
|
||||
(`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
|
||||
on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
|
||||
candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
|
||||
it); the text arm takes candidates by `ts_rank` in a subquery and scores
|
||||
only those.
|
||||
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
|
||||
(set with set_config per transaction), one role — the table owner — not
|
||||
a separate one. Chunks are deleted by the foreign-key cascade from their
|
||||
@@ -1005,7 +1023,7 @@ a request without the header, `none` turns it off, an explicit header with
|
||||
k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
|
||||
are not audited per request (the header and the kept request record them).
|
||||
|
||||
### F4. Adapter training (L, after F1–F3)
|
||||
### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
|
||||
A run-to-completion job placed by the control plane like a replica and
|
||||
supervised by the agent like an engine, in a Python environment the agent
|
||||
installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
|
||||
@@ -1013,9 +1031,227 @@ a cluster model through `/v1/batches`, reviewed by people. Output to the
|
||||
mirror, into the model's `adapters`, through canary and a quality suite.
|
||||
Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
|
||||
Never train on a collection some users of the adapter may not read.
|
||||
Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
|
||||
LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
|
||||
cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
|
||||
venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
|
||||
model wrote from the nine Demo/politikalar documents, through the gateway.
|
||||
On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
|
||||
4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
|
||||
documents (the meal card's amount, the office day). The same nine through
|
||||
retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
|
||||
facts come from F1–F3; a bigger base on a free card would do better but not
|
||||
change that. The training itself fits a small slot on a shared card, which
|
||||
the job design can rely on for small bases.
|
||||
|
||||
Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
|
||||
with each pair's collection); train_jobs/train_artifacts (migrations 0037,
|
||||
0038 owner); internal/control/train (queue, one job per node, artifacts in
|
||||
1 MiB pieces, a node that goes fails its job); internal/agent/train (the
|
||||
newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
|
||||
accelerate 1.15.0; llama.cpp's conversion code for the installed release,
|
||||
untarred from its source; the card with the most free memory, if enough;
|
||||
runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
|
||||
download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
|
||||
publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
|
||||
adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
|
||||
router (404, hidden from /v1/models); a job's owner defaults to its single
|
||||
collection's owner, must be able to read every collection the pairs came
|
||||
from, and pairs from a knowledge base are never public. Tried on the PoC:
|
||||
dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
|
||||
138 steps in about a minute) and sent back PEFT and GGUF.
|
||||
Left: a job dies with its agent's stream (no resume); no console screen for
|
||||
training yet; quality gating of a published adapter is by hand (ctl eval).
|
||||
|
||||
---
|
||||
|
||||
### F5. Large imports (M) — measured 2026-10-09, open
|
||||
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
|
||||
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
|
||||
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
|
||||
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||||
at a third of its CPUs.
|
||||
- An embedding model's profile with many slots by default (`--parallel`
|
||||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||||
longest chunk), or `ctl model serve` choosing that for a model with
|
||||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||||
the text and vector go up again (~11 KB).
|
||||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||||
0.7) and a larger store machine.
|
||||
- Done: a batch of embeddings checked text by text against the context
|
||||
(branch `embeddings-batch`); it was refused as one over-long prompt.
|
||||
- Done: `query_prefix` on an embedding model, before questions only.
|
||||
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
|
||||
tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
|
||||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||||
took 3 h. Done (2026-10-10, branch `admission-bulk`): a source per
|
||||
profile, so an embedding model's GPU profile runs vLLM and its CPU
|
||||
profile llama.cpp; a band's own `queue_ttl` for bulk imports, and
|
||||
`Retry-After` on a busy refusal, which is what had turned passages away
|
||||
(llmgw-kb retries only what says when to).
|
||||
- Retrieval on 40 questions written from the corpus, document in the top
|
||||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||||
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
|
||||
chat model, and a reranker.
|
||||
|
||||
## G. Jobs: 3D generation through ComfyUI (2026-10-10)
|
||||
|
||||
A request for a 3D model takes minutes on the one 24 GB card, and the card has to be
|
||||
emptied of the chat model first. That is not a request to hold a socket for
|
||||
— `internal/router/activate.go:46` already says so — it is a job: queued,
|
||||
answered with an id, asked after, its result downloaded. The job machinery
|
||||
is built first and tested without a GPU; the ComfyUI generalisation and the
|
||||
custom-node install come after. Decided 2026-10-10: a `jobs` table of its
|
||||
own rather than a one-line batch (the JSONL shape would be artificial and
|
||||
has nowhere for progress), and progress is shown, not just a status.
|
||||
|
||||
Which models (decided 2026-10-10): the assets go into a game sold on
|
||||
Steam, worldwide, the EU included, so the licence of the output decides
|
||||
before the quality does. Hunyuan3D (every version) is out: Tencent's
|
||||
licence excludes the EU, UK and South Korea, and whether a game shipped
|
||||
there with its outputs is "use in the Territory" is not a question to
|
||||
leave to a court. In, in this order: **TRELLIS.2** shape (MIT; the best
|
||||
geometry; its own texture baking goes through nvdiffrec and nvdiffrast,
|
||||
both under NVIDIA's non-commercial source licence, so that stage is not
|
||||
run), **Step1X-3D** (Apache 2.0, StepFun; shape and texture in one, the
|
||||
EU-safe textured path — its ComfyUI node and VRAM need are to be
|
||||
checked), **Direct3D-S2** (MIT, DreamTech; high-detail shape, has a
|
||||
ComfyUI node) if TRELLIS.2's detail falls short. Not in: TripoSR and
|
||||
InstantMesh (permissive but superseded), Stable Fast 3D (Stability's
|
||||
community licence, revenue-capped), SAM 3D Objects (Meta's own licence,
|
||||
to be read before it is considered). Steam's disclosure form (changed
|
||||
2026-01-16) lists pre-generated 3D models as content that must be
|
||||
declared on the store page: the licence notice G3 writes into the GLB is
|
||||
what the studio answers that form from.
|
||||
|
||||
### G1. Jobs: store, worker, endpoints (M) — done 2026-10-10 (branch `jobs-g1`)
|
||||
Built as written, with two choices: a job found running after its gateway
|
||||
died is run again from the start (the prompt id column is there for G2 to
|
||||
follow the same work instead), and the request is validated at the door
|
||||
only as far as costs nothing — key, model, allowlist, the request's shape
|
||||
— with limits, guardrails and admission applied when the worker replays it,
|
||||
said as the job's error. The result file's `content_type` is the job's; the
|
||||
files endpoint still serves every file as an octet stream.
|
||||
- Store: `jobs` (id, owner key/principal, kind, model, status, input
|
||||
(JSON), progress (JSON: queue position, percent, step), result file id,
|
||||
error, created/started/finished, lease owner/until), migrations for
|
||||
sqlite and postgres beside `internal/store/batches.go`, whose
|
||||
claim/renew/release lease pattern is copied, not shared: a job is one
|
||||
unit of work, a batch is many lines.
|
||||
- Worker: `internal/router/jobworker.go` after `batchworker.go`. Any
|
||||
gateway on the store claims a queued job, asks for the model with
|
||||
`activator.Want` and waits as long as it takes (no `ActivationWait`
|
||||
bound: nobody is holding a connection), runs it, stores the output
|
||||
through the files store, writes usage and price, marks it done. A
|
||||
gateway that dies mid-job loses its lease and another one picks it up;
|
||||
the ComfyUI prompt id is in the job row so the second gateway asks
|
||||
after the same work rather than queueing it again.
|
||||
- Endpoints: `POST /v1/3d/generations` validates like any inference
|
||||
request (key, allowlist, limits, admission, guardrails on the prompt),
|
||||
writes the job, answers 202 with `{id, status: "queued"}`;
|
||||
`GET /v1/3d/generations/{id}` returns status, progress and, when done,
|
||||
`output_file_id` and the usage; `POST .../{id}/cancel`; the file comes
|
||||
from `GET /v1/files/{id}/content`. Owner-scoped like batches. Results
|
||||
are kept for the batch files' retention and deleted with them: a
|
||||
departure from `images.go`'s "kept nowhere", because a mesh is a
|
||||
download, not a JSON field; the retention is the bound.
|
||||
- Done when: a job survives the gateway that took it being killed, a
|
||||
cancelled job leaves ComfyUI's queue (`comfy.Client.cancel`), two
|
||||
gateways never both run one job (test with the lease), and the usage
|
||||
row carries the job's price.
|
||||
|
||||
### G2. Progress from ComfyUI (S, after G1) — done 2026-10-10 (branch `jobs-3d`)
|
||||
Built as written: the gate's websocket under its own client id, the
|
||||
client_id of every /prompt rewritten, /progress/{prompt_id} on the gate,
|
||||
the router reading it and /queue into the job (only into a job of the same
|
||||
key). Left: a node that reports no steps (TripoSR's) shows its name and no
|
||||
percent; a vanished replica is said by the job failing, not by progress.
|
||||
The gate (`internal/comfy/gate.go`) passes seven paths and no websocket.
|
||||
Rather than open `/ws`, the gate subscribes to ComfyUI's own `/ws` once,
|
||||
with `golang.org/x/net/websocket` (already a dependency), under a client
|
||||
id of its own, and rewrites the `client_id` of every `/prompt` it
|
||||
forwards to that one. It keeps the last `executing`, `progress`
|
||||
(value/max/node) and `execution_error` message per prompt id and serves
|
||||
them at `GET /progress/{prompt_id}` on the gate; the router polls that
|
||||
beside `/history` and `/queue` and writes queue position, percent and the
|
||||
current node's class into the job's progress. The gate's buffer is
|
||||
bounded (the last N prompts) and the websocket reconnects. Done when a
|
||||
3D job shows "queued, 2 ahead" then "running 40%, step 12/30" in
|
||||
`ctl jobs` and on the account page, and a job whose replica vanished
|
||||
says so within one poll.
|
||||
|
||||
### G3. ComfyUI beyond pictures: 3D models and custom nodes (M, after G1) — done 2026-10-10 (branch `jobs-3d`), with TripoSR
|
||||
Built: output kinds, image upload through the gate, every output list
|
||||
fetched, OBJ to GLB, licence in /v1/models, the job and the GLB, per_mesh
|
||||
and usage.meshes (migration 0045), custom nodes cloned at their commit by
|
||||
the gate with packages beside the replica (a node's named packages at its
|
||||
range, the rest at ComfyUI's versions). Not as written: the model. On the
|
||||
PoC's Turing card TRELLIS.2 cannot run — it needs Meta's gated DINOv3,
|
||||
its published background remover (RMBG-2.0) is non-commercial, and its
|
||||
sparse attention and Direct3D-S2's need flash-attention (Ampere and
|
||||
newer). TripoSR (MIT) proves the chain on cuda01: a .glb from a picture
|
||||
in about 30 s. Left: background removal before TripoSR (BiRefNet, MIT,
|
||||
as a node), TRELLIS.2 or Direct3D-S2 on an Ampere card with an ungated
|
||||
encoder, a missing node refusing the replica by name.
|
||||
- Catalog: a model makes `pictures` or `meshes` (`MakesImages` becomes an
|
||||
output kind; `ServesComfyUI` stays). 3D weight folders in the gate's
|
||||
`modelFolders`; the workflow's placeholders gain `image` (the input
|
||||
picture, base64 in the request, written to the replica's `input/` by
|
||||
the gate through a new `POST /upload/image` the gate allows — or a
|
||||
base64-loading node, whichever TRELLIS.2's wrapper wants).
|
||||
- `comfy.Client.fetch` reads every output key, not `images` only, and
|
||||
fetches each file named by `/view`; `Image` becomes `Artifact` with its
|
||||
content type (model/gltf-binary, application/octet-stream for .obj).
|
||||
- Custom nodes: a model (or the engine config) names git repositories
|
||||
and a pip requirements list; the agent's provision step clones them
|
||||
into the replica's `custom_nodes/` (the gate's `--base-directory` is
|
||||
where ComfyUI looks, not the install directory) and installs the
|
||||
requirements into the engine's venv with the same PyTorch index and
|
||||
lock discipline as A8. A replica with a missing node refuses to start
|
||||
with the node's name in the error rather than ComfyUI's traceback.
|
||||
- Licence of the output (decided 2026-10-10: it matters). A catalog model
|
||||
carries `license: {name, url, notice}`; a job's answer, `/v1/models`
|
||||
and the portal page show it beside the result, and the mesh's GLB
|
||||
`asset.extras` carries the notice too, since the file travels without
|
||||
the API. What was checked on 2026-10-10: TRELLIS.2 weights are MIT and
|
||||
its outputs free, but its texture baking depends on nvdiffrast, whose
|
||||
NVIDIA Source Code License is non-commercial — a commercial service
|
||||
must render textures another way or skip that stage. Hunyuan3D 2.0 and
|
||||
2.1 claim no rights in outputs, but the licence excludes the EU, UK and
|
||||
South Korea, needs Tencent's permission above 1 M monthly active users,
|
||||
forbids using outputs to train other models, and asks for "Powered by
|
||||
Tencent Hunyuan"; the notice ships with any distribution. TripoSR is
|
||||
MIT, InstantMesh Apache 2.0. Every one of them trained on Objaverse,
|
||||
some of whose objects are CC-BY-NC: a provenance question no licence
|
||||
text settles, to be said in the notice rather than hidden.
|
||||
- Pricing: `per_image` stays; `per_mesh` beside it in `accounting.Price`,
|
||||
`Outcome.Meshes`, the usage sums and exports (as eaea671 did for
|
||||
pictures).
|
||||
- Done when: TRELLIS.2 on cuda01 with `scaling: {min: 0,
|
||||
idle_timeout: 10m}` makes a .glb from a picture through G1's endpoint
|
||||
with nvdiffrast and nvdiffrec not installed in its venv (the licence
|
||||
is kept by the install, not by a note), the GLB carries the MIT
|
||||
notice,
|
||||
the card is free again after the idle timeout, and the same request
|
||||
against flux-schnell is refused as "makes pictures, not meshes".
|
||||
|
||||
### G4. Portal and console (S, after G2) — done 2026-10-10 (branch `jobs-3d`)
|
||||
Built: /3d with model-viewer served from the gateway, progress, download
|
||||
and licence; `ctl 3d list|get|cancel` over /api/v1/client-jobs (ctl jobs
|
||||
was taken by the control plane's own work). Left: the console's Jobs view.
|
||||
A 3D page beside `/images` (upload a picture, watch progress, download
|
||||
the .glb, a viewer with `<model-viewer>` from the allowed CDNs), `ctl
|
||||
jobs list|get|cancel`, the console's Jobs view. The page sends down the
|
||||
same path as the API, like `imagesSend` does.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
+232
-1
@@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
@@ -26,6 +26,182 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## v0.14.3 — 2026-10-10
|
||||
|
||||
3D models from a picture, as jobs: made by ComfyUI on the cluster's cards,
|
||||
with their progress, their licence written into the mesh, a price per mesh
|
||||
and a page to make them. Also: a profile's own weights, a band for bulk
|
||||
work, and fixes for control planes behind a balancer and agents that
|
||||
restart.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains tables and columns (migrations 0043–0045: notify_sent,
|
||||
jobs, usage meshes); nothing to do.
|
||||
- **Nodes** need this release for their engines' output to survive an
|
||||
agent restart and for ComfyUI's custom nodes and progress; engines
|
||||
started by an older agent keep its pipe until they are restarted.
|
||||
- A 3D model needs a model that makes them (see install.md, "A model that
|
||||
makes them"); none is added.
|
||||
|
||||
### Added
|
||||
|
||||
- **Jobs: 3D models asked after, not waited for.** `POST
|
||||
/v1/3d/generations` answers 202 with a job id; `GET
|
||||
/v1/3d/generations/{id}` says its status, progress and, when done, the
|
||||
file its result is in (`/v1/files/{id}/content`); `POST .../cancel` stops
|
||||
it, in the engine's queue too. The request is replayed by whichever
|
||||
gateway holds the job's lease through the same path a client's request
|
||||
takes — key, allowlist, limits, admission, guardrails, accounting — so
|
||||
it is charged like any request, tagged `job=<id>`, and a model still
|
||||
loading is waited out for as long as the job has (24 h) rather than
|
||||
refused. A job survives the gateway that took it; results are kept for
|
||||
`router.batch.retain_days`. Migration 0044.
|
||||
- **ComfyUI models that make meshes** (`comfyui.output: meshes`). The
|
||||
picture is uploaded to the replica's ComfyUI and the workflow's
|
||||
`{{image}}`; the mesh the workflow saves is the job's file, an OBJ turned
|
||||
into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards
|
||||
stronger ones can be added the same way.
|
||||
- **Custom nodes** (`comfyui.nodes`): a repository at a full commit, put in
|
||||
each replica's own `custom_nodes` by the gate, with its packages
|
||||
installed beside the replica, never into ComfyUI's environment.
|
||||
- **A job's progress**: the gate listens to ComfyUI's websocket and the job
|
||||
says queued and how many ahead, then the node running, its step and
|
||||
percent.
|
||||
- **A model's `license`** (name, url, notice), in `/v1/models`, in the job,
|
||||
on the portal and in the GLB's `asset.extras`.
|
||||
- **`per_mesh`** in `auth.pricing`; usage rows, sums and exports count
|
||||
`meshes`.
|
||||
- **`/3d` on the portal**, beside `/images`: a picture chosen, the job
|
||||
followed, the mesh in a viewer with a download and its licence; and
|
||||
**`llmgw ctl 3d list|get|cancel`** for every owner's jobs.
|
||||
- **A profile's own `source`**: the weights it loads instead of the
|
||||
model's, so an embedding model can run on vLLM from its repository on a
|
||||
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
|
||||
nodes holding each profile's weights, `ctl model add` checks every
|
||||
source, and `ctl weights` collects none a profile loads.
|
||||
- **A band's own `queue_ttl`** (`router.admission.bands[].queue_ttl`): how
|
||||
long its requests may wait for room, instead of the admission's 30 s. A
|
||||
band for bulk work — a knowledge-base import — waits behind people rather
|
||||
than being turned away.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An engine's output no longer breaks when its agent restarts.** Engines
|
||||
wrote to a pipe of the agent's, which an agent that restarted and adopted
|
||||
them left with no reader; an engine's next large write failed, and in a
|
||||
Python engine took the request it was reporting on with it. Each process
|
||||
now writes to its own file in the agent's work directory, and an adopted
|
||||
engine's log keeps what it wrote before the restart.
|
||||
- **Single sign-on behind a balancer.** The cookie that carries a sign-in
|
||||
across the identity provider was signed with a key of each process's own,
|
||||
so with two control planes a callback that reached the other one was
|
||||
refused. The key is now a secret in the store, the same on every control
|
||||
plane.
|
||||
- **Notifications sent once by two control planes.** Each kept its own
|
||||
hour of quiet per event, so a budget spent on a team both served was
|
||||
announced twice. The control plane that claims an event in the store
|
||||
(migration 0043) sends it; with the store unreachable it is sent anyway.
|
||||
- **A request turned away for a busy cluster says when to come back.** A
|
||||
request that waited as long as it may (504) or found the queue full (503)
|
||||
had no `Retry-After`, so `llmgw-kb` counted the passage as failed instead
|
||||
of sending it again; both now say 10 seconds.
|
||||
- The usage sums and exports count pictures (`images`), as the request
|
||||
records did; pictures made before are added to the sums.
|
||||
|
||||
## v0.14.2 — 2026-10-10
|
||||
|
||||
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
|
||||
cluster's own cards. The agents install ComfyUI themselves, the gateway
|
||||
prices pictures by the picture, and the portal has a page to make them.
|
||||
Also: on-demand models that survive a failover, and knowledge-base
|
||||
embeddings fixes.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains a column (`usage.images`); nothing to do.
|
||||
- **ComfyUI is installed only where asked**: `llmgw ctl config set
|
||||
engines.comfyui 0.39.2` has every node with an NVIDIA GPU build it (about
|
||||
10 GB under `/opt/llmgw/engines`). Nodes need this release to install it
|
||||
or run engine `comfyui`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Pictures at `/v1/images/generations`**, OpenAI's shape, through the same
|
||||
key, allowlist, limit, admission and guardrail checks as any request. A
|
||||
model with profiles on engine `comfyui` is placed on a GPU node like any
|
||||
other. A ComfyUI server outside the cluster can be an external model with
|
||||
`api: comfyui` instead. The workflow is a built-in (`flux-schnell`, the
|
||||
default, or `flux-dev`) or a JSON file in ComfyUI's API format, with
|
||||
`{{prompt}}`, `{{width}}`, `{{seed}}` and the like where the request's
|
||||
values go. Requests may also send `seed`, `steps`, `negative_prompt` and
|
||||
`guidance`. The seed used comes back in `x-llmgw-seed`. Pictures come back
|
||||
as `b64_json`; ComfyUI's workflow is stripped from each PNG, and a client
|
||||
that leaves takes its workflow off the queue. `/v1/models` says `images`
|
||||
for such a model.
|
||||
- **Engine `comfyui`.** ComfyUI asks for no key, so the agent starts it
|
||||
behind `llmgw comfyui-gate`. ComfyUI listens on the loopback only. The
|
||||
gate listens on the replica's port, demands the replica's key, and lets
|
||||
through only what the router asks. The model's `comfyui:` block names
|
||||
the workflow and, for a source that is a set of files, the checkpoint.
|
||||
- **The agents install ComfyUI** (`engines.comfyui`). The release's source is
|
||||
checked against a digest llmgw holds, since GitHub publishes none, and a
|
||||
version without one is refused. Its requirements are resolved as PyPI
|
||||
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
|
||||
CUDA 12.9 one for an older driver, and the result is checked on the card.
|
||||
A ComfyUI installed by hand is named by `engines.comfyui.dir` in
|
||||
agent.yaml.
|
||||
- **A price per picture**: `per_image` in `auth.pricing`, on top of the
|
||||
prompt's tokens. A model priced only per picture is not free, so a spent
|
||||
budget still stops it. Each usage row keeps how many pictures were made,
|
||||
shown by `llmgw ctl request`.
|
||||
- **`/images` on the portal**, beside the chat, for the models that make
|
||||
pictures. The chat no longer offers them. The page offers size, how many,
|
||||
seed, steps and a negative prompt, and shows each picture with its seed
|
||||
and a download link. Nothing is kept but the usage.
|
||||
- **An embedding model's `query_prefix`**, put before the question a
|
||||
knowledge-base search embeds and never before documents, for Qwen3-Embedding
|
||||
and models trained like it.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An awake on-demand model survived no restart or change of leader of the
|
||||
control plane.** A control plane that had heard no requests read every
|
||||
on-demand replica its nodes reported as wanted by nobody, and drained them
|
||||
all. An adopted replica now gets a whole idle timeout.
|
||||
- `ctl models` said `not placed` for an on-demand model that had been woken
|
||||
and was serving; it now says `ready (on demand)`.
|
||||
- A batch of embeddings was refused as one over-long prompt when the
|
||||
passages together exceeded the context, though each one fit. The longest
|
||||
input is what has to fit.
|
||||
- The chat's kept conversation sat in the browser under one key for
|
||||
everybody, so the next person to sign in on the same browser opened the
|
||||
last one's conversation. It is kept per account, and signing out clears
|
||||
what the pages stored.
|
||||
|
||||
## v0.14.1 — 2026-10-09
|
||||
|
||||
Adapters trained on the nodes from a knowledge base and kept private to
|
||||
their owners; the cluster's and the agents' settings edited in one place;
|
||||
the tenancy tree and keys over the admin API and in Terraform; knowledge
|
||||
bases that hold up with many documents; the firewall setting on every
|
||||
machine.
|
||||
|
||||
**Upgrading.**
|
||||
- **Nodes** need this release to follow `agents.*` in the cluster's
|
||||
settings; until a node is upgraded its agent.yaml is what it runs on.
|
||||
- **A separate mirror machine and store members that run nothing else**
|
||||
follow the firewall setting once what reads it is installed: run `sudo
|
||||
llmgw mirror init` on the mirror again with the flags it was set up with,
|
||||
and `llmgw server init --patroni` again for the store's members. A control
|
||||
plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
|
||||
- **The store's PostgreSQL memory** applies after each member is restarted,
|
||||
one at a time, the leader last; until then it runs as before.
|
||||
- The store gains tables and columns; nothing to do. Existing knowledge-base
|
||||
collections get their full-text index when a control plane starts, and
|
||||
keep `vector` storage; only new collections use `halfvec`, and only on
|
||||
pgvector 0.7 or later.
|
||||
- Cached knowledge-base passages in vLLM are made again once, under the new
|
||||
`cache_salt`.
|
||||
|
||||
### Added
|
||||
|
||||
- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
|
||||
@@ -41,6 +217,61 @@ APIs change between minors without a deprecation period.
|
||||
that would not load are refused on save, and set aside on a node rather
|
||||
than keep its agent from starting. Nodes need this release to follow
|
||||
them.
|
||||
- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
|
||||
writes question/answer pairs from a collection with a model of the
|
||||
cluster. `llmgw ctl train start` trains an adapter on a node with its
|
||||
installed vLLM's torch and a small overlay, on a card with enough memory
|
||||
free (beside an engine, if it leaves that much), converting to GGUF for
|
||||
llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
|
||||
accepts uploads) and makes it the model's adapter.
|
||||
- **Adapters with owners.** An adapter's `owner` (org, team, project, user
|
||||
or key) is the only part of the tree that sees it; to anyone else it is a
|
||||
model that does not exist. An adapter trained on a knowledge base belongs
|
||||
to a part of the tree that may read it, and is never made public.
|
||||
- The knowledge bases' vLLM `cache_salt` follows the collections searched,
|
||||
not the organization, so two teams of one organization no longer share
|
||||
cached passages.
|
||||
- **Organizations, teams, projects, API keys and memberships in the admin
|
||||
API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
|
||||
`PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
|
||||
issue (the key is in the answer, once), list and revoke; `/api/v1/users`
|
||||
and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
|
||||
and `users` on a control plane before. The Terraform provider manages them
|
||||
as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
|
||||
sensitive `secret`) and `llmgw_membership`.
|
||||
- **The firewall setting reaches every machine.** A separate mirror and the
|
||||
store's members follow `firewall.enabled` and `firewall_networks` as nodes
|
||||
and control planes did: the leading control plane tells the mirror over
|
||||
its API, and keeps the state in the store for members that run no llmgw,
|
||||
which a timer on each reads. The mirror also lets in the network of every
|
||||
node the leader knows. `mirror init` and `server init --patroni` install
|
||||
what follows it; run them again on a cluster set up before.
|
||||
|
||||
### Changed
|
||||
|
||||
- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
|
||||
on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
|
||||
machine. Now each member gets a share of its own RAM, a smaller one where
|
||||
a control plane or agent shares the machine, and the share is read again
|
||||
every time `server init --patroni` runs. The new cache is applied by
|
||||
restarting one member at a time, the leader last.
|
||||
- **Knowledge-base searches hold up with many documents.** New collections
|
||||
store their index as `halfvec` where pgvector is 0.7 or later (half the
|
||||
memory, and embeddings up to 4000 dimensions instead of 2000); older
|
||||
collections, and clusters on pgvector 0.6, keep `vector`. Each collection
|
||||
has its own full-text index, as it has its own vector index, made for
|
||||
existing collections when the control plane starts. The text arm ranks by
|
||||
the cheap measure first and scores only those candidates.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A knowledge-base search asking for more than 10 passages
|
||||
(`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
|
||||
vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
|
||||
raised to the number wanted, up to 1000.
|
||||
- `llmgw server backup` and `restore` against a replicated store over TLS
|
||||
failed looking for `~/.postgresql/root.crt`: pg_dump was given
|
||||
`sslmode=verify-full` without the cluster's authority.
|
||||
|
||||
## v0.14.0 — 2026-10-08
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -50,6 +50,13 @@ budgets and rate limits decide who may use any of it.
|
||||
|
||||
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
||||
|
||||
- **Adapters, and the tenancy tree over the API** (v0.14.1). `llmgw ctl
|
||||
kb dataset` and `llmgw ctl train` make a LoRA adapter from a collection
|
||||
on a node's spare GPU memory, kept on the mirror and seen only by its
|
||||
owner. Organizations, teams, projects, keys and memberships are in the
|
||||
admin API and the Terraform provider. The agents' settings are edited
|
||||
with the cluster's, on the portal too, and the firewall setting reaches
|
||||
every machine.
|
||||
- **Knowledge bases** (v0.14.0). People put a company's documents in from
|
||||
their own machines with `llmgw-kb`, a separate download for Windows,
|
||||
macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
|
||||
|
||||
+502
-71
@@ -106,6 +106,8 @@ type AgentMessage struct {
|
||||
// *AgentMessage_FabricTest
|
||||
// *AgentMessage_Upgrade
|
||||
// *AgentMessage_KvEvents
|
||||
// *AgentMessage_TrainStatus
|
||||
// *AgentMessage_TrainArtifact
|
||||
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -220,6 +222,24 @@ func (x *AgentMessage) GetKvEvents() *KVEvents {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetTrainStatus() *TrainStatus {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_TrainStatus); ok {
|
||||
return x.TrainStatus
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_TrainArtifact); ok {
|
||||
return x.TrainArtifact
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isAgentMessage_Msg interface {
|
||||
isAgentMessage_Msg()
|
||||
}
|
||||
@@ -256,6 +276,14 @@ type AgentMessage_KvEvents struct {
|
||||
KvEvents *KVEvents `protobuf:"bytes,8,opt,name=kv_events,json=kvEvents,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_TrainStatus struct {
|
||||
TrainStatus *TrainStatus `protobuf:"bytes,9,opt,name=train_status,json=trainStatus,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_TrainArtifact struct {
|
||||
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*AgentMessage_Register) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
|
||||
@@ -272,6 +300,10 @@ func (*AgentMessage_Upgrade) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_KvEvents) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
|
||||
|
||||
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
|
||||
// by the agent from the engine's KV cache events and sent on for the router's
|
||||
// precise index (plan §7.5 tier 2).
|
||||
@@ -392,6 +424,8 @@ type ControlMessage struct {
|
||||
// *ControlMessage_Firewall
|
||||
// *ControlMessage_RestartAgent
|
||||
// *ControlMessage_AgentSettings
|
||||
// *ControlMessage_TrainStart
|
||||
// *ControlMessage_TrainCancel
|
||||
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -566,6 +600,24 @@ func (x *ControlMessage) GetAgentSettings() *AgentSettings {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrainStart() *TrainStart {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_TrainStart); ok {
|
||||
return x.TrainStart
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrainCancel() *TrainCancel {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_TrainCancel); ok {
|
||||
return x.TrainCancel
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrace() map[string]string {
|
||||
if x != nil {
|
||||
return x.Trace
|
||||
@@ -633,6 +685,14 @@ type ControlMessage_AgentSettings struct {
|
||||
AgentSettings *AgentSettings `protobuf:"bytes,15,opt,name=agent_settings,json=agentSettings,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_TrainStart struct {
|
||||
TrainStart *TrainStart `protobuf:"bytes,16,opt,name=train_start,json=trainStart,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_TrainCancel struct {
|
||||
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*ControlMessage_Start) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Stop) isControlMessage_Msg() {}
|
||||
@@ -661,6 +721,10 @@ func (*ControlMessage_RestartAgent) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_AgentSettings) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
|
||||
|
||||
// RestartAgent has the agent start again, as llmgw restart asks of every
|
||||
// node: its engines keep serving and are taken over again.
|
||||
type RestartAgent struct {
|
||||
@@ -4699,11 +4763,335 @@ func (x *LoraAdapter) GetSource() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
|
||||
// weights from its registry, on question/answer pairs, with a Python trainer
|
||||
// the agent carries, in an environment it makes over an installed engine's.
|
||||
// One at a time per node.
|
||||
type TrainStart struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
// base_repo and base_revision name the base model's weights on its
|
||||
// registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
|
||||
BaseRepo string `protobuf:"bytes,2,opt,name=base_repo,json=baseRepo,proto3" json:"base_repo,omitempty"`
|
||||
BaseRevision string `protobuf:"bytes,3,opt,name=base_revision,json=baseRevision,proto3" json:"base_revision,omitempty"`
|
||||
// data is the training pairs, NDJSON of {"q", "a"}.
|
||||
Data []byte `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`
|
||||
// params is the trainer's settings as JSON: rank, alpha, epochs, lr,
|
||||
// batch, max_len.
|
||||
Params string `protobuf:"bytes,5,opt,name=params,proto3" json:"params,omitempty"`
|
||||
// gpu_memory_mib is what the job needs of a card; the agent starts it
|
||||
// only when one has that much free, which lets it run beside an engine.
|
||||
GpuMemoryMib int32 `protobuf:"varint,6,opt,name=gpu_memory_mib,json=gpuMemoryMib,proto3" json:"gpu_memory_mib,omitempty"`
|
||||
// gguf asks for the adapter in llama.cpp's format too.
|
||||
Gguf bool `protobuf:"varint,7,opt,name=gguf,proto3" json:"gguf,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainStart) Reset() {
|
||||
*x = TrainStart{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[44]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainStart) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainStart) ProtoMessage() {}
|
||||
|
||||
func (x *TrainStart) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[44]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainStart.ProtoReflect.Descriptor instead.
|
||||
func (*TrainStart) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{44}
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetBaseRepo() string {
|
||||
if x != nil {
|
||||
return x.BaseRepo
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetBaseRevision() string {
|
||||
if x != nil {
|
||||
return x.BaseRevision
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetData() []byte {
|
||||
if x != nil {
|
||||
return x.Data
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetParams() string {
|
||||
if x != nil {
|
||||
return x.Params
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetGpuMemoryMib() int32 {
|
||||
if x != nil {
|
||||
return x.GpuMemoryMib
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetGguf() bool {
|
||||
if x != nil {
|
||||
return x.Gguf
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// TrainCancel stops a training job.
|
||||
type TrainCancel struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainCancel) Reset() {
|
||||
*x = TrainCancel{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[45]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainCancel) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainCancel) ProtoMessage() {}
|
||||
|
||||
func (x *TrainCancel) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[45]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainCancel.ProtoReflect.Descriptor instead.
|
||||
func (*TrainCancel) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{45}
|
||||
}
|
||||
|
||||
func (x *TrainCancel) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// TrainStatus is how a training job is going, sent as it changes.
|
||||
type TrainStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
// state is preparing, running, done or failed.
|
||||
State string `protobuf:"bytes,2,opt,name=state,proto3" json:"state,omitempty"`
|
||||
Step int32 `protobuf:"varint,3,opt,name=step,proto3" json:"step,omitempty"`
|
||||
Steps int32 `protobuf:"varint,4,opt,name=steps,proto3" json:"steps,omitempty"`
|
||||
Loss float64 `protobuf:"fixed64,5,opt,name=loss,proto3" json:"loss,omitempty"`
|
||||
// message is what is being done, or why it failed.
|
||||
Message string `protobuf:"bytes,6,opt,name=message,proto3" json:"message,omitempty"`
|
||||
// peak_mib is the most the job held of the card.
|
||||
PeakMib int32 `protobuf:"varint,7,opt,name=peak_mib,json=peakMib,proto3" json:"peak_mib,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainStatus) Reset() {
|
||||
*x = TrainStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[46]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainStatus) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainStatus) ProtoMessage() {}
|
||||
|
||||
func (x *TrainStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[46]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainStatus.ProtoReflect.Descriptor instead.
|
||||
func (*TrainStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{46}
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetState() string {
|
||||
if x != nil {
|
||||
return x.State
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetStep() int32 {
|
||||
if x != nil {
|
||||
return x.Step
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetSteps() int32 {
|
||||
if x != nil {
|
||||
return x.Steps
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetLoss() float64 {
|
||||
if x != nil {
|
||||
return x.Loss
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetMessage() string {
|
||||
if x != nil {
|
||||
return x.Message
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetPeakMib() int32 {
|
||||
if x != nil {
|
||||
return x.PeakMib
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// TrainArtifact is a piece of what a training job made, sent before its
|
||||
// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
|
||||
// adapter (kind gguf).
|
||||
type TrainArtifact struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
Kind string `protobuf:"bytes,2,opt,name=kind,proto3" json:"kind,omitempty"`
|
||||
Seq int32 `protobuf:"varint,3,opt,name=seq,proto3" json:"seq,omitempty"`
|
||||
Data []byte `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`
|
||||
Last bool `protobuf:"varint,5,opt,name=last,proto3" json:"last,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) Reset() {
|
||||
*x = TrainArtifact{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[47]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainArtifact) ProtoMessage() {}
|
||||
|
||||
func (x *TrainArtifact) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[47]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainArtifact.ProtoReflect.Descriptor instead.
|
||||
func (*TrainArtifact) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{47}
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetKind() string {
|
||||
if x != nil {
|
||||
return x.Kind
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetSeq() int32 {
|
||||
if x != nil {
|
||||
return x.Seq
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetData() []byte {
|
||||
if x != nil {
|
||||
return x.Data
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetLast() bool {
|
||||
if x != nil {
|
||||
return x.Last
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb9\x03\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
|
||||
"\fAgentMessage\x120\n" +
|
||||
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
|
||||
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
|
||||
@@ -4714,7 +5102,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
|
||||
"fabricTest\x122\n" +
|
||||
"\aupgrade\x18\a \x01(\v2\x16.llmgw.v1.UpgradeReplyH\x00R\aupgrade\x121\n" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEventsB\x05\n" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
|
||||
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
|
||||
"\x0etrain_artifact\x18\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
|
||||
"\x03msg\"\xb5\x01\n" +
|
||||
"\bKVEvents\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -4725,7 +5116,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"clearFirst\x12\x16\n" +
|
||||
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
|
||||
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\x83\a\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
|
||||
"\x0eControlMessage\x12,\n" +
|
||||
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
|
||||
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
|
||||
@@ -4743,7 +5134,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x0econtrol_planes\x18\v \x01(\v2\x17.llmgw.v1.ControlPlanesH\x00R\rcontrolPlanes\x120\n" +
|
||||
"\bfirewall\x18\r \x01(\v2\x12.llmgw.v1.FirewallH\x00R\bfirewall\x12=\n" +
|
||||
"\rrestart_agent\x18\x0e \x01(\v2\x16.llmgw.v1.RestartAgentH\x00R\frestartAgent\x12@\n" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x129\n" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
|
||||
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
|
||||
"trainStart\x12:\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
|
||||
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
|
||||
"\n" +
|
||||
"TraceEntry\x12\x10\n" +
|
||||
@@ -5150,7 +5544,32 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x05error\x18\x03 \x01(\tR\x05error\"9\n" +
|
||||
"\vLoraAdapter\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06source\x18\x02 \x01(\tR\x06source2O\n" +
|
||||
"\x06source\x18\x02 \x01(\tR\x06source\"\xcb\x01\n" +
|
||||
"\n" +
|
||||
"TrainStart\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x1b\n" +
|
||||
"\tbase_repo\x18\x02 \x01(\tR\bbaseRepo\x12#\n" +
|
||||
"\rbase_revision\x18\x03 \x01(\tR\fbaseRevision\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x16\n" +
|
||||
"\x06params\x18\x05 \x01(\tR\x06params\x12$\n" +
|
||||
"\x0egpu_memory_mib\x18\x06 \x01(\x05R\fgpuMemoryMib\x12\x12\n" +
|
||||
"\x04gguf\x18\a \x01(\bR\x04gguf\"$\n" +
|
||||
"\vTrainCancel\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\"\xad\x01\n" +
|
||||
"\vTrainStatus\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x14\n" +
|
||||
"\x05state\x18\x02 \x01(\tR\x05state\x12\x12\n" +
|
||||
"\x04step\x18\x03 \x01(\x05R\x04step\x12\x14\n" +
|
||||
"\x05steps\x18\x04 \x01(\x05R\x05steps\x12\x12\n" +
|
||||
"\x04loss\x18\x05 \x01(\x01R\x04loss\x12\x18\n" +
|
||||
"\amessage\x18\x06 \x01(\tR\amessage\x12\x19\n" +
|
||||
"\bpeak_mib\x18\a \x01(\x05R\apeakMib\"t\n" +
|
||||
"\rTrainArtifact\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x12\n" +
|
||||
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
|
||||
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -5169,7 +5588,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 52)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -5216,14 +5635,18 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*UpgradeAgent)(nil), // 42: llmgw.v1.UpgradeAgent
|
||||
(*UpgradeReply)(nil), // 43: llmgw.v1.UpgradeReply
|
||||
(*LoraAdapter)(nil), // 44: llmgw.v1.LoraAdapter
|
||||
nil, // 45: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 46: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 47: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 48: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 49: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 50: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 51: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 52: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*TrainStart)(nil), // 45: llmgw.v1.TrainStart
|
||||
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
|
||||
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
|
||||
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
|
||||
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 50: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 52: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 53: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -5234,62 +5657,66 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
41, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
|
||||
43, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
|
||||
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
|
||||
30, // 8: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 9: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 10: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 11: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 12: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 13: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 14: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 15: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 16: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 17: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 18: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 19: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 20: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 21: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 22: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
46, // 23: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 24: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 25: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
47, // 26: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 27: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 28: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 29: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 30: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 31: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 32: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 33: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
48, // 34: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 35: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 36: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 37: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
49, // 38: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 39: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 40: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 41: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
50, // 42: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 43: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 44: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 45: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 46: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 47: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 48: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 49: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
51, // 50: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 51: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 52: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 53: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 54: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
52, // 55: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 56: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 57: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 58: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
58, // [58:59] is the sub-list for method output_type
|
||||
57, // [57:58] is the sub-list for method input_type
|
||||
57, // [57:57] is the sub-list for extension type_name
|
||||
57, // [57:57] is the sub-list for extension extendee
|
||||
0, // [0:57] is the sub-list for field type_name
|
||||
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
|
||||
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
|
||||
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
62, // [62:63] is the sub-list for method output_type
|
||||
61, // [61:62] is the sub-list for method input_type
|
||||
61, // [61:61] is the sub-list for extension type_name
|
||||
61, // [61:61] is the sub-list for extension extendee
|
||||
0, // [0:61] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -5306,6 +5733,8 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*AgentMessage_FabricTest)(nil),
|
||||
(*AgentMessage_Upgrade)(nil),
|
||||
(*AgentMessage_KvEvents)(nil),
|
||||
(*AgentMessage_TrainStatus)(nil),
|
||||
(*AgentMessage_TrainArtifact)(nil),
|
||||
}
|
||||
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
|
||||
(*ControlMessage_Start)(nil),
|
||||
@@ -5322,6 +5751,8 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*ControlMessage_Firewall)(nil),
|
||||
(*ControlMessage_RestartAgent)(nil),
|
||||
(*ControlMessage_AgentSettings)(nil),
|
||||
(*ControlMessage_TrainStart)(nil),
|
||||
(*ControlMessage_TrainCancel)(nil),
|
||||
}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
@@ -5329,7 +5760,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 52,
|
||||
NumMessages: 56,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -28,6 +28,8 @@ message AgentMessage {
|
||||
FabricTestReply fabric_test = 6;
|
||||
UpgradeReply upgrade = 7;
|
||||
KVEvents kv_events = 8;
|
||||
TrainStatus train_status = 9;
|
||||
TrainArtifact train_artifact = 10;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -74,6 +76,8 @@ message ControlMessage {
|
||||
Firewall firewall = 13;
|
||||
RestartAgent restart_agent = 14;
|
||||
AgentSettings agent_settings = 15;
|
||||
TrainStart train_start = 16;
|
||||
TrainCancel train_cancel = 17;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -884,3 +888,55 @@ message LoraAdapter {
|
||||
string name = 1;
|
||||
string source = 2;
|
||||
}
|
||||
|
||||
// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
|
||||
// weights from its registry, on question/answer pairs, with a Python trainer
|
||||
// the agent carries, in an environment it makes over an installed engine's.
|
||||
// One at a time per node.
|
||||
message TrainStart {
|
||||
string job_id = 1;
|
||||
// base_repo and base_revision name the base model's weights on its
|
||||
// registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
|
||||
string base_repo = 2;
|
||||
string base_revision = 3;
|
||||
// data is the training pairs, NDJSON of {"q", "a"}.
|
||||
bytes data = 4;
|
||||
// params is the trainer's settings as JSON: rank, alpha, epochs, lr,
|
||||
// batch, max_len.
|
||||
string params = 5;
|
||||
// gpu_memory_mib is what the job needs of a card; the agent starts it
|
||||
// only when one has that much free, which lets it run beside an engine.
|
||||
int32 gpu_memory_mib = 6;
|
||||
// gguf asks for the adapter in llama.cpp's format too.
|
||||
bool gguf = 7;
|
||||
}
|
||||
|
||||
// TrainCancel stops a training job.
|
||||
message TrainCancel {
|
||||
string job_id = 1;
|
||||
}
|
||||
|
||||
// TrainStatus is how a training job is going, sent as it changes.
|
||||
message TrainStatus {
|
||||
string job_id = 1;
|
||||
// state is preparing, running, done or failed.
|
||||
string state = 2;
|
||||
int32 step = 3;
|
||||
int32 steps = 4;
|
||||
double loss = 5;
|
||||
// message is what is being done, or why it failed.
|
||||
string message = 6;
|
||||
// peak_mib is the most the job held of the card.
|
||||
int32 peak_mib = 7;
|
||||
}
|
||||
|
||||
// TrainArtifact is a piece of what a training job made, sent before its
|
||||
// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
|
||||
// adapter (kind gguf).
|
||||
message TrainArtifact {
|
||||
string job_id = 1;
|
||||
string kind = 2;
|
||||
int32 seq = 3;
|
||||
bytes data = 4;
|
||||
bool last = 5;
|
||||
}
|
||||
+16
-7
@@ -8,8 +8,9 @@ description is.
|
||||
anything they do can be done with `curl`, a generated client or the
|
||||
[Terraform provider](../terraform-provider-llmgw/README.md). It adds and
|
||||
removes models, scales them, cordons and upgrades nodes, sets capacity
|
||||
quotas, attaches guardrail policies, and reads the fleet, usage and the audit
|
||||
log.
|
||||
quotas, attaches guardrail policies, adds organizations, teams and projects,
|
||||
issues and revokes API keys, puts people in the tree, and reads the fleet,
|
||||
usage and the audit log.
|
||||
|
||||
## Reaching it
|
||||
|
||||
@@ -99,6 +100,14 @@ curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2,
|
||||
# A team's capacity quota.
|
||||
curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'
|
||||
|
||||
# An organization, a team in it and a project in that; then a key for the
|
||||
# project. The answer holds the key once: only its hash is kept.
|
||||
curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
|
||||
curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
|
||||
curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
|
||||
curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
|
||||
curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'
|
||||
|
||||
# Remove the model; a DELETE carries the content type too.
|
||||
curl -s "${H[@]}" -X DELETE $API/models/qwen-small
|
||||
```
|
||||
@@ -149,11 +158,11 @@ go generate ./internal/docgen
|
||||
|
||||
## What it does not cover
|
||||
|
||||
- **Organizations, teams, projects, people and API keys.** These live in the
|
||||
store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
|
||||
control plane, or in the web console's `/admin` pages on the client port
|
||||
behind a session. The admin API reads the tree only where a quota or a
|
||||
policy names a part of it.
|
||||
- **Making people's accounts.** People sign up, are invited, or come from
|
||||
the identity provider (OIDC, SCIM); the API lists them and puts them in
|
||||
the tree, and `llmgw users` approves, disables and deletes them. Key
|
||||
lifetime limits and spending limits are `llmgw keys ttl` and `llmgw keys
|
||||
limits`.
|
||||
- **The client-facing API** (`/v1/chat/completions` and the rest), which is
|
||||
OpenAI's and Anthropic's, on `listen`, with API keys.
|
||||
- **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+2
-1
@@ -547,7 +547,8 @@ its usual exception with the gateway's message in it. The ones worth knowing:
|
||||
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
|
||||
| 403 | The key may not use that model. |
|
||||
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the queue is full. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
| 504 | Every replica was busy for as long as the key's band may wait. `Retry-After` says when to send it again. |
|
||||
|
||||
## Priority
|
||||
|
||||
|
||||
+297
-10
@@ -201,6 +201,16 @@ this by running the same command again, on the control plane that holds the
|
||||
authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
|
||||
PostgreSQL serves throughout and no failover happens.
|
||||
|
||||
Each member's PostgreSQL is sized to the machine's memory, read each time
|
||||
the command runs: a quarter of it as cache (`shared_buffers`), three quarters
|
||||
counted on for the planner, and room to sort and build indexes, all capped. A
|
||||
member that also runs a control plane or an agent gets less: about 15% as
|
||||
cache, so the gateway is not pushed into swap. A knowledge base's vector
|
||||
index is searched well only from memory, so give a store machine more RAM and
|
||||
run the command again. The new cache takes a PostgreSQL restart, which Patroni
|
||||
does one member at a time, the leader last. The leader's restart costs a few
|
||||
seconds without writes, and there is no failover.
|
||||
|
||||
Every step looks before it acts. Running it again finishes an install that
|
||||
stopped halfway and changes nothing on one that did not; it never overwrites
|
||||
`server.yaml`, the authority, or an existing account's password.
|
||||
@@ -1028,6 +1038,27 @@ qwen2.5-coder-3b-instruct yes 1/1 ready coder
|
||||
No restart, and it survives one: models added this way are stored, while
|
||||
anything in `server.yaml` is declarative and returns on every start.
|
||||
|
||||
A profile may load weights of its own. An embedding model, for example, can
|
||||
be served by vLLM from its repository on a card and by llama.cpp from a GGUF
|
||||
on CPUs, as one model:
|
||||
|
||||
```yaml
|
||||
name: qwen3-embed-4b
|
||||
source: Qwen/Qwen3-Embedding-4B # what a profile loads unless it says
|
||||
capabilities: { embeddings: true }
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: vllm
|
||||
gpu_class: 6000-24g
|
||||
args: ["--runner", "pooling"]
|
||||
- name: cpu
|
||||
engine: llamacpp
|
||||
source: hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf
|
||||
```
|
||||
|
||||
Updates and canaries move the model's `source`; a profile with its own keeps
|
||||
it.
|
||||
|
||||
### Models that load when someone asks
|
||||
|
||||
A cluster with more models than GPUs can hold every model in its catalog and
|
||||
@@ -1260,6 +1291,128 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Pictures from ComfyUI
|
||||
|
||||
Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them
|
||||
made by ComfyUI: it fills a ComfyUI workflow with the request, queues it,
|
||||
waits, and sends the pictures back as `b64_json`.
|
||||
|
||||
**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
|
||||
GPU node like any other, started when asked for, and stopped when unused.
|
||||
|
||||
The node must offer the engine. Ask for it in the cluster's settings, and every
|
||||
node with an NVIDIA GPU installs it:
|
||||
|
||||
```
|
||||
llmgw ctl config set engines.comfyui 0.39.2
|
||||
```
|
||||
|
||||
The agent fetches ComfyUI's source from GitHub and checks it against the
|
||||
digest llmgw holds for that version. GitHub publishes none for a tag's archive,
|
||||
so a version llmgw does not know is refused. The agent then builds a Python
|
||||
environment beside the source, under `/opt/llmgw/engines/comfyui/`. Its
|
||||
dependencies are resolved as PyPI stood the day after the release. torch comes
|
||||
from PyPI (CUDA 13), or from PyTorch's CUDA 12.9 build where the driver is older
|
||||
than 580. The install is checked on the node's GPU, and takes about 10 GB.
|
||||
`engine_overrides` can keep it off nodes that should not have it.
|
||||
|
||||
Alternatively, install ComfyUI by hand. You need a directory with `main.py` and
|
||||
its environment in `.venv`, readable by the user engines run as
|
||||
(`llmgw-engine`). Name it in that node's `agent.yaml`; a node with one named
|
||||
there is not given the installed one:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
comfyui:
|
||||
dir: /opt/comfyui # python: defaults to /opt/comfyui/.venv/bin/python
|
||||
```
|
||||
|
||||
Then add the model:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors
|
||||
comfyui:
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
scaling: {min: 0, max: 1, idle_timeout: 15m}
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: comfyui
|
||||
gpu_class: 6000-24g
|
||||
tp: 1
|
||||
```
|
||||
|
||||
The source is fetched as for any model. It can be one file, such as the
|
||||
single-file checkpoints the built-in workflows load. It can also be a mirror
|
||||
set (a URL ending in `/`) for a model in several files, such as a diffusion
|
||||
model, its text encoders and its VAE. ComfyUI sees every file of a set by its
|
||||
own name, so a workflow of your own can load each one. A set must name the
|
||||
file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file
|
||||
is named by its own file name.
|
||||
|
||||
ComfyUI asks nobody for a key. The agent therefore starts it behind a gate
|
||||
(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate
|
||||
listens on the replica's port, demands the replica's key, and lets through only
|
||||
what the router asks: queueing, waiting, fetching a picture and cancelling.
|
||||
Uploads and node installs are refused. Each replica has its own directory,
|
||||
`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw
|
||||
ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`.
|
||||
`--disable-smart-memory` frees the card between pictures, at the cost of loading
|
||||
the model again for each one.
|
||||
|
||||
**On a ComfyUI server outside the cluster.** Make it an external model with
|
||||
`api: comfyui`:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: external
|
||||
external:
|
||||
url: http://10.0.0.147:8188 # ComfyUI, started with --listen
|
||||
model: flux1-schnell-fp8.safetensors # the checkpoint, as ComfyUI lists it
|
||||
api: comfyui
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
```
|
||||
|
||||
Like any external model, its requests count as leaving the cluster.
|
||||
|
||||
`flux-schnell` (the default) and `flux-dev` are built in. They suit the
|
||||
single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
|
||||
For any other model, export the workflow from ComfyUI with **Save (API)** and
|
||||
name the file. In the file, put placeholders where the request's values go:
|
||||
|
||||
- `{{prompt}}` and `{{negative_prompt}}` are the prompts.
|
||||
- `{{width}}` and `{{height}}` are the size.
|
||||
- `{{n}}` is the batch size.
|
||||
- `{{seed}}` is the seed.
|
||||
- `{{steps|4}}` and `{{guidance|3.5}}` take the value after the `|` when the
|
||||
request leaves them out.
|
||||
- `{{checkpoint}}` is the `model` above.
|
||||
|
||||
A request is OpenAI's: `prompt`, `n` (at most 4), `size` (each side a multiple
|
||||
of 16, from 256 to 2048) and `response_format: b64_json`. Beyond those, it may
|
||||
send `seed`, `steps`, `negative_prompt` and `guidance`. The seed used comes
|
||||
back in `x-llmgw-seed`. `url` is not offered, because pictures are kept
|
||||
nowhere. ComfyUI also writes the workflow into each PNG; llmgw strips it
|
||||
before the picture is sent.
|
||||
|
||||
Requests for pictures pass the same checks as any other request: keys,
|
||||
allowlists, limits, admission, and the guardrails on the prompt. A client that
|
||||
leaves takes its workflow off the ComfyUI queue.
|
||||
|
||||
Each request's usage row records how many pictures it made (`llmgw ctl request
|
||||
ID` shows them). A picture's price is set with `per_image`, in micro-units like
|
||||
the other prices, and is charged on top of the prompt's tokens:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
pricing:
|
||||
flux-schnell: {prompt_per_1k: 0, per_image: 20000}
|
||||
```
|
||||
|
||||
An external ComfyUI on a card that llmgw also places models on should be
|
||||
started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on
|
||||
the card after each picture, and llmgw sees that memory as somebody else's.
|
||||
|
||||
### Federating clusters
|
||||
|
||||
Another llmgw cluster — another site, another team's GPUs — can be a provider
|
||||
@@ -1540,6 +1693,7 @@ and accounting exactly as an API call does.
|
||||
| `POST /v1/audio/transcriptions`, `/v1/audio/translations` | the multipart form passed through as it came, to an engine that serves speech to text: vLLM with Whisper |
|
||||
| `POST /v1/moderations` | answered by the guardrails' judge model (`role: guard`): `flagged`, and the judge's own categories |
|
||||
| `/v1/files`, `/v1/batches` | batches; see [Batches](#batches) |
|
||||
| `/v1/3d/generations` | 3D models, as jobs asked after; see [Jobs](#jobs-3d-models) |
|
||||
| `GET /v1/models` | the models this key may use |
|
||||
|
||||
## Watching it
|
||||
@@ -1736,9 +1890,11 @@ the file again.
|
||||
|
||||
On a machine that also runs the control plane, PostgreSQL's defaults assume it
|
||||
owns the box. A gateway swapped out to give the database cache is a worse
|
||||
outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
|
||||
`max_connections = 40` are enough for a store whose working set is measured in
|
||||
megabytes.
|
||||
outcome than a smaller cache, so size it to the machine. `server init` does this
|
||||
for a PostgreSQL it installs: `shared_buffers` about 15% of memory (at least
|
||||
96MB), `effective_cache_size` half, `work_mem` and `maintenance_work_mem`
|
||||
scaled to it, and `max_connections = 40`. A store with a large knowledge base
|
||||
wants a machine of its own, where its vector index fits in memory.
|
||||
|
||||
## Tagging requests
|
||||
|
||||
@@ -1896,6 +2052,121 @@ Each line of the file is `{"custom_id": …, "method": "POST", "url":
|
||||
or past their retention — a project under a guardrails policy with
|
||||
`no_store` is refused it.
|
||||
|
||||
## Jobs: 3D models
|
||||
|
||||
A 3D model from a picture takes minutes on a card that first has to be
|
||||
emptied of whatever else was on it. No client should hold a connection for
|
||||
that, so it is a job: queued, answered with an id, asked after, its result
|
||||
downloaded.
|
||||
|
||||
```
|
||||
curl http://gateway:8080/v1/3d/generations -H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-d '{"model": "trellis2", "image": "<base64>"}'
|
||||
# → 202 {"id": "job_…", "status": "queued", …}
|
||||
curl http://gateway:8080/v1/3d/generations/job_… -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
# → {"status": "completed", "output_file_id": "file-…", "content_type": "model/gltf-binary", "usage": {…}}
|
||||
curl http://gateway:8080/v1/files/file-…/content -H "Authorization: Bearer $LLMGW_API_KEY" -o model.glb
|
||||
curl -X POST http://gateway:8080/v1/3d/generations/job_…/cancel -H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
- **Status:** `queued`, `running`, `completed`, `failed` (with `error`),
|
||||
`cancelling`, `cancelled`, `expired` (not finished within 24 h).
|
||||
`progress` carries what the engine said of the work so far, once it says
|
||||
anything; `GET /v1/3d/generations` lists the caller's jobs.
|
||||
- **Refused at the door:** no key, a model that is not there or not in the
|
||||
key's allowlist, a request the model cannot serve. Refused when served,
|
||||
and said as the job's error: a spent budget, the guardrails, a model
|
||||
that will not come up.
|
||||
- **How it is served:** the request is kept as it was sent and replayed by
|
||||
whichever gateway holds the job's lease, through the same path a
|
||||
client's request takes — limits, admission in the key's band, guardrails,
|
||||
accounting. It is one usage record under the job's id, tagged
|
||||
`job=<id>`, charged like any request; `GET` shows the usage once it is
|
||||
done. A model still loading, a rate limit or a full queue is waited out
|
||||
for as long as the job has: nobody is holding a connection, which is the
|
||||
point.
|
||||
- **Surviving a restart:** a gateway that stops hands its jobs to another
|
||||
within two minutes; a job it was running is run again from the start.
|
||||
- **What it keeps:** the request as sent, and the result as a file owned
|
||||
by the key's project, kept `router.batch.retain_days` (30) like a batch's
|
||||
files; the job's record goes with it. A project under a `no_store`
|
||||
policy is refused jobs, as it is batches.
|
||||
|
||||
A request may also send `seed` and `resolution` (the model's own grid, a
|
||||
multiple of 32 from 64 to 2048). The picture should show one object on a
|
||||
plain background: the model makes a mesh of what it sees, background
|
||||
included.
|
||||
|
||||
### A model that makes them
|
||||
|
||||
A 3D model is made by a ComfyUI model whose `comfyui.output` is `meshes`
|
||||
(see [Pictures from ComfyUI](#pictures-from-comfyui) for engine `comfyui`).
|
||||
Its workflow is a JSON file in ComfyUI's API format: `{{image}}` is the
|
||||
uploaded picture, which a `LoadImage` node reads; `{{seed}}`,
|
||||
`{{resolution|256}}` and `{{checkpoint}}` are as for pictures. The node that
|
||||
saves the mesh may save a GLB or an OBJ; an OBJ leaves the gateway as a GLB,
|
||||
vertex colours kept. TripoSR (MIT) on any card of 8 GB:
|
||||
|
||||
```yaml
|
||||
- name: triposr
|
||||
source: hf:stabilityai/TripoSR/model.ckpt
|
||||
comfyui:
|
||||
workflow: /etc/llmgw/workflows/triposr.json # on every control plane
|
||||
output: meshes
|
||||
nodes:
|
||||
- repo: https://github.com/flowtyone/ComfyUI-Flowty-TripoSR
|
||||
commit: a0c94ac60a7cc062604f61aeeea6d0d493521de3
|
||||
requirements: [omegaconf, trimesh, scikit-image, imageio,
|
||||
"transformers>=4.50.3,<5", "huggingface-hub<1", tokenizers]
|
||||
license:
|
||||
name: MIT
|
||||
url: https://huggingface.co/stabilityai/TripoSR/blob/main/LICENSE
|
||||
notice: "Made with TripoSR (Stability AI and Tripo AI, MIT). ..."
|
||||
scaling: {min: 0, max: 1, idle_timeout: 10m}
|
||||
profiles:
|
||||
- {name: gpu, engine: comfyui, gpu_class: 6000-24g, tp: 1}
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"1": {"class_type": "LoadImage", "inputs": {"image": "{{image}}"}},
|
||||
"2": {"class_type": "TripoSRModelLoader", "inputs": {"model": "{{checkpoint}}", "chunk_size": 8192}},
|
||||
"3": {"class_type": "TripoSRSampler", "inputs": {"model": ["2", 0], "reference_image": ["1", 0],
|
||||
"geometry_resolution": "{{resolution|256}}", "threshold": 25.0}},
|
||||
"4": {"class_type": "TripoSRViewer", "inputs": {"mesh": ["3", 0]}}
|
||||
}
|
||||
```
|
||||
|
||||
- **Custom nodes** are named by https repository and full commit (a branch
|
||||
is refused: it moves). The gate puts each in the replica's own
|
||||
`custom_nodes` and installs its `requirements` beside the replica, never
|
||||
into ComfyUI's environment. A package named there is installed at the
|
||||
range named; every other one at the version ComfyUI already runs. A
|
||||
package named is first on ComfyUI's path, so its range must suit ComfyUI
|
||||
as well. TripoSR's checkpoint loads with transformers 4, which ComfyUI's
|
||||
`>=4.50.3` allows.
|
||||
- **Progress:** the gate listens to ComfyUI's websocket and the router
|
||||
writes what it hears into the job: `{"state": "queued", "ahead": 2}`,
|
||||
then `{"state": "running", "node_class": "KSampler", "step": 12,
|
||||
"steps": 30, "percent": 40}` for a node that counts its steps, and
|
||||
`done` at the end.
|
||||
- **Licence:** a model's `license` is shown in `/v1/models`, in the job and
|
||||
on the portal, and written into the GLB's `asset.extras.license`, since
|
||||
the file travels without the API. Choose the model for its licence before
|
||||
its quality when what it makes is shipped. TRELLIS.2 needs Meta's gated
|
||||
DINOv3 and, as published, a background remover under a non-commercial
|
||||
licence; Hunyuan3D's licence excludes the EU, the UK and South Korea.
|
||||
- **Price:** `per_mesh` in `auth.pricing`, beside `per_image`; each usage
|
||||
row says how many meshes it made (`meshes`).
|
||||
- **Cards:** the strongest models today (TRELLIS.2, Direct3D-S2) need
|
||||
flash-attention, which runs on Ampere and newer. A Turing card such as
|
||||
the RTX 6000 runs TripoSR.
|
||||
|
||||
People make them at `/3d` on the portal: they choose a picture and watch
|
||||
the job, and the mesh is shown in a viewer (model-viewer, served from the
|
||||
gateway) with a download link and the licence. `llmgw ctl 3d list`, `get`
|
||||
and `cancel` show and stop every owner's jobs.
|
||||
|
||||
## Conversations across models
|
||||
|
||||
Off everywhere until a project asks. llmgw otherwise stores no message content
|
||||
@@ -2184,13 +2455,29 @@ llmgw ctl config set firewall.enabled false # every machine's table taken out
|
||||
llmgw ctl config set firewall.enabled true # and put back as it was
|
||||
```
|
||||
|
||||
It is in force within seconds: each node's agent takes its table out or puts
|
||||
it back, and each control plane's `llmgw-firewall-sync.path` (installed by
|
||||
`server init` and `server add`; `sudo llmgw firewall sync --install` on one
|
||||
set up before) does the same there. The rules files stay, so turning it back
|
||||
on restores them. Nodes added while it is off get none. A separate mirror
|
||||
machine and the store's members keep theirs for now. `firewall_networks`
|
||||
changes reach the nodes the same way.
|
||||
It is in force on every machine within a minute, each in its own way:
|
||||
|
||||
- **nodes**: the agent takes its table out or puts it back;
|
||||
- **control planes**: `llmgw-firewall-sync.path`, installed by `server
|
||||
init` and `server add` (`sudo llmgw firewall sync --install` on one set up
|
||||
before);
|
||||
- **a separate mirror machine**: the leading control plane tells the mirror,
|
||||
which writes it to `.firewall.json` in its directory, and the same path
|
||||
unit applies it. `mirror init` installs it; on a mirror set up before, run
|
||||
`sudo llmgw mirror init` again with the same flags;
|
||||
- **store members that are nothing else**: they run no llmgw, so
|
||||
`llmgw-firewall-follow.timer` reads what the leading control plane keeps in
|
||||
the store, from the member's own copy, once a minute. `server init
|
||||
--patroni` installs it; run it again on a cluster set up before.
|
||||
|
||||
The rules files stay, so turning it back on restores them. Nodes added while
|
||||
it is off get none. Networks added to `firewall_networks` reach every machine
|
||||
the same way. One taken out of it stays let in until it is taken out of
|
||||
`/etc/llmgw/firewall.nft` by hand: the rules are only ever added to.
|
||||
|
||||
The leading control plane also tells the mirror machine where every node and
|
||||
control plane is, so a node on a network the mirror had not let in can fetch
|
||||
its weights without a `firewall_networks` entry.
|
||||
|
||||
Only those ports are touched. The table's policy is to accept, so whatever
|
||||
else the machine serves — and whatever firewall it already has — is left as
|
||||
|
||||
+80
-3
@@ -45,6 +45,20 @@ is available to everyone who may use them.
|
||||
A collection stays with the model it was made with: vectors from two
|
||||
models cannot be compared.
|
||||
|
||||
Some embedding models are trained with an instruction before the
|
||||
question and none before the documents. Give it as the entry's
|
||||
`query_prefix`; llmgw puts it before questions only. For
|
||||
Qwen3-Embedding:
|
||||
|
||||
```yaml
|
||||
query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
|
||||
```
|
||||
|
||||
With pgvector 0.7 or later a collection's index is kept at half
|
||||
precision (`halfvec`), which halves its memory and takes models of up to
|
||||
4000 dimensions; with an older pgvector the limit is 2000. A collection
|
||||
keeps the kind it was made with.
|
||||
|
||||
## For administrators
|
||||
|
||||
### Make a collection
|
||||
@@ -132,6 +146,24 @@ Rules that apply:
|
||||
- Every document added or removed is in the audit log (`llmgw ctl audit`),
|
||||
by name. The text is never recorded.
|
||||
|
||||
### A large import
|
||||
|
||||
A few thousand documents ask the embedding model for far more than it can
|
||||
answer at once. Give the key that sends them a band of its own, below the
|
||||
people's, that may wait for room for minutes rather than the 30 seconds the
|
||||
others wait:
|
||||
|
||||
```sh
|
||||
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
|
||||
{name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
### Get it
|
||||
@@ -317,9 +349,11 @@ Behaviour to know about:
|
||||
(Codex, Claude Code) work the same way, since they are answered as chat
|
||||
completions.
|
||||
- **Separate prefix caches.** On vLLM replicas, a request given passages
|
||||
carries a `cache_salt` derived from its organization. One organization's
|
||||
passages then never share cached blocks with another's, so how fast an
|
||||
answer starts says nothing about what another tenant asked.
|
||||
carries a `cache_salt` derived from the collections they came from.
|
||||
Requests that could be given the same passages still share cached blocks,
|
||||
and no one else's do: not another organization's, nor another team's of
|
||||
the same one. How fast an answer starts therefore says nothing about what
|
||||
someone else asked.
|
||||
- **No answer cache.** Answers from knowledge bases are never served from
|
||||
the answer cache, because two keys of one project can read different
|
||||
collections.
|
||||
@@ -340,6 +374,49 @@ a product code or a name.
|
||||
Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
|
||||
`llmgw_kb_retrieval_seconds`.
|
||||
|
||||
## Training an adapter on a collection
|
||||
|
||||
An adapter (LoRA) changes how a model *writes*: the organisation's words,
|
||||
format and tone. It does not reliably teach facts. On the PoC, a small
|
||||
model trained on nine policy documents answered 4 of 9 rephrased questions
|
||||
right and mixed facts up on the rest, while the same questions through the
|
||||
knowledge base got 9 of 9. Use both: the knowledge base for what answers
|
||||
say, an adapter for how they sound.
|
||||
|
||||
```sh
|
||||
# 1. Questions and answers from the collection, written by a model of the
|
||||
# cluster. Read the file through and correct it.
|
||||
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
|
||||
|
||||
# 2. Train on a node. --gpu-memory is what the job needs of a card; it
|
||||
# runs beside an engine when the engine leaves that much free.
|
||||
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
|
||||
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
|
||||
llmgw ctl train show JOB --follow
|
||||
|
||||
# 3. Make it the model's adapter, kept on the mirror.
|
||||
llmgw ctl train publish JOB --restart
|
||||
```
|
||||
|
||||
The node trains with its installed vLLM's torch and a small overlay (peft
|
||||
and accelerate), as the engines' user. For a model llama.cpp serves, it also
|
||||
converts the adapter to GGUF with the installed llama.cpp's own code. What
|
||||
it makes is kept by the control plane (`llmgw ctl train download`).
|
||||
|
||||
An adapter is private like its collection:
|
||||
|
||||
- **It has an owner.** An adapter trained on a collection carries some of
|
||||
what the collection says in its weights, so it belongs to a part of the
|
||||
tree.
|
||||
- By default the owner is the collection's owner.
|
||||
- `--owner` names another, which must be able to read every collection
|
||||
the pairs came from.
|
||||
- Only callers in the owner's part of the tree see the adapter. To anyone
|
||||
else it is a model that does not exist, absent from `/v1/models` too.
|
||||
- **It is public only by choice.** `--public` makes an adapter every
|
||||
caller's. It is refused for pairs from a knowledge base, so it is only for
|
||||
pairs written by hand.
|
||||
|
||||
## How it is kept apart
|
||||
|
||||
Passages live in PostgreSQL, partitioned by organization, with an index per
|
||||
|
||||
@@ -18,6 +18,7 @@ rather than repeated.
|
||||
| `name` | `string` | [Adapter](#adapter) |
|
||||
| `source` | `string` | [Adapter](#adapter) |
|
||||
| `scale` | `float64` | [Adapter](#adapter) |
|
||||
| `owner` | `string` | [Adapter](#adapter) |
|
||||
| `enabled` | `bool` | [Admission](#admission) |
|
||||
| `bands` | `[]Band` | [Admission](#admission) |
|
||||
| `default_band` | `string` | [Admission](#admission) |
|
||||
@@ -59,6 +60,14 @@ rather than repeated.
|
||||
| `tools` | `bool` | [Capabilities](#capabilities) |
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
| `images` | `bool` | [Capabilities](#capabilities) |
|
||||
| `meshes` | `bool` | [Capabilities](#capabilities) |
|
||||
| `workflow` | `string` | [ComfyUI](#comfyui) |
|
||||
| `checkpoint` | `string` | [ComfyUI](#comfyui) |
|
||||
| `output` | `string` | [ComfyUI](#comfyui) |
|
||||
| `nodes` | `[]CustomNode` | [ComfyUI](#comfyui) |
|
||||
| `dir` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `python` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `interval` | `time.Duration` | [Control](#control) |
|
||||
| `base_backoff` | `time.Duration` | [Control](#control) |
|
||||
| `max_backoff` | `time.Duration` | [Control](#control) |
|
||||
@@ -70,11 +79,15 @@ rather than repeated.
|
||||
| `eval_dir` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `repo` | `string` | [CustomNode](#customnode) |
|
||||
| `commit` | `string` | [CustomNode](#customnode) |
|
||||
| `requirements` | `[]string` | [CustomNode](#customnode) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
| `url` | `string` | [Endpoint](#endpoint) |
|
||||
| `api_key` | `string` | [Endpoint](#endpoint) |
|
||||
| `models` | `[]string` | [Endpoint](#endpoint) |
|
||||
| `engine` | `string` | [Endpoint](#endpoint) |
|
||||
| `workflow` | `string` | [Endpoint](#endpoint) |
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
@@ -89,6 +102,9 @@ rather than repeated.
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `output` | `string` | [External](#external) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -117,6 +133,9 @@ rather than repeated.
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
| `docs_sign_in` | `bool` | [Landing](#landing) |
|
||||
| `name` | `string` | [License](#license) |
|
||||
| `url` | `string` | [License](#license) |
|
||||
| `notice` | `string` | [License](#license) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -134,6 +153,8 @@ rather than repeated.
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `comfyui` | `*ComfyUI` | [Model](#model) |
|
||||
| `license` | `*License` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
@@ -144,6 +165,7 @@ rather than repeated.
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
| `max_output` | `int` | [Model](#model) |
|
||||
| `capabilities` | `Capabilities` | [Model](#model) |
|
||||
| `query_prefix` | `string` | [Model](#model) |
|
||||
| `role` | `string` | [Model](#model) |
|
||||
| `guard` | `*GuardModel` | [Model](#model) |
|
||||
| `origin` | `*Origin` | [Model](#model) |
|
||||
@@ -209,6 +231,7 @@ rather than repeated.
|
||||
| `engine` | `string` | [Profile](#profile) |
|
||||
| `engine_version` | `string` | [Profile](#profile) |
|
||||
| `gpu_class` | `string` | [Profile](#profile) |
|
||||
| `source` | `string` | [Profile](#profile) |
|
||||
| `tp` | `int` | [Profile](#profile) |
|
||||
| `pp` | `int` | [Profile](#profile) |
|
||||
| `gpus_per_node` | `int` | [Profile](#profile) |
|
||||
@@ -341,9 +364,12 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_mesh` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
| `priority` | `int` | [admission.Band](#admissionband) |
|
||||
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) |
|
||||
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
|
||||
| `default_band` | `string` | [admission.Config](#admissionconfig) |
|
||||
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
|
||||
@@ -373,14 +399,18 @@ rather than repeated.
|
||||
| `name` | `string` | [config.Band](#configband) |
|
||||
| `priority` | `int` | [config.Band](#configband) |
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
| `queue_ttl` | `time.Duration` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `per_mesh` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -426,6 +456,16 @@ weights are.
|
||||
|
||||
Scale is how strongly it is applied; zero takes 1.
|
||||
|
||||
### `owner`
|
||||
|
||||
`string`
|
||||
|
||||
Owner is the part of the tenancy tree the adapter belongs to —
|
||||
org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
|
||||
it see it: to anyone else it is a model that does not exist. An
|
||||
adapter trained on an organisation's documents carries some of what
|
||||
they say in its weights. Empty is every caller the model allows.
|
||||
|
||||
## Admission
|
||||
|
||||
### `enabled`
|
||||
@@ -704,6 +744,70 @@ token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
||||
|
||||
`bool`
|
||||
|
||||
### `images`
|
||||
|
||||
`bool`
|
||||
|
||||
Images is a model that makes pictures, asked for at
|
||||
/v1/images/generations. Implied by an external model whose API is
|
||||
ComfyUI's.
|
||||
|
||||
### `meshes`
|
||||
|
||||
`bool`
|
||||
|
||||
Meshes is a model that makes 3D models from a picture, asked for as a
|
||||
job at /v1/3d/generations. Implied by a ComfyUI model whose output is
|
||||
meshes.
|
||||
|
||||
## ComfyUI
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is a built-in's name (flux-schnell, the default, or
|
||||
flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
format.
|
||||
|
||||
### `checkpoint`
|
||||
|
||||
`string`
|
||||
|
||||
Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
source's own name unless set, which a source that is a set of files
|
||||
must.
|
||||
|
||||
### `output`
|
||||
|
||||
`string`
|
||||
|
||||
Output is what the workflow makes: pictures (the default), asked for
|
||||
at /v1/images/generations, or meshes, asked for as a job at
|
||||
/v1/3d/generations from a picture.
|
||||
|
||||
### `nodes`
|
||||
|
||||
`[]CustomNode`
|
||||
|
||||
Nodes are custom nodes the workflow needs, put in each replica's own
|
||||
custom_nodes at the commit named.
|
||||
|
||||
## ComfyUIConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
`string`
|
||||
|
||||
Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
engines run as.
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
|
||||
## Control
|
||||
|
||||
### `interval`
|
||||
@@ -789,6 +893,31 @@ see the claim. Long enough to survive a slow query or a brief network
|
||||
stall, short enough that a standby takes over before anyone has finished
|
||||
reading the alert. Zero takes the default.
|
||||
|
||||
## CustomNode
|
||||
|
||||
### `repo`
|
||||
|
||||
`string`
|
||||
|
||||
Repo is the repository, https only.
|
||||
|
||||
### `commit`
|
||||
|
||||
`string`
|
||||
|
||||
Commit is the full commit the node is used at: a branch moves, and a
|
||||
node that changed under a model is a model that changed unasked.
|
||||
|
||||
### `requirements`
|
||||
|
||||
`[]string`
|
||||
|
||||
Requirements are the PyPI packages the node needs, by name and version
|
||||
range only. They are installed beside the replica, never into
|
||||
ComfyUI's environment: a package named here at the range named, and
|
||||
every other one at the version ComfyUI already runs. One named here
|
||||
is first on ComfyUI's path, so its range must suit ComfyUI too.
|
||||
|
||||
## Endpoint
|
||||
|
||||
### `id`
|
||||
@@ -817,6 +946,13 @@ with one.
|
||||
Engine names the inference engine, which decides what its metrics are
|
||||
called. Empty means vLLM's names.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
name or its JSON; flux-schnell unless set.
|
||||
|
||||
### `pool`
|
||||
|
||||
`string`
|
||||
@@ -908,6 +1044,29 @@ a model that federation added (`llmgw ctl federation add`) rather
|
||||
than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
goes with the peer's own list, and is sent only while the peer is up.
|
||||
|
||||
### `api`
|
||||
|
||||
`string`
|
||||
|
||||
API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
sent the workflow below with the request's prompt in it. Model is then
|
||||
the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
(flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
plane.
|
||||
|
||||
### `output`
|
||||
|
||||
`string`
|
||||
|
||||
Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1138,6 +1297,25 @@ llmgw in general — no address, model or key of this installation —
|
||||
which the project's repository publishes anyway. On, a visitor is sent
|
||||
to sign in, and without accounts the portal needs an API key.
|
||||
|
||||
## License
|
||||
|
||||
### `name`
|
||||
|
||||
`string`
|
||||
|
||||
Name is the licence's own: MIT, Apache-2.0.
|
||||
|
||||
### `url`
|
||||
|
||||
`string`
|
||||
|
||||
### `notice`
|
||||
|
||||
`string`
|
||||
|
||||
Notice is what must accompany what was made: attribution, terms on
|
||||
the output, what the weights were trained on.
|
||||
|
||||
## LlamaCppConfig
|
||||
|
||||
### `bin`
|
||||
@@ -1294,6 +1472,25 @@ allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`*ComfyUI`
|
||||
|
||||
ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
pictures: the workflow, and the file it loads.
|
||||
|
||||
Takes a [ComfyUI](#comfyui) block.
|
||||
|
||||
### `license`
|
||||
|
||||
`*License`
|
||||
|
||||
License is the terms of what the model makes, for a model whose
|
||||
output somebody ships: shown with every result and written into a
|
||||
mesh.
|
||||
|
||||
Takes a [License](#license) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
@@ -1378,6 +1575,16 @@ Capabilities is what this model can do beyond plain completion.
|
||||
|
||||
Takes a [Capabilities](#capabilities) block.
|
||||
|
||||
### `query_prefix`
|
||||
|
||||
`string`
|
||||
|
||||
QueryPrefix is put before a question an embedding model embeds for a
|
||||
knowledge-base search; documents are embedded without it. Models
|
||||
trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
"Instruct: Given a web search query, retrieve relevant passages that
|
||||
answer the query\nQuery: " for one.
|
||||
|
||||
### `role`
|
||||
|
||||
`string`
|
||||
@@ -1815,6 +2022,16 @@ version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not
|
||||
|
||||
`string`
|
||||
|
||||
### `source`
|
||||
|
||||
`string`
|
||||
|
||||
Source is the weights this profile loads, when not the model's: an
|
||||
embedding model served by vLLM from its repository on a card and by
|
||||
llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
loads the model's. Updates and canaries move the model's source; a
|
||||
profile with its own keeps it.
|
||||
|
||||
### `tp`
|
||||
|
||||
`int`
|
||||
@@ -2878,6 +3095,21 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is what one picture costs, for a model that makes them: a
|
||||
picture's cost is the card's time, which its prompt's few tokens do
|
||||
not measure.
|
||||
|
||||
### `per_mesh`
|
||||
|
||||
`int64`
|
||||
|
||||
PerMesh is what one 3D model costs: minutes of a card, which no
|
||||
token count measures.
|
||||
|
||||
## admission.Band
|
||||
|
||||
### `name`
|
||||
@@ -2897,6 +3129,15 @@ Priority orders bands; higher is served first.
|
||||
SaturationCeiling is the pool saturation above which this band waits.
|
||||
1.0 means "never hold this band back".
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL, set, is how long this band's requests may wait instead of
|
||||
the controller's: a band for bulk work — a knowledge-base import, a
|
||||
backfill — whose requests had rather wait ten minutes behind people
|
||||
than be turned away after thirty seconds.
|
||||
|
||||
## admission.Config
|
||||
|
||||
### `bands`
|
||||
@@ -3110,6 +3351,14 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`float64`
|
||||
|
||||
### `queue_ttl`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
QueueTTL is how long this band's requests may wait for room; the
|
||||
admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
had rather wait behind people than be turned away.
|
||||
|
||||
## config.Price
|
||||
|
||||
### `prompt_per_1k`
|
||||
@@ -3120,6 +3369,19 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is the cost of one picture, for a model that makes them
|
||||
(/v1/images/generations), on top of its prompt's tokens.
|
||||
|
||||
### `per_mesh`
|
||||
|
||||
`int64`
|
||||
|
||||
PerMesh is the cost of one 3D model (/v1/3d/generations).
|
||||
|
||||
## engines.Config
|
||||
|
||||
### `disabled`
|
||||
@@ -3163,6 +3425,14 @@ Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
Takes a [SimConfig](#simconfig) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`ComfyUIConfig`
|
||||
|
||||
ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
|
||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||
|
||||
## guard.Config
|
||||
|
||||
### `enabled`
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images, u.Meshes)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,17 +39,27 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000); got != 4000 {
|
||||
if got := p.Cost(1000, 1000, 0, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3, 0); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
t.Error("a model priced per picture reads as free, and would pass a spent budget")
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroPricedModelsStayAvailableWhenTheBudgetIsSpent(t *testing.T) {
|
||||
// This is what makes budget fallback work: a key with nothing left can
|
||||
// still be served something that costs nothing.
|
||||
|
||||
@@ -18,6 +18,13 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
// PerMesh is what one 3D model costs: minutes of a card, which no
|
||||
// token count measures.
|
||||
PerMesh int64 `koanf:"per_mesh"`
|
||||
}
|
||||
|
||||
// Zero reports whether this model spends nothing.
|
||||
@@ -25,14 +32,17 @@ type Price struct {
|
||||
// It matters beyond arithmetic: a zero-cost model stays available to a key
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 }
|
||||
func (p Price) Zero() bool {
|
||||
return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 && p.PerMesh == 0
|
||||
}
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000)
|
||||
func (p Price) Cost(promptTokens, completionTokens, images, meshes int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage + max(meshes, 0)*p.PerMesh
|
||||
}
|
||||
|
||||
func divUp(n, d int64) int64 {
|
||||
|
||||
@@ -34,6 +34,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
@@ -140,6 +141,9 @@ type Agent struct {
|
||||
// settingsCheck says whether the cluster's settings for this agent
|
||||
// would load beside its agent.yaml; see SetSettingsCheck.
|
||||
settingsCheck func([]byte) error
|
||||
|
||||
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
|
||||
trainer *train.Runner
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -186,6 +190,18 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
kvSubs: map[string]*kvevents.Subscriber{},
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
runAs := cfg.Superv.RunAs
|
||||
if runAs == "" {
|
||||
runAs = supervisor.DefaultRunAs
|
||||
}
|
||||
// Where the engines are, as the installer has it when the file says
|
||||
// nothing.
|
||||
enginesDir := cfg.EnginesDir
|
||||
if enginesDir == "" {
|
||||
enginesDir = provision.DefaultRoot
|
||||
}
|
||||
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
|
||||
RunAs: runAs, Log: log})
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -886,6 +902,10 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.applyFirewall(m.Firewall)
|
||||
case *llmgwv1.ControlMessage_AgentSettings:
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_TrainStart:
|
||||
go a.trainJob(ctx, m.TrainStart, out)
|
||||
case *llmgwv1.ControlMessage_TrainCancel:
|
||||
a.trainer.Cancel(m.TrainCancel.GetJobId())
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
go func() {
|
||||
a.log.Info("restarting, as the control plane asks; the engines keep serving",
|
||||
|
||||
@@ -0,0 +1,137 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// ComfyUIConfig locates a ComfyUI installation.
|
||||
type ComfyUIConfig struct {
|
||||
// Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
// engines run as.
|
||||
Dir string `koanf:"dir"`
|
||||
// Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
Python string `koanf:"python"`
|
||||
}
|
||||
|
||||
// ComfyUI runs ComfyUI as an engine, behind llmgw's gate (llmgw
|
||||
// comfyui-gate): ComfyUI asks for no key, so it listens on the loopback and
|
||||
// the gate, which does, on the replica's port. Its replicas make pictures
|
||||
// for /v1/images/generations.
|
||||
type ComfyUI struct {
|
||||
cfg ComfyUIConfig
|
||||
installed atomic.Pointer[string]
|
||||
}
|
||||
|
||||
// Name identifies the engine in the catalog.
|
||||
func (*ComfyUI) Name() string { return "comfyui" }
|
||||
|
||||
// HealthPath is the gate's, which answers once ComfyUI does.
|
||||
func (*ComfyUI) HealthPath() string { return "/health" }
|
||||
|
||||
// MetricsPath is empty: ComfyUI publishes no metrics, and the router reads
|
||||
// none from it.
|
||||
func (*ComfyUI) MetricsPath() string { return "" }
|
||||
|
||||
// ServesHTTP is true: a ComfyUI replica is one process on one node.
|
||||
func (*ComfyUI) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// UseInstalled points the engine at the environment the agent installed:
|
||||
// its interpreter, with ComfyUI's code beside its bin, in comfyui/.
|
||||
func (c *ComfyUI) UseInstalled(python string) { c.installed.Store(&python) }
|
||||
|
||||
// Configured reports whether the configuration names a ComfyUI.
|
||||
func (c *ComfyUI) Configured() bool { return c.cfg.Dir != "" }
|
||||
|
||||
func (c *ComfyUI) dir() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return filepath.Join(filepath.Dir(filepath.Dir(*p)), "comfyui")
|
||||
}
|
||||
return c.cfg.Dir
|
||||
}
|
||||
|
||||
func (c *ComfyUI) python() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return *p
|
||||
}
|
||||
if c.cfg.Python != "" {
|
||||
return c.cfg.Python
|
||||
}
|
||||
return filepath.Join(c.cfg.Dir, ".venv", "bin", "python")
|
||||
}
|
||||
|
||||
var comfyVersion = regexp.MustCompile(`__version__\s*=\s*"([^"]+)"`)
|
||||
|
||||
// Probe reports ComfyUI's version, from the file its build writes it to:
|
||||
// importing ComfyUI to ask would load torch and look for cards.
|
||||
func (c *ComfyUI) Probe(context.Context) (string, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return "", fmt.Errorf("no ComfyUI directory is configured")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "main.py")); err != nil {
|
||||
return "", fmt.Errorf("%s is not ComfyUI: %w", dir, err)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "comfyui_version.py"))
|
||||
if err != nil {
|
||||
return Unknown, nil //nolint:nilerr // installed, of a version it does not say
|
||||
}
|
||||
if m := comfyVersion.FindSubmatch(b); m != nil {
|
||||
return string(m[1]), nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card, on one node. ComfyUI runs a
|
||||
// workflow on one device and has no tensor parallelism to spread it by.
|
||||
func (*ComfyUI) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: ComfyUI runs on one GPU of one node", p.Name)
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
}
|
||||
|
||||
// Commands is the gate, which starts ComfyUI itself.
|
||||
func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return nil, fmt.Errorf("comfyui: no ComfyUI directory is configured (engines.comfyui.dir)")
|
||||
}
|
||||
bin, err := os.Executable()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := s.Profile
|
||||
args := []string{"comfyui-gate",
|
||||
"--listen", net.JoinHostPort(s.Host(), strconv.Itoa(s.HTTPPort)),
|
||||
"--model", p.ServedModelName,
|
||||
"--weights", p.ModelPath,
|
||||
// Relative to where engines run — their user's home — so it is
|
||||
// theirs to write.
|
||||
"--state", filepath.Join("comfyui", s.ReplicaID),
|
||||
"--python", c.python(),
|
||||
"--comfyui", dir,
|
||||
"--",
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
env := baseEnv(s)
|
||||
if s.APIKey != "" {
|
||||
env["LLMGW_ENGINE_KEY"] = s.APIKey
|
||||
}
|
||||
// A model's custom nodes, which only the catalog sets — a profile's env
|
||||
// may not name the variable — and so taken before baseEnv drops it.
|
||||
if nodes := p.Env[comfyNodesEnv]; nodes != "" {
|
||||
env[comfyNodesEnv] = nodes
|
||||
}
|
||||
return []Command{{Name: "comfyui", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
|
||||
// comfyNodesEnv is the custom nodes the catalog gives a ComfyUI model, as
|
||||
// JSON; catalog.NodesEnv and comfy.NodesEnv by the same name.
|
||||
const comfyNodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
@@ -0,0 +1,80 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestComfyUIRunsBehindTheGateWithItsKeyOutOfSight(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/comfyui"}}
|
||||
cmds, err := c.Commands(ShardSpec{
|
||||
ReplicaID: "r-flux-1", HTTPPort: 8100, BindAddr: "10.0.0.147", APIKey: "sekret", GPUIndices: []int{0},
|
||||
Profile: Profile{Name: "p", Engine: "comfyui", ModelPath: "/w/flux.safetensors", ServedModelName: "flux",
|
||||
Args: []string{"--disable-smart-memory"}},
|
||||
})
|
||||
if err != nil || len(cmds) != 1 {
|
||||
t.Fatalf("%v %v", cmds, err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
for _, want := range []string{"comfyui-gate", "--listen 10.0.0.147:8100", "--model flux", "--weights /w/flux.safetensors",
|
||||
"--state comfyui/r-flux-1", "--python /opt/comfyui/.venv/bin/python", "--comfyui /opt/comfyui", "-- --disable-smart-memory"} {
|
||||
if !strings.Contains(args, want) {
|
||||
t.Errorf("%q not in %q", want, args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(args, "sekret") {
|
||||
t.Error("the key is on the command line, where every user can read it")
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "LLMGW_ENGINE_KEY=sekret") || !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0") {
|
||||
t.Error("the key or the card is not in the environment")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIRunsOnOneCard(t *testing.T) {
|
||||
c := &ComfyUI{}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 2}, h100(2)); err == nil {
|
||||
t.Error("a two-card ComfyUI profile was accepted")
|
||||
}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 1}, h100(1)); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if _, err := c.Commands(ShardSpec{Profile: Profile{ModelPath: "/w"}}); err == nil {
|
||||
t.Error("commands without a ComfyUI directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsOfferedByTheVersionItsBuildWrote(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: dir}}
|
||||
if _, err := c.Probe(t.Context()); err == nil {
|
||||
t.Error("a directory without main.py passed for ComfyUI")
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "main.py"), nil, 0o644)
|
||||
_ = os.WriteFile(filepath.Join(dir, "comfyui_version.py"), []byte("__version__ = \"0.39.2\"\n"), 0o644)
|
||||
if v, err := c.Probe(t.Context()); err != nil || v != "0.39.2" {
|
||||
t.Errorf("%q %v", v, err)
|
||||
}
|
||||
if got := NewRegistry(Config{ComfyUI: ComfyUIConfig{Dir: dir}}).ProbeConfigured(t.Context()); got["comfyui"] != "0.39.2" {
|
||||
t.Errorf("a configured ComfyUI is reported as %v", got)
|
||||
}
|
||||
if err := UnsafeArgs("comfyui", []string{"--enable-manager"}); err == nil {
|
||||
t.Error("a profile may install ComfyUI nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnInstalledComfyUIIsFoundBesideItsInterpreter(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/by-hand"}}
|
||||
c.UseInstalled("/opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python")
|
||||
cmds, err := c.Commands(ShardSpec{ReplicaID: "r", HTTPPort: 1, Profile: Profile{ModelPath: "/w", ServedModelName: "m"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
if !strings.Contains(args, "--python /opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python") ||
|
||||
!strings.Contains(args, "--comfyui /opt/llmgw/engines/comfyui/0.39.2-cu13/comfyui") {
|
||||
t.Errorf("the installed ComfyUI is not what runs: %s", args)
|
||||
}
|
||||
}
|
||||
@@ -214,6 +214,7 @@ func NewRegistry(cfg Config) *Registry {
|
||||
r.Add(&LlamaCpp{cfg: cfg.LlamaCpp})
|
||||
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
||||
r.Add(&Sim{cfg: cfg.Sim})
|
||||
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
||||
return r
|
||||
}
|
||||
|
||||
@@ -262,6 +263,8 @@ type Config struct {
|
||||
// offers it only when its interpreter is named here.
|
||||
TRTLLM TRTLLMConfig `koanf:"trtllm"`
|
||||
Sim SimConfig `koanf:"sim"`
|
||||
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
||||
}
|
||||
|
||||
// --- shared helpers ---
|
||||
|
||||
@@ -477,7 +477,7 @@ func TestValidateRequiresModelPath(t *testing.T) {
|
||||
|
||||
func TestRegistry(t *testing.T) {
|
||||
r := NewRegistry(Config{})
|
||||
if got := r.Names(); !slices.Equal(got, []string{"llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
t.Errorf("registry contents: %v", got)
|
||||
}
|
||||
if _, err := r.Get("vllm"); err != nil {
|
||||
|
||||
@@ -119,6 +119,19 @@ var flagsNever = map[string]map[string]string{
|
||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||
},
|
||||
// ComfyUI's main.py, behind the gate that sets where it listens and
|
||||
// where its files are.
|
||||
"comfyui": {
|
||||
"--listen": "is set by the agent", "--base-directory": "is set by the agent",
|
||||
"--extra-model-paths-config": "is set by the agent",
|
||||
"--output-directory": "writes files", "--temp-directory": "writes files",
|
||||
"--input-directory": "reads files", "--user-directory": "writes files",
|
||||
"--database-url": "writes a database elsewhere", "--front-end-root": "serves a directory",
|
||||
"--enable-cors-header": "lets other sites' pages call it",
|
||||
"--enable-manager": "installs nodes, which run Python",
|
||||
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
||||
"--comfy-api-base": "sends work to another service",
|
||||
},
|
||||
// trtllm-serve spells its flags with underscores, so the shared list's
|
||||
// hyphenated names do not reach them.
|
||||
"trtllm": {
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// comfyuiSource is a ComfyUI release archive as GitHub makes one, and the
|
||||
// recipe pointed at it.
|
||||
func comfyuiSource(t *testing.T, in *Installer, rel *release, digest string) {
|
||||
t.Helper()
|
||||
archive := tarGz(t, map[string]string{
|
||||
"ComfyUI-0.39.2/main.py": "print('comfy')\n",
|
||||
"ComfyUI-0.39.2/comfyui_version.py": "__version__ = \"0.39.2\"\n",
|
||||
"ComfyUI-0.39.2/requirements.txt": "comfyui-frontend-package==1.53.10\ntorch\ntorchvision\n" +
|
||||
"numpy>=1.25.0\n\n#non essential dependencies:\nkornia>=0.7.1\n",
|
||||
}, nil)
|
||||
rel.assets["v0.39.2.tar.gz"] = archive
|
||||
sum := sha256.Sum256(archive)
|
||||
if digest == "" {
|
||||
digest = hex.EncodeToString(sum[:])
|
||||
}
|
||||
old := comfyuiReleases
|
||||
comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{"0.39.2": {digest, "2026-10-07T21:58:07Z"}}
|
||||
t.Cleanup(func() { comfyuiReleases = old })
|
||||
}
|
||||
|
||||
var turingNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
|
||||
NVIDIADriver: "580.82.07", MinComputeCapability: 7.5}
|
||||
|
||||
func TestComfyUIIsInstalledFromItsSourceCheckedAgainstTheRecipe(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready {
|
||||
t.Fatalf("not ready: %+v", got[0])
|
||||
}
|
||||
dir := filepath.Join(in.Root, "comfyui", "0.39.2-cu13")
|
||||
if got[0].Binary != filepath.Join(dir, "bin", "python") {
|
||||
t.Errorf("binary %s, want the environment's interpreter", got[0].Binary)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "comfyui", "main.py")); err != nil {
|
||||
t.Errorf("ComfyUI's code is not beside the interpreter: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
for _, want := range []string{
|
||||
// Not PyPI's date, which ComfyUI has none of: the recipe's, a day on.
|
||||
"--exclude-newer 2026-10-08T21:58:07Z",
|
||||
"comfyui-frontend-package==1.53.10", "numpy>=1.25.0", "kornia>=0.7.1", "torch",
|
||||
"--require-hashes",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
if strings.Contains(all, "/pypi/") {
|
||||
t.Error("PyPI was asked when ComfyUI was published")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIOnAnOlderDriverTakesPyTorchsCUDA12Torch(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
host := turingNode
|
||||
host.NVIDIADriver = "550.163.01"
|
||||
in, rel := pythonInstaller(t, host, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("got %+v", got[0])
|
||||
}
|
||||
var project string
|
||||
for _, c := range sys.commands {
|
||||
if strings.HasPrefix(c, "pyproject: ") {
|
||||
project = c
|
||||
}
|
||||
}
|
||||
// torch from PyTorch's index alone, named once, as the build names it.
|
||||
if !strings.Contains(project, `torch = { index = "extra" }`) || strings.Count(project, `"torch"`) != 1 {
|
||||
t.Errorf("torch is not from PyTorch's index alone:\n%s", project)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIWhoseSourceIsNotTheOneTheRecipeNamesIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, strings.Repeat("ab", 32))
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "does not match") {
|
||||
t.Fatalf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") {
|
||||
t.Errorf("installed from a source that did not match: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsInstalledOnlyWhereItCanRun(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, ubuntu, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}}); got[0].State != Skipped {
|
||||
t.Errorf("a node with no GPU: %+v", got[0])
|
||||
}
|
||||
in, _ = pythonInstaller(t, turingNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.1.0"}})
|
||||
if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "knows the source digest of ComfyUI 0.39.2") {
|
||||
t.Errorf("a version without a digest: %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequirementsFileIsReadForPackagesAndNothingElse(t *testing.T) {
|
||||
got, err := requirementsIn("torch\ntorchvision>=0.20\nPillow\n# a comment\naiohttp>=3.11 # inline\n", pytorchPackages)
|
||||
if err != nil || strings.Join(got, ",") != "Pillow,aiohttp>=3.11" {
|
||||
t.Errorf("%v %v", got, err)
|
||||
}
|
||||
for _, bad := range []string{"-r other.txt", "--index-url https://evil", "x @ https://evil/x.whl", "-e ."} {
|
||||
if _, err := requirementsIn(bad+"\n", nil); err == nil {
|
||||
t.Errorf("%q was accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -66,12 +66,17 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
|
||||
}
|
||||
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
var asOf string
|
||||
if !b.AsOf.IsZero() {
|
||||
asOf = b.AsOf.UTC().Format(time.RFC3339)
|
||||
} else {
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
asOf = published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
}
|
||||
asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
|
||||
cache := filepath.Join(in.root(), ".uv-cache")
|
||||
env := []string{
|
||||
@@ -96,7 +101,14 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
reqs := b.Requirements
|
||||
reqs := slices.Clone(b.Requirements)
|
||||
if b.Source != nil {
|
||||
more, err := in.unpackSource(ctx, w, b, tmp, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reqs = append(reqs, more...)
|
||||
}
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
@@ -320,3 +332,68 @@ func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Buil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackSource fetches an engine's source archive, checks it against the
|
||||
// digest the recipe holds, puts its code in the environment, and returns the
|
||||
// requirements its requirements file names — less the packages a second
|
||||
// index gives, which the build names itself.
|
||||
func (in *Installer) unpackSource(ctx context.Context, w Want, b Build, env string, st *Status) ([]string, error) {
|
||||
src := b.Source
|
||||
url := src.URL
|
||||
if strings.HasPrefix(url, "/") {
|
||||
url = strings.TrimSuffix(in.downloadBase(), "/") + url
|
||||
}
|
||||
scratch := env + ".source"
|
||||
_ = os.RemoveAll(scratch)
|
||||
if err := os.MkdirAll(scratch, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(scratch) //nolint:errcheck // scratch
|
||||
archive := filepath.Join(scratch, "source.tar.gz")
|
||||
name := w.Engine + " " + w.Version + "'s source"
|
||||
if err := in.download(ctx, url, name, archive, src.SHA256, st); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
in.progress(st, "unpacking %s", name)
|
||||
if err := untarGz(archive, scratch); err != nil {
|
||||
return nil, fmt.Errorf("unpacking %s: %w", name, err)
|
||||
}
|
||||
dst := filepath.Join(env, src.Dir)
|
||||
if err := os.Rename(filepath.Join(scratch, src.Top), dst); err != nil {
|
||||
return nil, fmt.Errorf("%s has no %s at its top: %w", name, src.Top, err)
|
||||
}
|
||||
if src.Requirements == "" {
|
||||
return nil, nil
|
||||
}
|
||||
f, err := os.ReadFile(filepath.Join(dst, src.Requirements))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
return requirementsIn(string(f), b.IndexPackages)
|
||||
}
|
||||
|
||||
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
||||
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
||||
// or install what the lock never saw.
|
||||
func requirementsIn(text string, skip []string) ([]string, error) {
|
||||
var out []string
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line, _, _ = strings.Cut(line, "#")
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "-") || strings.Contains(line, "://") || strings.Contains(line, " @ ") {
|
||||
return nil, fmt.Errorf("the requirements name %q, which is not a package from PyPI", line)
|
||||
}
|
||||
name := strings.ToLower(line)
|
||||
if i := strings.IndexAny(name, "<>=!~;[ "); i >= 0 {
|
||||
name = name[:i]
|
||||
}
|
||||
if slices.Contains(skip, name) {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -53,6 +53,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return []byte("error: Hash mismatch for torch\n"), errExit
|
||||
}
|
||||
venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
|
||||
_ = os.WriteFile(filepath.Join(venv, "bin", "python"), []byte("#!/bin/sh\n"), 0o755)
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
@@ -68,12 +69,31 @@ type Build struct {
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// Source, for a Python engine that is not on PyPI, is its own code: an
|
||||
// archive pinned by its digest, unpacked into the environment, whose
|
||||
// requirements file is installed with Requirements. AsOf is then the
|
||||
// date its dependencies are resolved as of — the day after its version
|
||||
// was published, which PyPI cannot be asked for.
|
||||
Source *Source
|
||||
AsOf time.Time
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Source is an engine's code as an archive.
|
||||
type Source struct {
|
||||
// URL is fetched as is; SHA256 is what it must hash to, written into
|
||||
// the recipe because a project's source archives come with no published
|
||||
// digest of their own.
|
||||
URL, SHA256 string
|
||||
// Top is the archive's one top directory, put in the environment as Dir.
|
||||
Top, Dir string
|
||||
// Requirements is the requirements file, in Dir.
|
||||
Requirements string
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
@@ -104,6 +124,7 @@ var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"comfyui": {Candidates: comfyuiCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
@@ -556,3 +577,78 @@ func orUnknown(s string) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// --- ComfyUI ---
|
||||
|
||||
// comfyuiReleases are the ComfyUI versions this agent installs: each source
|
||||
// archive's digest, taken when it was added here and confirmed from a second
|
||||
// machine, and when the version was published. GitHub publishes no digest
|
||||
// for a tag's archive, so a version not listed is refused rather than
|
||||
// installed unchecked.
|
||||
var comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{
|
||||
"0.39.2": {"1d655e22cf546700fc1d6d2536e27701695c5832a51f63bb16c41866ac09be0b", "2026-10-07T21:58:07Z"},
|
||||
}
|
||||
|
||||
// ComfyUIRepo is where ComfyUI's source is published.
|
||||
const ComfyUIRepo = "Comfy-Org/ComfyUI"
|
||||
|
||||
// comfyuiDir is where ComfyUI's code is put in its environment: main.py is
|
||||
// <environment>/comfyui/main.py, beside bin/python.
|
||||
const comfyuiDir = "comfyui"
|
||||
|
||||
// comfyuiCandidates are the ComfyUI builds for a machine: its source, in a
|
||||
// Python environment with the torch for the node's driver — PyPI's, built
|
||||
// for CUDA 13, or PyTorch's CUDA 12.9 build for a driver CUDA 13 does not
|
||||
// run on.
|
||||
func comfyuiCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("ComfyUI", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rel, ok := comfyuiReleases[version]
|
||||
if !ok {
|
||||
known := make([]string, 0, len(comfyuiReleases))
|
||||
for v := range comfyuiReleases {
|
||||
known = append(known, v)
|
||||
}
|
||||
slices.Sort(known)
|
||||
return nil, fmt.Errorf("this agent knows the source digest of ComfyUI %s, not of %s; "+
|
||||
"install that one by hand and name it in agent.yaml (engines.comfyui.dir)",
|
||||
strings.Join(known, ", "), version)
|
||||
}
|
||||
published, err := time.Parse(time.RFC3339, rel.published)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
build := func(variant string) Build {
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Source: &Source{
|
||||
URL: "/" + ComfyUIRepo + "/archive/refs/tags/v" + version + ".tar.gz",
|
||||
SHA256: rel.sha256, Top: "ComfyUI-" + version, Dir: comfyuiDir,
|
||||
Requirements: "requirements.txt",
|
||||
},
|
||||
AsOf: published.Add(24 * time.Hour),
|
||||
// ComfyUI is started as a script, so what the agent is given is
|
||||
// the environment's interpreter; the proof is importing what it
|
||||
// imports first, from where its code is.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", `import os, sys, torch, aiohttp, safetensors
|
||||
d = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(sys.executable))), "` + comfyuiDir + `")
|
||||
assert os.path.isfile(os.path.join(d, "main.py")), d + " has no main.py"
|
||||
print("comfyui version", "` + version + `", "on torch", torch.__version__)`},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 15,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := build("cu129")
|
||||
cu129.ExtraIndex, cu129.IndexPackages = pytorchCU129, pytorchPackages
|
||||
return pick("ComfyUI "+version, variant, []candidate{
|
||||
{build("cu13"), gpuNeeds{minDriverForCUDA13, 7.5}, "ComfyUI " + version + " on CUDA 13's torch"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, 7.0}, "ComfyUI " + version + " on CUDA 12.9's torch"},
|
||||
}, h)
|
||||
}
|
||||
@@ -0,0 +1,139 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// An engine's output goes to a file of its own, not to a pipe of the
|
||||
// agent's. A pipe has one reader, and that reader is the agent that
|
||||
// started the engine: when that agent restarts, the engine it left running
|
||||
// — and adopts a moment later — writes to a pipe nobody reads, and its next
|
||||
// write past the pipe's buffer fails with EPIPE. A Python engine's progress
|
||||
// bar then raises, and takes the request it was reporting on with it. A
|
||||
// file has no reader to lose: the engine appends to it, every agent reads
|
||||
// it, and the output from before a restart is there for the one after.
|
||||
|
||||
// maxOutputBytes bounds one process's file: past it the file is emptied and
|
||||
// the process goes on appending to it. A progress bar rewriting itself for
|
||||
// an hour would otherwise fill the disk.
|
||||
const maxOutputBytes = 64 << 20
|
||||
|
||||
// outputPath is where a shard's process writes.
|
||||
func (s *Supervisor) outputPath(shardID, name string) string {
|
||||
if name != "" {
|
||||
return filepath.Join(s.cfg.WorkDir, "shard-"+shardID+"."+name+".log")
|
||||
}
|
||||
return filepath.Join(s.cfg.WorkDir, "shard-"+shardID+".log")
|
||||
}
|
||||
|
||||
// openOutput is the file a process is started with: new, for a process
|
||||
// that is new.
|
||||
func (s *Supervisor) openOutput(shardID, name string) (*os.File, error) {
|
||||
return os.OpenFile(s.outputPath(shardID, name), os.O_CREATE|os.O_WRONLY|os.O_APPEND|os.O_TRUNC, 0o600) //nolint:gosec // the agent's own directory
|
||||
}
|
||||
|
||||
// follow reads a process's file into the shard's log as it is written,
|
||||
// until the shard's output is stopped and the file has been read to its
|
||||
// end. A file an earlier agent began is read from the start, so the output
|
||||
// from before a restart is in the log too.
|
||||
func (s *Supervisor) follow(sh *Shard, path, name string) {
|
||||
defer sh.outDone.Done()
|
||||
f, err := os.Open(path) //nolint:gosec // the agent's own directory
|
||||
if err != nil {
|
||||
sh.logs.add("[" + name + "] output unavailable: " + err.Error())
|
||||
return
|
||||
}
|
||||
defer f.Close()
|
||||
var offset int64
|
||||
var rest []byte
|
||||
buf := make([]byte, 64<<10)
|
||||
read := func() {
|
||||
for {
|
||||
st, err := f.Stat()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if st.Size() < offset {
|
||||
// Emptied, by this agent or another: begin again.
|
||||
offset, rest = 0, nil
|
||||
}
|
||||
if st.Size() > maxOutputBytes {
|
||||
_ = os.Truncate(path, 0)
|
||||
offset, rest = 0, nil
|
||||
continue
|
||||
}
|
||||
n, err := f.ReadAt(buf, offset)
|
||||
if n > 0 {
|
||||
offset += int64(n)
|
||||
rest = append(rest, buf[:n]...)
|
||||
for {
|
||||
i := bytes.IndexByte(rest, '\n')
|
||||
if i < 0 {
|
||||
break
|
||||
}
|
||||
s.addLine(sh, name, rest[:i])
|
||||
rest = rest[i+1:]
|
||||
}
|
||||
// A progress bar is a line rewritten with \r and never ended:
|
||||
// its latest state is what is wanted, not a line a megabyte
|
||||
// long. Kept, as llama.cpp and tqdm write them.
|
||||
if len(rest) > 1<<20 {
|
||||
s.addLine(sh, name, rest)
|
||||
rest = nil
|
||||
}
|
||||
}
|
||||
if err != nil || n == 0 {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
t := time.NewTicker(200 * time.Millisecond)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-sh.outStop:
|
||||
read()
|
||||
if len(rest) > 0 {
|
||||
s.addLine(sh, name, rest)
|
||||
}
|
||||
return
|
||||
case <-t.C:
|
||||
read()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Supervisor) addLine(sh *Shard, name string, b []byte) {
|
||||
// The last state of a line rewritten with \r, as a terminal would show.
|
||||
if i := bytes.LastIndexByte(b, '\r'); i >= 0 {
|
||||
b = b[i+1:]
|
||||
}
|
||||
line := string(b)
|
||||
if name != "" {
|
||||
// Prefixed so the two processes of a Ray shard stay distinguishable
|
||||
// in a single log tail.
|
||||
line = "[" + name + "] " + line
|
||||
}
|
||||
sh.logs.add(line)
|
||||
s.log.Debug("engine", "shard", sh.ID, "line", line)
|
||||
}
|
||||
|
||||
// removeOutput drops a shard's files once nothing writes to them.
|
||||
func (s *Supervisor) removeOutput(shardID string) {
|
||||
matches, _ := filepath.Glob(filepath.Join(s.cfg.WorkDir, "shard-"+shardID+".*log"))
|
||||
for _, m := range matches {
|
||||
_ = os.Remove(m)
|
||||
}
|
||||
_ = os.Remove(s.outputPath(shardID, ""))
|
||||
}
|
||||
|
||||
// stopOutput has every follower read what is left and stop. Called once
|
||||
// the process has ended, before the shard is done, so what it wrote last
|
||||
// is in the log by the time anyone reads the log for why it ended.
|
||||
func (s *Supervisor) stopOutput(sh *Shard) {
|
||||
sh.outOnce.Do(func() { close(sh.outStop) })
|
||||
sh.outDone.Wait()
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package supervisor
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
@@ -90,3 +91,25 @@ func credentialFor(u *user.User, workDir string) (*runAs, error) {
|
||||
func (r *runAs) env() []string {
|
||||
return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
|
||||
}
|
||||
|
||||
// AsEngineUser has cmd run as the engine user, as an engine would, with its
|
||||
// home under workDir: for work the agent does that is not an engine but
|
||||
// reads what an engine reads — training an adapter. It answers that home
|
||||
// and a way to hand a path to the user, or why it runs as this process.
|
||||
func AsEngineUser(cmd *exec.Cmd, name, workDir string) (home string, give func(path string) error, why string) {
|
||||
r, why := resolveRunAs(name, workDir)
|
||||
if r == nil {
|
||||
return "", func(string) error { return nil }, why
|
||||
}
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Credential: r.cred, Setpgid: true}
|
||||
cmd.Env = append(cmd.Env, r.env()...)
|
||||
uid, gid := int(r.cred.Uid), int(r.cred.Gid)
|
||||
return r.home, func(path string) error {
|
||||
return filepath.WalkDir(path, func(p string, _ os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Lchown(p, uid, gid)
|
||||
})
|
||||
}, ""
|
||||
}
|
||||
@@ -1,12 +1,10 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"os"
|
||||
@@ -104,6 +102,12 @@ type Shard struct {
|
||||
ID string
|
||||
Spec engines.ShardSpec
|
||||
|
||||
// outStop ends the followers of the processes' output files, and
|
||||
// outDone is their having read to the end.
|
||||
outStop chan struct{}
|
||||
outOnce sync.Once
|
||||
outDone sync.WaitGroup
|
||||
|
||||
mu sync.Mutex
|
||||
state State
|
||||
pid int
|
||||
@@ -248,6 +252,7 @@ func (s *Supervisor) Start(ctx context.Context, id string, spec engines.ShardSpe
|
||||
logs: newRingBuffer(s.cfg.LogLines),
|
||||
started: time.Now(),
|
||||
done: make(chan struct{}),
|
||||
outStop: make(chan struct{}),
|
||||
}
|
||||
s.shards[id] = sh
|
||||
s.mu.Unlock()
|
||||
@@ -315,14 +320,15 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
// CommandContext kills only the leader, so replace that with a group kill.
|
||||
c.Cancel = func() error { return killGroup(c.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := c.StdoutPipe()
|
||||
out, err := s.openOutput(sh.ID, "")
|
||||
if err != nil {
|
||||
cancel()
|
||||
return err
|
||||
}
|
||||
c.Stderr = c.Stdout.(io.Writer) // interleave, as the engine writes them
|
||||
|
||||
if err := c.Start(); err != nil {
|
||||
c.Stdout, c.Stderr = out, out // interleaved, as the engine writes them
|
||||
err = c.Start()
|
||||
out.Close() // the process holds its own
|
||||
if err != nil {
|
||||
cancel()
|
||||
return fmt.Errorf("starting %s: %w", cmd.Path, err)
|
||||
}
|
||||
@@ -339,7 +345,8 @@ func (s *Supervisor) launch(ctx context.Context, sh *Shard, cmds []engines.Comma
|
||||
"shard", sh.ID, "err", err)
|
||||
}
|
||||
|
||||
go s.pump(sh, stdout, cmd.Name)
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, ""), cmd.Name)
|
||||
go s.wait(sh)
|
||||
return nil
|
||||
}
|
||||
@@ -353,16 +360,18 @@ func (s *Supervisor) spawn(ctx context.Context, sh *Shard, c engines.Command) (*
|
||||
s.dropTo(proc)
|
||||
proc.Cancel = func() error { return killGroup(proc.Process.Pid, syscall.SIGKILL) }
|
||||
|
||||
stdout, err := proc.StdoutPipe()
|
||||
out, err := s.openOutput(sh.ID, c.Name)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
proc.Stderr = proc.Stdout.(io.Writer)
|
||||
|
||||
if err := proc.Start(); err != nil {
|
||||
proc.Stdout, proc.Stderr = out, out
|
||||
err = proc.Start()
|
||||
out.Close()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("starting %s: %w", c.Path, err)
|
||||
}
|
||||
go s.pump(sh, stdout, c.Name)
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, c.Name), c.Name)
|
||||
// Reaped so it does not become a zombie; its exit does not end the shard.
|
||||
go func() { _ = proc.Wait() }()
|
||||
return proc, nil
|
||||
@@ -395,24 +404,6 @@ func waitForPort(ctx context.Context, addr string, timeout time.Duration) (err e
|
||||
}
|
||||
|
||||
// pump copies a process's output into the ring buffer and the log.
|
||||
func (s *Supervisor) pump(sh *Shard, r io.Reader, name string) {
|
||||
sc := bufio.NewScanner(r)
|
||||
// Engine tracebacks and progress bars produce long lines; the default
|
||||
// 64 KiB limit truncates them exactly when they matter.
|
||||
sc.Buffer(make([]byte, 0, 64<<10), 1<<20)
|
||||
|
||||
for sc.Scan() {
|
||||
line := sc.Text()
|
||||
if name != "" {
|
||||
// Prefixed so the two processes of a Ray shard stay distinguishable
|
||||
// in a single log tail.
|
||||
line = "[" + name + "] " + line
|
||||
}
|
||||
sh.logs.add(line)
|
||||
s.log.Debug("engine", "shard", sh.ID, "line", line)
|
||||
}
|
||||
}
|
||||
|
||||
// wait reaps the process and records why it ended.
|
||||
func (s *Supervisor) wait(sh *Shard) {
|
||||
defer close(sh.done)
|
||||
@@ -440,6 +431,7 @@ func (s *Supervisor) wait(sh *Shard) {
|
||||
final, code := sh.state, sh.exitCode
|
||||
sh.mu.Unlock()
|
||||
|
||||
s.stopOutput(sh)
|
||||
s.removeState(sh)
|
||||
if final == Failed {
|
||||
s.log.Error("shard failed", "shard", sh.ID, "exit_code", code, "err", err,
|
||||
@@ -760,6 +752,7 @@ func (s *Supervisor) writeState(sh *Shard) error {
|
||||
|
||||
func (s *Supervisor) removeState(sh *Shard) {
|
||||
_ = os.Remove(s.statePath(sh.ID))
|
||||
s.removeOutput(sh.ID)
|
||||
}
|
||||
|
||||
// Orphan is an engine process left behind by a previous agent.
|
||||
@@ -838,15 +831,18 @@ func (s *Supervisor) Adopt(o Orphan) (*Shard, error) {
|
||||
logs: newRingBuffer(s.cfg.LogLines),
|
||||
started: o.StartedAt,
|
||||
done: make(chan struct{}),
|
||||
outStop: make(chan struct{}),
|
||||
adopted: true,
|
||||
cmdline: o.Cmdline,
|
||||
}
|
||||
s.shards[o.ShardID] = sh
|
||||
s.mu.Unlock()
|
||||
|
||||
// Output is gone with the previous agent; say so rather than presenting an
|
||||
// empty log as if the engine had been silent.
|
||||
sh.logs.add("[adopted from a previous agent; earlier output is not available]")
|
||||
// Its output file is read from the start: what it wrote under the
|
||||
// previous agent is there, and what it writes now goes on arriving.
|
||||
sh.logs.add("[adopted from a previous agent]")
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, s.outputPath(sh.ID, ""), "")
|
||||
|
||||
go s.poll(sh)
|
||||
return sh, nil
|
||||
@@ -874,6 +870,7 @@ func (s *Supervisor) poll(sh *Shard) {
|
||||
}
|
||||
sh.mu.Unlock()
|
||||
|
||||
s.stopOutput(sh)
|
||||
s.removeState(sh)
|
||||
s.log.Info("adopted shard exited", "shard", sh.ID, "expected", expected)
|
||||
return
|
||||
@@ -900,6 +897,7 @@ func (s *Supervisor) KillOrphan(o Orphan) error {
|
||||
}
|
||||
// An already-missing state file is success: the previous agent may have
|
||||
// reaped the process itself between the scan and this call.
|
||||
s.removeOutput(o.ShardID)
|
||||
if err := os.Remove(s.statePath(o.ShardID)); err != nil && !errors.Is(err, os.ErrNotExist) {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package supervisor
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -905,3 +906,78 @@ func TestAnAdoptedShardsComputeIsKeptWithoutCrashing(t *testing.T) {
|
||||
t.Fatalf("lost: %+v", again)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdoptedEngineGoesOnWritingAndItsEarlierOutputIsKept(t *testing.T) {
|
||||
// An engine's output went to a pipe of the agent's. The agent that
|
||||
// restarted and adopted the engine left that pipe with no reader, and
|
||||
// the engine's next write past the pipe's buffer failed with EPIPE —
|
||||
// which, in a Python engine, raised inside whatever was being reported
|
||||
// on and failed the request. Output goes to a file now: a reader is not
|
||||
// something a file can lose, and the lines from before the restart are
|
||||
// there for the agent after it.
|
||||
s, _ := newSupervisor(t)
|
||||
// 256 KiB at once outgrows a pipe's buffer; a reader gone means EPIPE.
|
||||
sh, err := s.Start(t.Context(), "a", engines.ShardSpec{ReplicaID: "r"},
|
||||
shellCommand(`echo before-restart; sleep 1; head -c 262144 /dev/zero | tr '\0' x; echo; echo after-restart-done; sleep 30`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, 3*time.Second, "the first line was never captured", func() bool {
|
||||
return strings.Contains(strings.Join(sh.Logs(10), "\n"), "before-restart")
|
||||
})
|
||||
// The agent goes away without stopping the engine: nothing reads the
|
||||
// output between here and the adoption.
|
||||
pid := sh.Status().PID
|
||||
sh.outOnce.Do(func() { close(sh.outStop) })
|
||||
sh.outDone.Wait()
|
||||
|
||||
s2, err := New(Config{WorkDir: s.cfg.WorkDir, StopTimeout: 3 * time.Second, LogLines: 64},
|
||||
&scriptedHealth{}, slog.New(slog.DiscardHandler))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
orphans, _ := s2.FindOrphans()
|
||||
adopted, err := s2.Adopt(orphans[0])
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
eventually(t, 5*time.Second, "the engine's output after the restart never arrived: "+strings.Join(adopted.Logs(5), " | "), func() bool {
|
||||
return strings.Contains(strings.Join(adopted.Logs(64), "\n"), "after-restart-done")
|
||||
})
|
||||
if !strings.Contains(strings.Join(adopted.Logs(64), "\n"), "before-restart") {
|
||||
t.Error("the output from before the restart is not in the adopted log")
|
||||
}
|
||||
if !processAlive(pid) {
|
||||
t.Fatal("the engine died writing output nobody was reading")
|
||||
}
|
||||
if err := s2.Stop(t.Context(), "a"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := os.Stat(s2.outputPath("a", "")); !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("the output file was not removed with the shard: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOutputFileIsEmptiedPastItsBound(t *testing.T) {
|
||||
s, _ := newSupervisor(t)
|
||||
path := s.outputPath("a", "")
|
||||
// A file already past the bound, as a long-running engine leaves one.
|
||||
big, _ := os.OpenFile(path, os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
_, _ = big.Write(bytes.Repeat([]byte("x"), maxOutputBytes+1))
|
||||
big.Close()
|
||||
sh := &Shard{ID: "a", logs: newRingBuffer(8), done: make(chan struct{}), outStop: make(chan struct{})}
|
||||
sh.outDone.Add(1)
|
||||
go s.follow(sh, path, "")
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
f, _ := os.OpenFile(path, os.O_WRONLY|os.O_APPEND, 0o644)
|
||||
_, _ = f.WriteString("still here\n")
|
||||
f.Close()
|
||||
s.stopOutput(sh)
|
||||
st, _ := os.Stat(path)
|
||||
if st.Size() > 1024 {
|
||||
t.Errorf("the file was not emptied: %d bytes", st.Size())
|
||||
}
|
||||
if !strings.Contains(strings.Join(sh.Logs(8), "\n"), "still here") {
|
||||
t.Errorf("a line written after the file was emptied was lost: %v", sh.Logs(8))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,574 @@
|
||||
// Package train runs LoRA training jobs on a node (BACKLOG F4): with an
|
||||
// installed engine's Python — its torch and transformers, already matched to
|
||||
// the node's CUDA — and a small overlay of what training needs besides, so a
|
||||
// node that serves can train without a second multi-gigabyte environment.
|
||||
package train
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
_ "embed"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
)
|
||||
|
||||
//go:embed trainer.py
|
||||
var trainerPy []byte
|
||||
|
||||
// overlayPackages are what training needs besides an engine's Python, pinned
|
||||
// and installed without their dependencies: those are the engine's own, and
|
||||
// letting an installer resolve them pulled a second torch.
|
||||
var overlayPackages = []string{"peft==0.21.2", "accelerate==1.15.0"}
|
||||
|
||||
// Job is one training job, as the control plane sent it.
|
||||
type Job struct {
|
||||
ID string
|
||||
BaseRepo string
|
||||
BaseRevision string
|
||||
// Data is NDJSON of {"q", "a"}.
|
||||
Data []byte
|
||||
// Params is the trainer's settings as JSON.
|
||||
Params string
|
||||
GPUMemoryMiB int
|
||||
GGUF bool
|
||||
}
|
||||
|
||||
// Status is how a job is going.
|
||||
type Status struct {
|
||||
State string
|
||||
Message string
|
||||
Step int
|
||||
Steps int
|
||||
Loss float64
|
||||
PeakMiB int
|
||||
}
|
||||
|
||||
// Env is where a node keeps what training needs.
|
||||
type Env struct {
|
||||
// EnginesDir is where engines are installed (the agent's engines_dir).
|
||||
EnginesDir string
|
||||
// WorkDir is the trainer's own directory: its overlay, the llama.cpp
|
||||
// conversion code, and each job's files while it runs.
|
||||
WorkDir string
|
||||
// RunAs is the user jobs run as: the engines' user.
|
||||
RunAs string
|
||||
Log *slog.Logger
|
||||
// Python, UV, ConvDir and GPUs replace what would be found, for tests.
|
||||
Python, UV, ConvDir string
|
||||
GPUs func(ctx context.Context) ([]GPU, error)
|
||||
}
|
||||
|
||||
// GPU is one card and how much of it is free.
|
||||
type GPU struct {
|
||||
Index int
|
||||
FreeMiB int
|
||||
}
|
||||
|
||||
// Runner runs one job at a time.
|
||||
type Runner struct {
|
||||
env Env
|
||||
mu sync.Mutex
|
||||
cancel context.CancelFunc
|
||||
job string
|
||||
}
|
||||
|
||||
// New is a runner.
|
||||
func New(env Env) *Runner {
|
||||
if env.Log == nil {
|
||||
env.Log = slog.New(slog.DiscardHandler)
|
||||
}
|
||||
return &Runner{env: env}
|
||||
}
|
||||
|
||||
// Cancel stops a job if it is the one running.
|
||||
func (r *Runner) Cancel(id string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.job == id && r.cancel != nil {
|
||||
r.cancel()
|
||||
}
|
||||
}
|
||||
|
||||
// ErrBusy is a node already training.
|
||||
var ErrBusy = errors.New("this node is already training")
|
||||
|
||||
// Run runs a job to its end, telling status as it goes and artifact each
|
||||
// piece of what it made. It answers the error it failed with.
|
||||
func (r *Runner) Run(ctx context.Context, job Job, status func(Status), artifact func(kind string, seq int, data []byte, last bool) error) error {
|
||||
r.mu.Lock()
|
||||
if r.job != "" {
|
||||
r.mu.Unlock()
|
||||
return ErrBusy
|
||||
}
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
r.job, r.cancel = job.ID, cancel
|
||||
r.mu.Unlock()
|
||||
defer func() {
|
||||
cancel()
|
||||
r.mu.Lock()
|
||||
r.job, r.cancel = "", nil
|
||||
r.mu.Unlock()
|
||||
}()
|
||||
|
||||
status(Status{State: "preparing", Message: "preparing the trainer"})
|
||||
py, err := r.python()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
overlay, err := r.overlay(ctx, py)
|
||||
if err != nil {
|
||||
return fmt.Errorf("installing %s: %w", strings.Join(overlayPackages, ", "), err)
|
||||
}
|
||||
conv := ""
|
||||
if job.GGUF {
|
||||
if conv, err = r.conversion(ctx); err != nil {
|
||||
return fmt.Errorf("fetching llama.cpp's conversion code: %w", err)
|
||||
}
|
||||
}
|
||||
gpu, err := r.pickGPU(ctx, job.GPUMemoryMiB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
dir := filepath.Join(r.env.WorkDir, "jobs", job.ID)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
out := filepath.Join(dir, "adapter")
|
||||
var params map[string]any
|
||||
if job.Params != "" {
|
||||
if err := json.Unmarshal([]byte(job.Params), ¶ms); err != nil {
|
||||
return fmt.Errorf("the job's params: %w", err)
|
||||
}
|
||||
}
|
||||
spec, _ := json.Marshal(map[string]any{"base_repo": job.BaseRepo, "base_revision": job.BaseRevision,
|
||||
"data": filepath.Join(dir, "data.jsonl"), "out": out, "params": params})
|
||||
for name, b := range map[string][]byte{"data.jsonl": job.Data, "job.json": spec, "trainer.py": trainerPy} {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), b, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
var baseDir string
|
||||
err = r.python3(ctx, py, dir, []string{"PYTHONPATH=" + overlay, "CUDA_VISIBLE_DEVICES=" + strconv.Itoa(gpu)},
|
||||
[]string{filepath.Join(dir, "trainer.py"), filepath.Join(dir, "job.json")},
|
||||
func(line map[string]any) {
|
||||
s := Status{State: str(line["state"]), Message: str(line["message"]), Step: num(line["step"]),
|
||||
Steps: num(line["steps"]), PeakMiB: num(line["peak_mib"])}
|
||||
if v, ok := line["loss"].(float64); ok {
|
||||
s.Loss = v
|
||||
}
|
||||
if b := str(line["base_dir"]); b != "" {
|
||||
baseDir = b
|
||||
}
|
||||
if s.State == "trained" {
|
||||
s.State = "running"
|
||||
s.Message = "trained; packing the adapter"
|
||||
}
|
||||
status(s)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if job.GGUF {
|
||||
status(Status{State: "running", Message: "converting the adapter to GGUF for llama.cpp"})
|
||||
gguf := filepath.Join(dir, "adapter.gguf")
|
||||
err := r.python3(ctx, py, dir, []string{"PYTHONPATH=" + strings.Join([]string{filepath.Join(conv, "gguf-py"), conv, overlay}, ":")},
|
||||
[]string{filepath.Join(conv, "convert_lora_to_gguf.py"), "--base", baseDir, "--outtype", "f16", "--outfile", gguf, out}, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("converting to GGUF: %w", err)
|
||||
}
|
||||
if err := sendFile(gguf, "gguf", artifact); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
var tgz bytes.Buffer
|
||||
if err := tarGz(&tgz, out); err != nil {
|
||||
return err
|
||||
}
|
||||
return sendBytes(tgz.Bytes(), "peft", artifact)
|
||||
}
|
||||
|
||||
func str(v any) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func num(v any) int {
|
||||
f, _ := v.(float64)
|
||||
return int(f)
|
||||
}
|
||||
|
||||
// python3 runs a Python script as the engines' user, passing each JSON line
|
||||
// it prints to line, and answers the last of what it wrote to stderr when it
|
||||
// fails.
|
||||
func (r *Runner) python3(ctx context.Context, py, dir string, env, args []string, line func(map[string]any)) error {
|
||||
cmd := exec.CommandContext(ctx, py, args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", "TOKENIZERS_PARALLELISM=false",
|
||||
"HF_HUB_DISABLE_PROGRESS_BARS=1", "PYTHONUNBUFFERED=1")
|
||||
cmd.Env = append(cmd.Env, env...)
|
||||
home, give, why := supervisor.AsEngineUser(cmd, r.env.RunAs, r.env.WorkDir)
|
||||
if why != "" {
|
||||
r.env.Log.Warn(why)
|
||||
}
|
||||
if home != "" {
|
||||
cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(home, ".cache", "huggingface"))
|
||||
} else {
|
||||
cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(r.env.WorkDir, "hf"))
|
||||
}
|
||||
if err := give(dir); err != nil {
|
||||
return fmt.Errorf("handing %s to %s: %w", dir, r.env.RunAs, err)
|
||||
}
|
||||
// The whole process group goes when the job is cancelled: torch starts
|
||||
// workers of its own.
|
||||
if cmd.SysProcAttr == nil {
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
}
|
||||
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var stderr tailBuffer
|
||||
cmd.Stderr = &stderr
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
var m map[string]any
|
||||
if line != nil && json.Unmarshal(sc.Bytes(), &m) == nil {
|
||||
line(m)
|
||||
}
|
||||
}
|
||||
err = cmd.Wait()
|
||||
if ctx.Err() != nil {
|
||||
return errors.New("cancelled")
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("%v: %s", err, stderr.last())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// tailBuffer keeps the end of what a process wrote: the reason it failed is
|
||||
// at the end, and the rest is progress bars.
|
||||
type tailBuffer struct{ b []byte }
|
||||
|
||||
func (t *tailBuffer) Write(p []byte) (int, error) {
|
||||
t.b = append(t.b, p...)
|
||||
if len(t.b) > 16<<10 {
|
||||
t.b = t.b[len(t.b)-16<<10:]
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (t *tailBuffer) last() string {
|
||||
lines := strings.Split(strings.TrimSpace(string(t.b)), "\n")
|
||||
if len(lines) > 3 {
|
||||
lines = lines[len(lines)-3:]
|
||||
}
|
||||
return strings.Join(lines, " | ")
|
||||
}
|
||||
|
||||
// python is the newest installed vLLM's interpreter: its torch is built for
|
||||
// this node's CUDA.
|
||||
func (r *Runner) python() (string, error) {
|
||||
if r.env.Python != "" {
|
||||
return r.env.Python, nil
|
||||
}
|
||||
found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "vllm", "*", "bin", "python3"))
|
||||
if len(found) == 0 {
|
||||
return "", errors.New("training needs vLLM installed on this node (its torch is what trains); " +
|
||||
"llmgw ctl engine-build, or serving any vLLM model here, installs it")
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found[len(found)-1], nil
|
||||
}
|
||||
|
||||
func (r *Runner) uv() (string, error) {
|
||||
if r.env.UV != "" {
|
||||
return r.env.UV, nil
|
||||
}
|
||||
var found []string
|
||||
_ = filepath.WalkDir(filepath.Join(r.env.EnginesDir, "uv"), func(p string, d os.DirEntry, err error) error {
|
||||
if err == nil && !d.IsDir() && d.Name() == "uv" {
|
||||
found = append(found, p)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if len(found) == 0 {
|
||||
return "", errors.New("no uv under " + filepath.Join(r.env.EnginesDir, "uv"))
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found[len(found)-1], nil
|
||||
}
|
||||
|
||||
// overlay installs the pinned packages once per Python and set of pins.
|
||||
func (r *Runner) overlay(ctx context.Context, py string) (string, error) {
|
||||
sum := sha256.Sum256([]byte(py + "\x00" + strings.Join(overlayPackages, " ")))
|
||||
dir := filepath.Join(r.env.WorkDir, "overlay-"+hex.EncodeToString(sum[:6]))
|
||||
if _, err := os.Stat(filepath.Join(dir, ".done")); err == nil {
|
||||
return dir, nil
|
||||
}
|
||||
uv, err := r.uv()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
tmp := dir + ".new"
|
||||
_ = os.RemoveAll(tmp)
|
||||
args := append([]string{"pip", "install", "--quiet", "--python", py, "--no-deps", "--target", tmp}, overlayPackages...)
|
||||
cmd := exec.CommandContext(ctx, uv, args...)
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
return "", fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(tmp, ".done"), nil, 0o644)
|
||||
_ = os.RemoveAll(dir)
|
||||
if err := os.Rename(tmp, dir); err != nil {
|
||||
return "", err
|
||||
}
|
||||
// Readable by the engines' user, which runs the trainer.
|
||||
_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
|
||||
if err == nil {
|
||||
mode := os.FileMode(0o644)
|
||||
if d.IsDir() {
|
||||
mode = 0o755
|
||||
}
|
||||
_ = os.Chmod(p, mode)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return dir, nil
|
||||
}
|
||||
|
||||
// llamaCppTag is the release of the newest llama.cpp installed: its
|
||||
// conversion code is the one whose GGUF that llama.cpp reads.
|
||||
func (r *Runner) llamaCppTag() (string, error) {
|
||||
found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "llamacpp", "b*"))
|
||||
if len(found) == 0 {
|
||||
return "", errors.New("a GGUF adapter needs llama.cpp installed on this node, to match its release")
|
||||
}
|
||||
sort.Strings(found)
|
||||
tag, _, _ := strings.Cut(filepath.Base(found[len(found)-1]), "-")
|
||||
return tag, nil
|
||||
}
|
||||
|
||||
// conversionSource is where llama.cpp's source is fetched from, by tag.
|
||||
var conversionSource = "https://codeload.github.com/ggml-org/llama.cpp/tar.gz/refs/tags/"
|
||||
|
||||
// conversion fetches llama.cpp's LoRA conversion code for the installed
|
||||
// release, once.
|
||||
func (r *Runner) conversion(ctx context.Context) (string, error) {
|
||||
if r.env.ConvDir != "" {
|
||||
return r.env.ConvDir, nil
|
||||
}
|
||||
tag, err := r.llamaCppTag()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
dir := filepath.Join(r.env.WorkDir, "llama.cpp-"+tag)
|
||||
if _, err := os.Stat(filepath.Join(dir, "convert_lora_to_gguf.py")); err == nil {
|
||||
return dir, nil
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, conversionSource+tag, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
res, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer res.Body.Close()
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return "", fmt.Errorf("%s answered %s", conversionSource+tag, res.Status)
|
||||
}
|
||||
tmp := dir + ".new"
|
||||
_ = os.RemoveAll(tmp)
|
||||
if err := untarConversion(res.Body, tmp); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_ = os.RemoveAll(dir)
|
||||
return dir, os.Rename(tmp, dir)
|
||||
}
|
||||
|
||||
// untarConversion keeps, of llama.cpp's source, the conversion scripts and
|
||||
// the Python packages they import.
|
||||
func untarConversion(r io.Reader, dir string) error {
|
||||
gz, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
kept := 0
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, rel, ok := strings.Cut(h.Name, "/")
|
||||
if !ok || h.Typeflag != tar.TypeReg || strings.Contains(rel, "..") {
|
||||
continue
|
||||
}
|
||||
if rel != "convert_lora_to_gguf.py" && rel != "convert_hf_to_gguf.py" &&
|
||||
!strings.HasPrefix(rel, "conversion/") && !strings.HasPrefix(rel, "gguf-py/") {
|
||||
continue
|
||||
}
|
||||
p := filepath.Join(dir, filepath.FromSlash(rel))
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(p, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(f, io.LimitReader(tr, 64<<20))
|
||||
f.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
kept++
|
||||
}
|
||||
if kept == 0 {
|
||||
return errors.New("no conversion code in llama.cpp's source")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// pickGPU is the card with the most memory free, if that is enough.
|
||||
func (r *Runner) pickGPU(ctx context.Context, need int) (int, error) {
|
||||
list := r.env.GPUs
|
||||
if list == nil {
|
||||
list = nvidiaSMI
|
||||
}
|
||||
gpus, err := list(ctx)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("reading the cards' free memory: %w", err)
|
||||
}
|
||||
if len(gpus) == 0 {
|
||||
return 0, errors.New("this node has no GPU to train on")
|
||||
}
|
||||
best := gpus[0]
|
||||
for _, g := range gpus[1:] {
|
||||
if g.FreeMiB > best.FreeMiB {
|
||||
best = g
|
||||
}
|
||||
}
|
||||
if best.FreeMiB < need {
|
||||
return 0, fmt.Errorf("no card here has %d MiB free (the most free is GPU %d with %d MiB); "+
|
||||
"a smaller base, or a card freed for it, would do", need, best.Index, best.FreeMiB)
|
||||
}
|
||||
return best.Index, nil
|
||||
}
|
||||
|
||||
func nvidiaSMI(ctx context.Context) ([]GPU, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "nvidia-smi", "--query-gpu=index,memory.free", "--format=csv,noheader,nounits").Output()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return parseGPUs(string(out)), nil
|
||||
}
|
||||
|
||||
func parseGPUs(s string) []GPU {
|
||||
var out []GPU
|
||||
for _, line := range strings.Split(strings.TrimSpace(s), "\n") {
|
||||
a, b, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
i, err1 := strconv.Atoi(strings.TrimSpace(a))
|
||||
f, err2 := strconv.Atoi(strings.TrimSpace(b))
|
||||
if err1 == nil && err2 == nil {
|
||||
out = append(out, GPU{Index: i, FreeMiB: f})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// tarGz packs a directory.
|
||||
func tarGz(w io.Writer, dir string) error {
|
||||
gz := gzip.NewWriter(w)
|
||||
tw := tar.NewWriter(gz)
|
||||
err := filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
rel, _ := filepath.Rel(dir, p)
|
||||
info, err := d.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.WriteHeader(&tar.Header{Name: filepath.ToSlash(rel), Mode: 0o644, Size: info.Size(), ModTime: info.ModTime()}); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.Open(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
_, err = io.Copy(tw, f)
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
return gz.Close()
|
||||
}
|
||||
|
||||
// artifactChunk is how much of an artifact goes in one message.
|
||||
const artifactChunk = 1 << 20
|
||||
|
||||
func sendFile(path, kind string, send func(string, int, []byte, bool) error) error {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return sendBytes(b, kind, send)
|
||||
}
|
||||
|
||||
func sendBytes(b []byte, kind string, send func(string, int, []byte, bool) error) error {
|
||||
seq := 0
|
||||
for {
|
||||
n := min(artifactChunk, len(b))
|
||||
last := n == len(b)
|
||||
if err := send(kind, seq, b[:n], last); err != nil {
|
||||
return err
|
||||
}
|
||||
if last {
|
||||
return nil
|
||||
}
|
||||
b = b[n:]
|
||||
seq++
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
package train
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakePython stands in for an engine's interpreter: as the trainer it
|
||||
// prints progress and writes an adapter; as the converter it writes a GGUF;
|
||||
// told to fail, it fails.
|
||||
const fakePython = `#!/bin/sh
|
||||
case "$1" in
|
||||
*trainer.py)
|
||||
out=$(sed -n 's/.*"out": *"\([^"]*\)".*/\1/p' "$2")
|
||||
grep -q FAIL "$(dirname "$2")/data.jsonl" && { echo "CUDA out of memory" >&2; exit 1; }
|
||||
echo '{"state":"preparing","message":"fetching base"}'
|
||||
echo '{"state":"running","step":5,"steps":10,"loss":0.9,"peak_mib":1900}'
|
||||
mkdir -p "$out"; echo '{"r":16}' > "$out/adapter_config.json"; echo weights > "$out/adapter_model.safetensors"
|
||||
echo '{"state":"trained","step":10,"steps":10,"peak_mib":2100,"base_dir":"/base"}'
|
||||
;;
|
||||
*convert_lora_to_gguf.py)
|
||||
while [ $# -gt 0 ]; do [ "$1" = "--outfile" ] && echo GGUF > "$2"; shift; done
|
||||
;;
|
||||
esac
|
||||
`
|
||||
|
||||
func runner(t *testing.T, free int) *Runner {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
py := filepath.Join(dir, "python3")
|
||||
_ = os.WriteFile(py, []byte(fakePython), 0o755)
|
||||
uv := filepath.Join(dir, "uv")
|
||||
_ = os.WriteFile(uv, []byte("#!/bin/sh\nwhile [ $# -gt 0 ]; do [ \"$1\" = \"--target\" ] && mkdir -p \"$2\"; shift; done\n"), 0o755)
|
||||
return New(Env{WorkDir: filepath.Join(dir, "work"), Python: py, UV: uv, ConvDir: dir,
|
||||
GPUs: func(context.Context) ([]GPU, error) { return []GPU{{0, 500}, {1, free}}, nil }})
|
||||
}
|
||||
|
||||
type sink struct {
|
||||
mu sync.Mutex
|
||||
statuses []Status
|
||||
parts map[string][]byte
|
||||
lasts int
|
||||
}
|
||||
|
||||
func (s *sink) status(st Status) { s.mu.Lock(); s.statuses = append(s.statuses, st); s.mu.Unlock() }
|
||||
func (s *sink) artifact(kind string, _ int, b []byte, last bool) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.parts == nil {
|
||||
s.parts = map[string][]byte{}
|
||||
}
|
||||
s.parts[kind] = append(s.parts[kind], b...)
|
||||
if last {
|
||||
s.lasts++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAJobTrainsOnTheFreestCardAndSendsWhatItMade(t *testing.T) {
|
||||
r := runner(t, 3000)
|
||||
s := &sink{}
|
||||
err := r.Run(context.Background(), Job{ID: "j1", BaseRepo: "Qwen/x", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500, GGUF: true},
|
||||
s.status, s.artifact)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var running bool
|
||||
for _, st := range s.statuses {
|
||||
if st.State == "running" && st.Step == 5 && st.Steps == 10 && st.PeakMiB == 1900 {
|
||||
running = true
|
||||
}
|
||||
}
|
||||
if !running {
|
||||
t.Errorf("statuses %+v", s.statuses)
|
||||
}
|
||||
if string(s.parts["gguf"]) != "GGUF\n" || s.lasts != 2 {
|
||||
t.Errorf("gguf %q, lasts %d", s.parts["gguf"], s.lasts)
|
||||
}
|
||||
files := map[string]string{}
|
||||
gz, err := gzip.NewReader(bytes.NewReader(s.parts["peft"]))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
b, _ := io.ReadAll(tr)
|
||||
files[h.Name] = string(b)
|
||||
}
|
||||
if files["adapter_model.safetensors"] != "weights\n" || files["adapter_config.json"] == "" {
|
||||
t.Errorf("peft archive %v", files)
|
||||
}
|
||||
// Nothing of the job is left on the node.
|
||||
if entries, _ := os.ReadDir(filepath.Join(r.env.WorkDir, "jobs")); len(entries) != 0 {
|
||||
t.Errorf("left behind: %v", entries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobWaitsForNoCardItDoesNotFit(t *testing.T) {
|
||||
r := runner(t, 1000)
|
||||
err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500}, (&sink{}).status, (&sink{}).artifact)
|
||||
if err == nil || !strings.Contains(err.Error(), "no card here has 2500 MiB free") {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestATrainerThatFailsSaysWhy(t *testing.T) {
|
||||
r := runner(t, 3000)
|
||||
err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"FAIL","a":"b"}`)}, (&sink{}).status, (&sink{}).artifact)
|
||||
if err == nil || !strings.Contains(err.Error(), "CUDA out of memory") {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOneJobAtATime(t *testing.T) {
|
||||
r := runner(t, 3000)
|
||||
r.job = "other"
|
||||
if err := r.Run(context.Background(), Job{ID: "j2"}, (&sink{}).status, (&sink{}).artifact); err != ErrBusy {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCardsAreReadFromNvidiaSMI(t *testing.T) {
|
||||
got := parseGPUs("0, 2310\n1, 80000\nbad line\n")
|
||||
if len(got) != 2 || got[1].FreeMiB != 80000 || got[0].Index != 0 {
|
||||
t.Errorf("parsed %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheConversionCodeIsKeptOfLlamaCpp(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
gz := gzip.NewWriter(&buf)
|
||||
tw := tar.NewWriter(gz)
|
||||
for _, name := range []string{"llama.cpp-b1/convert_lora_to_gguf.py", "llama.cpp-b1/conversion/qwen.py",
|
||||
"llama.cpp-b1/gguf-py/gguf/__init__.py", "llama.cpp-b1/src/llama.cpp", "llama.cpp-b1/../evil"} {
|
||||
_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
|
||||
_, _ = tw.Write([]byte("x"))
|
||||
}
|
||||
_ = tw.Close()
|
||||
_ = gz.Close()
|
||||
dir := filepath.Join(t.TempDir(), "conv")
|
||||
if err := untarConversion(&buf, dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{"convert_lora_to_gguf.py", "conversion/qwen.py", "gguf-py/gguf/__init__.py"} {
|
||||
if _, err := os.Stat(filepath.Join(dir, want)); err != nil {
|
||||
t.Errorf("%s not kept", want)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "src")); err == nil {
|
||||
t.Error("the rest of llama.cpp was kept")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
# llmgw's LoRA trainer (BACKLOG F4), run by the node agent with an installed
|
||||
# engine's Python (its torch and transformers) and a small overlay (peft,
|
||||
# accelerate). Reads a job, trains, writes the PEFT adapter, and says how it
|
||||
# goes as one JSON object a line on standard output.
|
||||
import json, math, os, random, sys, threading, subprocess, time
|
||||
|
||||
def say(**kw):
|
||||
print(json.dumps(kw), flush=True)
|
||||
|
||||
job = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||
p = {"rank": 16, "alpha": 32, "dropout": 0.05, "epochs": 6, "lr": 2e-4, "batch": 4, "max_len": 512, "seed": 0}
|
||||
p.update(job.get("params") or {})
|
||||
|
||||
say(state="preparing", message="loading libraries")
|
||||
import torch
|
||||
from huggingface_hub import snapshot_download
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
random.seed(p["seed"]); torch.manual_seed(p["seed"])
|
||||
if not torch.cuda.is_available():
|
||||
raise SystemExit("no CUDA device is visible to the trainer")
|
||||
|
||||
say(state="preparing", message="fetching " + job["base_repo"])
|
||||
base_dir = snapshot_download(job["base_repo"], revision=job.get("base_revision") or None,
|
||||
allow_patterns=["*.json", "*.safetensors", "*.txt", "*.model", "*.tiktoken"])
|
||||
say(state="preparing", message="loading the base model", base_dir=base_dir)
|
||||
|
||||
# bf16 where the card has it; fp16 with a loss scaler where it does not
|
||||
# (Turing and older).
|
||||
bf16 = torch.cuda.is_bf16_supported()
|
||||
dtype = torch.bfloat16 if bf16 else torch.float16
|
||||
tok = AutoTokenizer.from_pretrained(base_dir)
|
||||
if tok.pad_token_id is None:
|
||||
tok.pad_token = tok.eos_token
|
||||
model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype).to("cuda")
|
||||
|
||||
rows = [json.loads(l) for l in open(job["data"], encoding="utf-8") if l.strip()]
|
||||
rows = [r for r in rows if r.get("q") and r.get("a")]
|
||||
if not rows:
|
||||
raise SystemExit("no {q, a} pairs in the data")
|
||||
|
||||
def encode(r):
|
||||
prompt = tok.apply_chat_template([{"role": "user", "content": r["q"]}], add_generation_prompt=True, tokenize=False)
|
||||
full = tok.apply_chat_template([{"role": "user", "content": r["q"]}, {"role": "assistant", "content": r["a"]}], tokenize=False)
|
||||
pi = tok(prompt, add_special_tokens=False)["input_ids"]
|
||||
fi = tok(full, add_special_tokens=False)["input_ids"][: p["max_len"]]
|
||||
return fi, ([-100] * len(pi) + fi[len(pi):])[: len(fi)]
|
||||
|
||||
data = [encode(r) for r in rows]
|
||||
|
||||
model.gradient_checkpointing_enable()
|
||||
model.enable_input_require_grads()
|
||||
model = get_peft_model(model, LoraConfig(r=int(p["rank"]), lora_alpha=int(p["alpha"]), lora_dropout=float(p["dropout"]),
|
||||
task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
|
||||
trainable = [x for x in model.parameters() if x.requires_grad]
|
||||
for x in trainable: # fp32 trainable weights, so a loss scaler can unscale them
|
||||
x.data = x.data.float()
|
||||
|
||||
peak = {"mib": 0}
|
||||
stop = threading.Event()
|
||||
def watch():
|
||||
while not stop.is_set():
|
||||
try:
|
||||
out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
for line in out.splitlines():
|
||||
pid, mib = [v.strip() for v in line.split(",")]
|
||||
if int(pid) == os.getpid():
|
||||
peak["mib"] = max(peak["mib"], int(mib))
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
threading.Thread(target=watch, daemon=True).start()
|
||||
|
||||
opt = torch.optim.AdamW(trainable, lr=float(p["lr"]), weight_decay=0.0)
|
||||
scaler = torch.amp.GradScaler("cuda", enabled=not bf16)
|
||||
bs, epochs = int(p["batch"]), int(p["epochs"])
|
||||
steps = epochs * math.ceil(len(data) / bs)
|
||||
warm = max(1, steps // 10)
|
||||
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, (s + 1) / warm) * max(0.0, 1 - s / steps))
|
||||
model.train(); step = 0; last = time.time()
|
||||
for ep in range(epochs):
|
||||
random.shuffle(data)
|
||||
for i in range(0, len(data), bs):
|
||||
batch = data[i:i + bs]; n = max(len(x[0]) for x in batch)
|
||||
ids = torch.tensor([x[0] + [tok.pad_token_id] * (n - len(x[0])) for x in batch], device="cuda")
|
||||
lab = torch.tensor([x[1] + [-100] * (n - len(x[1])) for x in batch], device="cuda")
|
||||
att = torch.tensor([[1] * len(x[0]) + [0] * (n - len(x[0])) for x in batch], device="cuda")
|
||||
with torch.autocast("cuda", dtype=dtype):
|
||||
loss = model(input_ids=ids, attention_mask=att, labels=lab).loss
|
||||
scaler.scale(loss).backward(); scaler.unscale_(opt)
|
||||
torch.nn.utils.clip_grad_norm_(trainable, 1.0)
|
||||
scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True); sched.step(); step += 1
|
||||
if time.time() - last > 2 or step == steps:
|
||||
last = time.time()
|
||||
say(state="running", message="training", step=step, steps=steps, loss=round(loss.item(), 4), peak_mib=peak["mib"])
|
||||
|
||||
model.save_pretrained(job["out"])
|
||||
stop.set()
|
||||
say(state="trained", step=step, steps=steps, peak_mib=peak["mib"], base_dir=base_dir)
|
||||
@@ -0,0 +1,46 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
)
|
||||
|
||||
// trainJob runs a training job the control plane sent, telling it how the
|
||||
// job goes and sending what it made. A stream that ends ends the job: the
|
||||
// control plane, which sees the node go, marks it failed, and it is started
|
||||
// again from the beginning rather than resumed.
|
||||
func (a *Agent) trainJob(ctx context.Context, m *llmgwv1.TrainStart, out chan<- *llmgwv1.AgentMessage) {
|
||||
id := m.GetJobId()
|
||||
send := func(msg *llmgwv1.AgentMessage) bool {
|
||||
select {
|
||||
case out <- msg:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
}
|
||||
}
|
||||
status := func(s train.Status) {
|
||||
send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainStatus{TrainStatus: &llmgwv1.TrainStatus{
|
||||
JobId: id, State: s.State, Message: s.Message, Step: int32(s.Step), Steps: int32(s.Steps), //nolint:gosec // counts
|
||||
Loss: s.Loss, PeakMib: int32(s.PeakMiB)}}}) //nolint:gosec // MiB
|
||||
}
|
||||
artifact := func(kind string, seq int, data []byte, last bool) error {
|
||||
if !send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainArtifact{TrainArtifact: &llmgwv1.TrainArtifact{
|
||||
JobId: id, Kind: kind, Seq: int32(seq), Data: data, Last: last}}}) { //nolint:gosec // chunk count
|
||||
return ctx.Err()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
a.log.Info("training an adapter", "job", id, "base", m.GetBaseRepo())
|
||||
err := a.trainer.Run(ctx, train.Job{ID: id, BaseRepo: m.GetBaseRepo(), BaseRevision: m.GetBaseRevision(),
|
||||
Data: m.GetData(), Params: m.GetParams(), GPUMemoryMiB: int(m.GetGpuMemoryMib()), GGUF: m.GetGguf()}, status, artifact)
|
||||
if err != nil {
|
||||
a.log.Warn("training failed", "job", id, "err", err)
|
||||
status(train.Status{State: "failed", Message: err.Error()})
|
||||
return
|
||||
}
|
||||
a.log.Info("trained an adapter", "job", id)
|
||||
status(train.Status{State: "done"})
|
||||
}
|
||||
@@ -249,7 +249,14 @@ func pgEnv(dsn string) ([]string, error) {
|
||||
set("PGPASSWORD", pw)
|
||||
}
|
||||
set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
|
||||
set("PGSSLMODE", u.Query().Get("sslmode"))
|
||||
// The TLS settings go with the mode: verify-full without the cluster's
|
||||
// authority sent libpq looking for ~/.postgresql/root.crt, and the dump
|
||||
// failed on every control plane whose store is replicated.
|
||||
q := u.Query()
|
||||
set("PGSSLMODE", q.Get("sslmode"))
|
||||
set("PGSSLROOTCERT", q.Get("sslrootcert"))
|
||||
set("PGSSLCERT", q.Get("sslcert"))
|
||||
set("PGSSLKEY", q.Get("sslkey"))
|
||||
// A standby if there is one, else whichever answers — not the address's
|
||||
// read-write: a dump only reads, it need not load the primary, and one
|
||||
// that insists on the primary fails exactly while the cluster is failing
|
||||
|
||||
@@ -238,3 +238,17 @@ func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
|
||||
t.Error("an empty host in the list was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPgDumpIsGivenTheClusterAuthorityAlongWithVerifyFull(t *testing.T) {
|
||||
// Without it libpq looked for ~/.postgresql/root.crt and the dump failed.
|
||||
env, err := pgEnv("postgres://llmgw:p@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fpki%2Fca.crt&target_session_attrs=read-write")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := strings.Join(env, " ")
|
||||
for _, want := range []string{"PGSSLMODE=verify-full", "PGSSLROOTCERT=/etc/llmgw/pki/ca.crt"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("%s is not in %s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
)
|
||||
|
||||
// comfyKeyEnv carries the replica's key to the gate: the environment, which
|
||||
// only the engine's user and root can read, rather than the command line,
|
||||
// which every user on the node can.
|
||||
const comfyKeyEnv = "LLMGW_ENGINE_KEY"
|
||||
|
||||
func newComfyUIGateCmd() *cobra.Command {
|
||||
var cfg comfy.GateConfig
|
||||
cmd := &cobra.Command{
|
||||
Use: "comfyui-gate [flags] [-- main.py flags]",
|
||||
Short: "Run ComfyUI as an engine: behind a key, on the loopback (started by the agent)",
|
||||
Hidden: true,
|
||||
Long: "Started by the node agent for a replica whose engine is comfyui. ComfyUI asks\n" +
|
||||
"nobody for a key, so it listens on the loopback only and this gate on the\n" +
|
||||
"replica's port, letting through, with the replica's key, only what the router\n" +
|
||||
"asks of it. It also lays the fetched weights out where ComfyUI looks for them.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
cfg.Key = os.Getenv(comfyKeyEnv)
|
||||
_ = os.Unsetenv(comfyKeyEnv)
|
||||
cfg.Args = args
|
||||
cfg.Log = logger()
|
||||
return comfy.RunGate(ctx, cfg)
|
||||
},
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&cfg.Listen, "listen", "", "the replica's address")
|
||||
f.StringVar(&cfg.Model, "model", "", "the model's name, for /v1/models")
|
||||
f.StringVar(&cfg.Weights, "weights", "", "the fetched weights: a file or a directory")
|
||||
f.StringVar(&cfg.State, "state", "", "the replica's own directory")
|
||||
f.StringVar(&cfg.Python, "python", "python3", "the interpreter of ComfyUI's environment")
|
||||
f.StringVar(&cfg.Dir, "comfyui", "", "ComfyUI's directory, where main.py is")
|
||||
for _, n := range []string{"listen", "weights", "state", "comfyui"} {
|
||||
_ = cmd.MarkFlagRequired(n)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
@@ -116,6 +116,11 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
// A stream, as it comes: NDJSON a long run writes line by line.
|
||||
if w, ok := out.(io.Writer); ok {
|
||||
_, err := io.Copy(w, resp.Body)
|
||||
return err
|
||||
}
|
||||
// An older control plane answers a path it does not know with its
|
||||
// dashboard: a page, which read as JSON said only "invalid character '<'".
|
||||
if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
|
||||
@@ -173,6 +178,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newRetryCmd(client),
|
||||
newUpgradeCmd(client),
|
||||
newJobsCmd(client, &asJSON),
|
||||
newMeshJobsCmd(client, &asJSON),
|
||||
newEngineCheckCmd(),
|
||||
newEngineBuildCmd(client),
|
||||
newToolCheckCmd(client),
|
||||
@@ -181,6 +187,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newGuardCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
@@ -325,6 +332,9 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
return "stopped"
|
||||
case m.Ready >= m.Min && m.Min > 0:
|
||||
return "ready"
|
||||
case m.Min == 0 && m.Ready > 0 && m.Ready >= m.Total:
|
||||
// Asked for, and up until it goes unused for long enough.
|
||||
return "ready (on demand)"
|
||||
case m.Total > m.Ready:
|
||||
return "starting"
|
||||
default:
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
)
|
||||
|
||||
// newMeshJobsCmd is the 3D models clients asked for, as jobs: every owner's,
|
||||
// what each is doing, and a way to stop one. (ctl jobs is the control
|
||||
// plane's own work, serving and updating models.)
|
||||
func newMeshJobsCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
cmd := &cobra.Command{
|
||||
Use: "3d",
|
||||
Short: "3D models clients asked for: the jobs, how far along each is, and stopping one",
|
||||
}
|
||||
var status string
|
||||
var limit int
|
||||
list := &cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Every owner's 3D jobs, newest first",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
q := url.Values{}
|
||||
if status != "" {
|
||||
q.Set("status", status)
|
||||
}
|
||||
if limit > 0 {
|
||||
q.Set("limit", fmt.Sprint(limit))
|
||||
}
|
||||
var jobs []adminapi.JobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/client-jobs?"+q.Encode(), &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "no 3D jobs")
|
||||
return nil
|
||||
}
|
||||
w := tabTo(cmd.OutOrStdout())
|
||||
fmt.Fprintln(w, "JOB\tMODEL\tSTATUS\tPROGRESS\tOWNER\tASKED")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s ago\n", j.ID, j.Model, j.Status, progressText(j),
|
||||
j.Owner, time.Since(j.CreatedAt).Round(time.Second))
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
}
|
||||
list.Flags().StringVar(&status, "status", "", "only jobs in this status: queued, running, completed, failed…")
|
||||
list.Flags().IntVar(&limit, "limit", 0, "the most jobs to show (50)")
|
||||
get := &cobra.Command{
|
||||
Use: "get JOB",
|
||||
Short: "One 3D job: its status, progress, result or error",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var j adminapi.JobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/client-jobs/"+url.PathEscape(args[0]), &j); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return emitJSON(j)
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "job %s\nmodel %s\nstatus %s\n", j.ID, j.Model, j.Status)
|
||||
if p := progressText(j); p != "" {
|
||||
fmt.Fprintf(out, "progress %s\n", p)
|
||||
}
|
||||
fmt.Fprintf(out, "owner %s (key %s)\nasked %s\n", j.Owner, j.KeyID, j.CreatedAt.UTC().Format(time.RFC3339))
|
||||
if j.FinishedAt != nil && j.StartedAt != nil {
|
||||
fmt.Fprintf(out, "took %s\n", j.FinishedAt.Sub(*j.StartedAt).Round(time.Second))
|
||||
}
|
||||
if j.OutputFileID != "" {
|
||||
fmt.Fprintf(out, "result %s (%s)\n", j.OutputFileID, j.ContentType)
|
||||
}
|
||||
if j.Error != "" {
|
||||
fmt.Fprintf(out, "error %s\n", j.Error)
|
||||
}
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cancel := &cobra.Command{
|
||||
Use: "cancel JOB",
|
||||
Short: "Stop a 3D job, in the engine's queue too",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var j adminapi.JobView
|
||||
if err := client().post(cmd.Context(), "/api/v1/client-jobs/"+url.PathEscape(args[0])+"/cancel", nil, &j); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s is %s\n", j.ID, j.Status)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
cmd.AddCommand(list, get, cancel)
|
||||
return cmd
|
||||
}
|
||||
|
||||
// progressText is a job's progress as a person reads it.
|
||||
func progressText(j adminapi.JobView) string {
|
||||
if len(j.Progress) == 0 {
|
||||
return ""
|
||||
}
|
||||
var p struct {
|
||||
State string `json:"state"`
|
||||
Ahead int `json:"ahead"`
|
||||
NodeClass string `json:"node_class"`
|
||||
Step int `json:"step"`
|
||||
Steps int `json:"steps"`
|
||||
Percent int `json:"percent"`
|
||||
}
|
||||
if json.Unmarshal(j.Progress, &p) != nil {
|
||||
return ""
|
||||
}
|
||||
switch {
|
||||
case p.State == "queued" && p.Ahead > 0:
|
||||
return fmt.Sprintf("queued, %d ahead", p.Ahead)
|
||||
case p.State == "running" && p.Steps > 0:
|
||||
return fmt.Sprintf("running %d%%, %s step %d/%d", p.Percent, p.NodeClass, p.Step, p.Steps)
|
||||
case p.State == "running" && p.NodeClass != "":
|
||||
return "running " + p.NodeClass
|
||||
}
|
||||
return p.State
|
||||
}
|
||||
@@ -56,6 +56,12 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
fmt.Fprintf(out, "status %d\n", v.Status)
|
||||
fmt.Fprintf(out, "time %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
|
||||
fmt.Fprintf(out, "tokens %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
|
||||
if v.Images > 0 {
|
||||
fmt.Fprintf(out, "images %d\n", v.Images)
|
||||
}
|
||||
if v.Meshes > 0 {
|
||||
fmt.Fprintf(out, "meshes %d\n", v.Meshes)
|
||||
}
|
||||
if v.CacheHit {
|
||||
fmt.Fprintf(out, "cache answered from the project's answer cache: nothing generated, nothing charged\n")
|
||||
}
|
||||
|
||||
+49
-21
@@ -7,12 +7,14 @@ import (
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
)
|
||||
|
||||
// firewallStateFile is where a control plane writes what the cluster's
|
||||
@@ -20,11 +22,9 @@ import (
|
||||
// the server itself runs as llmgw and cannot change the ruleset.
|
||||
const firewallStateFile = "/var/lib/llmgw/firewall.json"
|
||||
|
||||
// firewallState is what is written there.
|
||||
type firewallState struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
}
|
||||
// firewallState is what is written there: the same as the mirror is told
|
||||
// and keeps beside its models.
|
||||
type firewallState = mirror.FirewallState
|
||||
|
||||
// writeFirewallState records the settings' firewall for this machine's sync
|
||||
// unit. Without the state directory — a server not run by systemd — there
|
||||
@@ -46,25 +46,34 @@ func writeFirewallState(cfg config.Server) error {
|
||||
|
||||
// The units that apply it: a path unit watching the file, and the oneshot it
|
||||
// starts.
|
||||
const (
|
||||
firewallSyncPathUnit = "llmgw-firewall-sync.path"
|
||||
firewallSyncUnitFile = `[Unit]
|
||||
Description=llmgw firewall: apply the cluster's settings on this control plane
|
||||
const firewallSyncPathUnit = "llmgw-firewall-sync.path"
|
||||
|
||||
// firewallSyncUnits are the two units for a state file: on a control plane
|
||||
// the one its server writes, on a mirror the one the control plane has the
|
||||
// mirror write.
|
||||
func firewallSyncUnits(state string) (service, pathUnit string) {
|
||||
arg := ""
|
||||
if state != firewallStateFile {
|
||||
arg = " --state " + state
|
||||
}
|
||||
service = `[Unit]
|
||||
Description=llmgw firewall: apply the cluster's settings on this machine
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/llmgw firewall sync
|
||||
ExecStart=/usr/local/bin/llmgw firewall sync` + arg + `
|
||||
`
|
||||
firewallSyncPathFile = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this control plane
|
||||
pathUnit = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this machine
|
||||
|
||||
[Path]
|
||||
PathChanged=` + firewallStateFile + `
|
||||
PathChanged=` + state + `
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
)
|
||||
return service, pathUnit
|
||||
}
|
||||
|
||||
func newFirewallCmd() *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
@@ -76,6 +85,7 @@ func newFirewallCmd() *cobra.Command {
|
||||
"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
|
||||
}
|
||||
var install bool
|
||||
state := firewallStateFile
|
||||
sync := &cobra.Command{
|
||||
Use: "sync",
|
||||
Short: "Apply what the cluster's settings say of the firewall, here (root)",
|
||||
@@ -85,21 +95,21 @@ func newFirewallCmd() *cobra.Command {
|
||||
return errors.New("the firewall is changed as root: run it with sudo")
|
||||
}
|
||||
if install {
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
|
||||
if _, err := os.Stat(firewallStateFile); err != nil {
|
||||
return nil // the server writes it when it starts
|
||||
if _, err := os.Stat(state); err != nil {
|
||||
return nil // written when the settings are first told
|
||||
}
|
||||
}
|
||||
b, err := os.ReadFile(firewallStateFile)
|
||||
b, err := os.ReadFile(state)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var st firewallState
|
||||
if err := json.Unmarshal(b, &st); err != nil {
|
||||
return fmt.Errorf("%s: %w", firewallStateFile, err)
|
||||
return fmt.Errorf("%s: %w", state, err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
@@ -112,14 +122,17 @@ func newFirewallCmd() *cobra.Command {
|
||||
},
|
||||
}
|
||||
sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
|
||||
sync.Flags().StringVar(&state, "state", firewallStateFile,
|
||||
"the file the settings are written to: a control plane's, or "+mirror.FirewallFile+" in a mirror's directory")
|
||||
c.AddCommand(sync)
|
||||
return c
|
||||
}
|
||||
|
||||
// installFirewallSync writes the sync units on a machine and has the path
|
||||
// unit follow the state file.
|
||||
func installFirewallSync(ctx context.Context, r firewall.Runner) error {
|
||||
body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
|
||||
func installFirewallSync(ctx context.Context, r firewall.Runner, state string) error {
|
||||
service, pathUnit := firewallSyncUnits(state)
|
||||
body := service + "\x00" + pathUnit
|
||||
out, err := r.Run(ctx, `set -e
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
@@ -172,5 +185,20 @@ func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (str
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
// The cluster's machines: a node on a network this one has not let in
|
||||
// yet, which the mirror's files are for.
|
||||
if len(st.Toward) > 0 {
|
||||
added, err := firewall.Admit(ctx, r, st.Toward)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(added) > 0 {
|
||||
shown := make([]string, len(added))
|
||||
for i, p := range added {
|
||||
shown[i] = p.String()
|
||||
}
|
||||
return "llmgw's firewall is on here, and now lets in " + strings.Join(shown, ", "), nil
|
||||
}
|
||||
}
|
||||
return "llmgw's firewall is on here", nil
|
||||
}
|
||||
@@ -2,8 +2,11 @@ package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
)
|
||||
|
||||
type scriptLog struct {
|
||||
@@ -49,3 +52,32 @@ func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
|
||||
t.Errorf("on: %q %v", msg, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMachineLetsInTheClusterMachinesItIsTold(t *testing.T) {
|
||||
// The rules hold 10.0.0.0/24; a node on 192.168.7.0/24 is let in, as
|
||||
// the route to it says, and one already in is not asked about.
|
||||
r := &scriptLog{reply: func(s string) string {
|
||||
switch {
|
||||
case strings.Contains(s, "echo loaded"):
|
||||
return "loaded\n"
|
||||
case strings.Contains(s, "cat /etc/llmgw/firewall.nft"):
|
||||
return firewallRulesFor("10.0.0.0/24")
|
||||
case strings.Contains(s, "ip -o route get 192.168.7.20"):
|
||||
return "192.168.7.20 dev eth1 src 192.168.7.1 uid 0\n2: eth1 inet 192.168.7.1/24 brd 192.168.7.255 scope global eth1\n"
|
||||
case strings.Contains(s, "nft -f"):
|
||||
return "changed\n"
|
||||
}
|
||||
return ""
|
||||
}}
|
||||
msg, err := syncFirewall(context.Background(), r, firewallState{Enabled: true, Toward: []string{"10.0.0.147", "192.168.7.20"}})
|
||||
if err != nil || !strings.Contains(msg, "192.168.7.0/24") {
|
||||
t.Fatalf("%q %v", msg, err)
|
||||
}
|
||||
if strings.Contains(strings.Join(r.scripts, "\n"), "route get 10.0.0.147") {
|
||||
t.Error("asked the route to a machine already let in")
|
||||
}
|
||||
}
|
||||
|
||||
func firewallRulesFor(nets string) string {
|
||||
return firewall.Script(firewall.Spec{Ports: []firewall.Port{firewall.MirrorFiles}, Allow: []netip.Prefix{netip.MustParsePrefix(nets)}})
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// firewallStateKey is where, in the store, the firewall's state is kept
|
||||
// for the store's members to read.
|
||||
const firewallStateKey = "firewall"
|
||||
|
||||
// firewallPublisher tells the machines that cannot ask a control plane
|
||||
// what the cluster's settings say of their firewall: the mirror, over its
|
||||
// API, and the store's members, through the store they hold a copy of.
|
||||
// Nodes are told over their agents' streams and control planes by their
|
||||
// own servers. On the leader only, at start, at every change to the
|
||||
// firewall's settings, and every few minutes for nodes come and gone.
|
||||
type firewallPublisher struct {
|
||||
settings func() config.Server
|
||||
st *store.DB
|
||||
nodes func() []string
|
||||
leader func() bool
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
mirror *mirror.Client
|
||||
kick chan struct{}
|
||||
}
|
||||
|
||||
func newFirewallPublisher(settings func() config.Server, st *store.DB, nodes func() []string,
|
||||
leader func() bool, log *slog.Logger) *firewallPublisher {
|
||||
return &firewallPublisher{settings: settings, st: st, nodes: nodes, leader: leader, log: log,
|
||||
kick: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// setMirror has the mirror told too.
|
||||
func (p *firewallPublisher) setMirror(m *mirror.Client) {
|
||||
p.mu.Lock()
|
||||
p.mirror = m
|
||||
p.mu.Unlock()
|
||||
p.Kick()
|
||||
}
|
||||
|
||||
// Kick has the state worked out and told again now.
|
||||
func (p *firewallPublisher) Kick() {
|
||||
select {
|
||||
case p.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func (p *firewallPublisher) run(ctx context.Context, every time.Duration) {
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
p.once(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-p.kick:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// state is what the settings say now, with the cluster's machines.
|
||||
func (p *firewallPublisher) state(ctx context.Context) mirror.FirewallState {
|
||||
cfg := p.settings()
|
||||
st := mirror.FirewallState{Enabled: cfg.Firewall.Enabled, Networks: slices.Clone(cfg.FirewallNetworks)}
|
||||
var toward []string
|
||||
if p.nodes != nil {
|
||||
toward = append(toward, p.nodes()...)
|
||||
}
|
||||
if p.st != nil {
|
||||
if cps, err := p.st.ControlPlanes(ctx); err == nil {
|
||||
for _, c := range cps {
|
||||
toward = append(toward, c.URL)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, t := range toward {
|
||||
if a, ok := hostAddr(t); ok {
|
||||
st.Toward = append(st.Toward, a)
|
||||
}
|
||||
}
|
||||
slices.Sort(st.Toward)
|
||||
st.Toward = slices.Compact(st.Toward)
|
||||
return st
|
||||
}
|
||||
|
||||
// hostAddr is the address in an address, host:port or URL; names are not
|
||||
// looked up here.
|
||||
func hostAddr(s string) (string, bool) {
|
||||
if u, err := url.Parse(s); err == nil && u.Host != "" {
|
||||
s = u.Hostname()
|
||||
} else if h, _, err := net.SplitHostPort(s); err == nil {
|
||||
s = h
|
||||
}
|
||||
a, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
return a.Unmap().String(), true
|
||||
}
|
||||
|
||||
func (p *firewallPublisher) once(ctx context.Context) {
|
||||
if p.leader != nil && !p.leader() {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
st := p.state(ctx)
|
||||
p.mu.Lock()
|
||||
m := p.mirror
|
||||
p.mu.Unlock()
|
||||
if m != nil {
|
||||
changed, err := m.SetFirewall(ctx, st)
|
||||
switch {
|
||||
case err != nil:
|
||||
p.log.Warn("could not tell the mirror what the settings say of its firewall", "err", err)
|
||||
case changed:
|
||||
p.log.Info("the mirror's firewall follows the cluster's settings", "enabled", st.Enabled)
|
||||
}
|
||||
}
|
||||
if p.st != nil {
|
||||
b, _ := json.Marshal(st)
|
||||
changed, err := p.st.PutClusterState(ctx, firewallStateKey, string(b))
|
||||
switch {
|
||||
case err != nil:
|
||||
p.log.Warn("could not record the firewall's state for the store's members", "err", err)
|
||||
case changed:
|
||||
p.log.Info("the store's members' firewall follows the cluster's settings", "enabled", st.Enabled)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestTheLeaderTellsTheMirrorAndTheStoreWhatTheFirewallSettingsSay(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
st, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
if err := st.PutControlPlane(ctx, store.ControlPlane{URL: "https://10.0.0.150:7443", Name: "ctl02"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
srv := httptest.NewServer(mirror.NewAPI(mirror.NewManager(root, slog.New(slog.DiscardHandler)), "http://m:8081").Handler())
|
||||
defer srv.Close()
|
||||
mc, _ := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
|
||||
|
||||
cfg := config.Server{Firewall: config.FirewallSettings{Enabled: false}, FirewallNetworks: []string{"192.168.5.0/24"}}
|
||||
leader := false
|
||||
p := newFirewallPublisher(func() config.Server { return cfg }, st,
|
||||
func() []string { return []string{"10.0.0.147", "", "10.0.0.147", "node-by-name"} },
|
||||
func() bool { return leader }, slog.New(slog.DiscardHandler))
|
||||
p.setMirror(mc)
|
||||
|
||||
p.once(ctx)
|
||||
if _, err := st.ClusterState(ctx, firewallStateKey); err == nil {
|
||||
t.Error("a standby told the store")
|
||||
}
|
||||
leader = true
|
||||
p.once(ctx)
|
||||
var got mirror.FirewallState
|
||||
v, err := st.ClusterState(ctx, firewallStateKey)
|
||||
if err != nil || json.Unmarshal([]byte(v), &got) != nil {
|
||||
t.Fatalf("the store holds %q %v", v, err)
|
||||
}
|
||||
if got.Enabled || got.Networks[0] != "192.168.5.0/24" || len(got.Toward) != 2 ||
|
||||
got.Toward[0] != "10.0.0.147" || got.Toward[1] != "10.0.0.150" {
|
||||
t.Errorf("state %+v", got)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(root, mirror.FirewallFile))
|
||||
if err != nil || string(b) != v {
|
||||
t.Errorf("the mirror holds %s, the store %s", b, v)
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,14 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -212,6 +216,45 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
def.Flags().BoolVar(&clear, "clear", false, "no default any more")
|
||||
c.AddCommand(def)
|
||||
|
||||
var dsModel, dsOut string
|
||||
var dsPer, dsMax int
|
||||
dataset := &cobra.Command{
|
||||
Use: "dataset COLLECTION --model MODEL",
|
||||
Short: "Write training data from a collection: questions and answers, for an adapter",
|
||||
Long: "A chat model of the cluster reads each chunk of the collection and writes\n" +
|
||||
"questions a person might ask of it, with the answers it holds, one JSON object\n" +
|
||||
"a line ({\"q\", \"a\", \"doc\", \"ord\"}). Read it through and correct it before\n" +
|
||||
"training on it (llmgw ctl train): what the model got wrong, an adapter learns.\n\n" +
|
||||
"An adapter learns how answers sound — the organisation's words, format and\n" +
|
||||
"tone — far better than what they say; the facts stay the knowledge base's.",
|
||||
Example: " llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
w := cmd.OutOrStdout()
|
||||
if dsOut != "" {
|
||||
f, err := os.Create(dsOut)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
w = f
|
||||
}
|
||||
lw := &datasetWriter{pairs: w, problems: cmd.ErrOrStderr()}
|
||||
body, _ := json.Marshal(adminapi.KBDatasetRequest{Model: dsModel, PerChunk: dsPer, MaxChunks: dsMax})
|
||||
err := client().patient(6*time.Hour).do(cmd.Context(), "POST",
|
||||
"/api/v1/kb/collections/"+url.PathEscape(args[0])+"/dataset", json.RawMessage(body), lw)
|
||||
lw.flush()
|
||||
fmt.Fprintf(cmd.ErrOrStderr(), "%d pairs from %d chunks; %d chunks gave none\n", lw.n, len(lw.chunks), lw.failed)
|
||||
return err
|
||||
},
|
||||
}
|
||||
dataset.Flags().StringVar(&dsModel, "model", "", "the chat model that writes the questions and answers")
|
||||
dataset.Flags().StringVar(&dsOut, "out", "", "the file to write (default: standard output)")
|
||||
dataset.Flags().IntVar(&dsPer, "per-chunk", 8, "pairs from each chunk")
|
||||
dataset.Flags().IntVar(&dsMax, "max-chunks", 500, "the most chunks read")
|
||||
_ = dataset.MarkFlagRequired("model")
|
||||
c.AddCommand(dataset)
|
||||
|
||||
var k int
|
||||
search := &cobra.Command{
|
||||
Use: "search COLLECTION QUERY...",
|
||||
@@ -240,3 +283,50 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c.AddCommand(search)
|
||||
return c
|
||||
}
|
||||
|
||||
// datasetWriter splits the dataset stream: pairs to the file, problems to
|
||||
// the terminal, counted as they pass.
|
||||
type datasetWriter struct {
|
||||
pairs, problems io.Writer
|
||||
buf []byte
|
||||
n, failed int
|
||||
chunks map[string]bool
|
||||
}
|
||||
|
||||
func (d *datasetWriter) Write(p []byte) (int, error) {
|
||||
d.buf = append(d.buf, p...)
|
||||
for {
|
||||
i := bytes.IndexByte(d.buf, '\n')
|
||||
if i < 0 {
|
||||
return len(p), nil
|
||||
}
|
||||
d.line(d.buf[:i])
|
||||
d.buf = d.buf[i+1:]
|
||||
}
|
||||
}
|
||||
|
||||
func (d *datasetWriter) flush() {
|
||||
if len(bytes.TrimSpace(d.buf)) > 0 {
|
||||
d.line(d.buf)
|
||||
}
|
||||
d.buf = nil
|
||||
}
|
||||
|
||||
func (d *datasetWriter) line(b []byte) {
|
||||
var l adminapi.KBDatasetLine
|
||||
if json.Unmarshal(b, &l) != nil {
|
||||
return
|
||||
}
|
||||
if d.chunks == nil {
|
||||
d.chunks = map[string]bool{}
|
||||
}
|
||||
d.chunks[fmt.Sprintf("%s#%d", l.Doc, l.Ord)] = true
|
||||
if l.Error != "" {
|
||||
d.failed++
|
||||
fmt.Fprintf(d.problems, " %s #%d: %s\n", l.Doc, l.Ord, l.Error)
|
||||
return
|
||||
}
|
||||
d.n++
|
||||
out, _ := json.Marshal(l)
|
||||
fmt.Fprintln(d.pairs, string(out))
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -65,6 +66,14 @@ Safe to run again: it changes only what differs.`,
|
||||
}
|
||||
if !noFirewall {
|
||||
mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
|
||||
// And after, as the cluster's settings say: the control plane
|
||||
// tells the mirror, which writes it beside its models.
|
||||
state := filepath.Join(*dir, mirror.FirewallFile)
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " !! firewall: %v; the cluster's settings will not change it here\n", err)
|
||||
} else {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " ok firewall: follows the cluster's settings ("+firewallSyncPathUnit+")")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
|
||||
+11
-8
@@ -80,15 +80,18 @@ func newModelAddCmd(client func() *ctlClient) *cobra.Command {
|
||||
// resolve is otherwise discovered by a replica failing to start
|
||||
// minutes later on a node — with an error naming the node rather
|
||||
// than the typo.
|
||||
// Every source the model's profiles load, not the model's alone.
|
||||
if !skipCheck {
|
||||
size, err := sourcecheck.Check(cmd.Context(), m.Source)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(m.Source); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(m.Source, size))
|
||||
for _, src := range m.WeightSources() {
|
||||
size, err := sourcecheck.Check(cmd.Context(), src)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w\n Pass --no-source-check to add it anyway, "+
|
||||
"which is right when the mirror is not up yet or the nodes "+
|
||||
"reach it by an address this machine does not.", err)
|
||||
}
|
||||
if _, isURL := sourcecheck.HTTPSource(src); isURL {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), sourceNote(src, size))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -54,6 +54,7 @@ func Root() *cobra.Command {
|
||||
newMirrorCmd(),
|
||||
newBundleCmd(),
|
||||
newSimEngineCmd(),
|
||||
newComfyUIGateCmd(),
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
|
||||
+52
-10
@@ -49,6 +49,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
|
||||
@@ -209,6 +210,10 @@ func newServerCmd() *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if st := rt.Store(); st != nil {
|
||||
// Each control plane raises the same events; one sends each.
|
||||
notifier.SetClaimer(st)
|
||||
}
|
||||
rt.SetNotifier(notifier)
|
||||
if gate != nil && notifier != nil {
|
||||
gate.SetJudgeAlarm(func(model string, err error) {
|
||||
@@ -232,6 +237,9 @@ func newServerCmd() *cobra.Command {
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
var admin *control.Controller
|
||||
// What the settings say of the firewall, for the mirror and the
|
||||
// store's members; nil without a control plane.
|
||||
var fwPub *firewallPublisher
|
||||
updateMetrics := &modelserve.UpdateCollector{}
|
||||
// The store's cluster, when it is one; read by the metrics and
|
||||
// by `ctl status`.
|
||||
@@ -275,6 +283,19 @@ func newServerCmd() *cobra.Command {
|
||||
ctl.Reconciler().SetNotifier(notifier)
|
||||
admin = ctl
|
||||
agentPath, agentHandler = svc.Handler()
|
||||
fwPub = newFirewallPublisher(func() config.Server { return *current.Load() }, rt.Store(),
|
||||
func() []string {
|
||||
var out []string
|
||||
for _, n := range ctl.Inventory().Nodes() {
|
||||
out = append(out, n.MgmtAddr)
|
||||
}
|
||||
return out
|
||||
},
|
||||
func() bool { return ctl.Leadership() == nil || ctl.Leadership().IsLeader() }, log)
|
||||
go fwPub.run(ctx, 5*time.Minute)
|
||||
if l := ctl.Leadership(); l != nil {
|
||||
l.OnLead(fwPub.Kick)
|
||||
}
|
||||
|
||||
// Before AdminHandler, which is where the registry is built —
|
||||
// registering after it silently publishes nothing, which is
|
||||
@@ -387,6 +408,13 @@ func newServerCmd() *cobra.Command {
|
||||
// Knowledge bases, with the cluster's own models to embed a
|
||||
// probe (a collection's dimension) and an operator's search.
|
||||
api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
|
||||
// LoRA adapters trained on the nodes.
|
||||
if agentSvc != nil {
|
||||
trainer := train.New(st, agentSvc, log)
|
||||
agentSvc.SetTrain(trainer)
|
||||
go trainer.Run(ctx)
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
@@ -449,6 +477,7 @@ func newServerCmd() *cobra.Command {
|
||||
api.SetUsage(st)
|
||||
// Capacity quotas, set and read over this API.
|
||||
api.SetQuotas(st)
|
||||
api.SetTenancy(st)
|
||||
}
|
||||
api.Register(adminMux)
|
||||
|
||||
@@ -510,6 +539,16 @@ func newServerCmd() *cobra.Command {
|
||||
return fmt.Errorf("the model mirror: %w", err)
|
||||
}
|
||||
rt.SetMirror(mc)
|
||||
if fwPub != nil {
|
||||
fwPub.setMirror(mc)
|
||||
}
|
||||
// Adapters trained on the nodes are kept there too; sending one
|
||||
// takes longer than the mirror's usual answers.
|
||||
if adminAPI != nil {
|
||||
if up, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 10*time.Minute); err == nil {
|
||||
adminAPI.SetTrainMirror(up)
|
||||
}
|
||||
}
|
||||
// Browsing the registry is what turns "paste a resolve URL"
|
||||
// into "find the model". It needs a way out to the internet
|
||||
// from this machine, which the mirror's own fetching does not.
|
||||
@@ -727,6 +766,9 @@ func newServerCmd() *cobra.Command {
|
||||
if agentSvc != nil {
|
||||
agentSvc.BroadcastFirewall()
|
||||
}
|
||||
if fwPub != nil {
|
||||
fwPub.Kick()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
|
||||
// A moment apart per node, so off this goroutine.
|
||||
@@ -1131,18 +1173,18 @@ func servableCheck(llamaVersion string) func(context.Context, catalog.Model) err
|
||||
if llamaVersion == "" {
|
||||
return nil
|
||||
}
|
||||
usesLlama := false
|
||||
// Each source a llama.cpp profile loads: its own, or the model's.
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == "llamacpp" {
|
||||
usesLlama = true
|
||||
if p.Engine != "llamacpp" {
|
||||
continue
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.SourceOf(p))
|
||||
if !ok || !gguf.IsGGUF(url) {
|
||||
continue
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
}
|
||||
url, ok := sourcecheck.HTTPSource(m.Source)
|
||||
if !usesLlama || !ok || !gguf.IsGGUF(url) {
|
||||
return nil
|
||||
}
|
||||
if v := gguf.Check(ctx, url, llamaVersion); !v.Servable {
|
||||
return fmt.Errorf("%s cannot be served: %s", m.Name, v.Reason)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -160,7 +160,7 @@ func (sa *serverAdd) firewall(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
// What applies the cluster's firewall setting there when it changes.
|
||||
if err := installFirewallSync(ctx, sa.remote); err != nil {
|
||||
if err := installFirewallSync(ctx, sa.remote, firewallStateFile); err != nil {
|
||||
fmt.Fprintf(sa.out, " !! firewall: %v\n", err)
|
||||
}
|
||||
if !cfg.Firewall.Enabled {
|
||||
|
||||
@@ -93,6 +93,11 @@ func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, se
|
||||
} else {
|
||||
in.kept("firewall on %s", m.Name)
|
||||
}
|
||||
// And after, as the cluster's settings say, read from the
|
||||
// member's own copy of the store.
|
||||
if err := firewall.InstallFollower(ctx, m.Remote); err != nil {
|
||||
fmt.Fprintf(in.out, " !! firewall on %s: %v; the cluster's settings will not change it\n", m.Name, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
@@ -135,5 +140,5 @@ func loopbackListen(listen string) bool {
|
||||
// ensureFirewallSync installs the units that apply the cluster's firewall
|
||||
// setting on this control plane.
|
||||
func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
|
||||
return installFirewallSync(ctx, firewall.Local{})
|
||||
return installFirewallSync(ctx, firewall.Local{}, firewallStateFile)
|
||||
}
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -348,15 +349,23 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
|
||||
}
|
||||
|
||||
if installed {
|
||||
// Its defaults assume it owns the machine. A gateway swapped out to
|
||||
// give the database cache is worse than a smaller cache, and a store
|
||||
// whose working set is megabytes needs little. Only a PostgreSQL init
|
||||
// installed itself: one that was already here may serve something
|
||||
// else that needs its connections.
|
||||
for _, q := range []string{
|
||||
"ALTER SYSTEM SET shared_buffers = '96MB'",
|
||||
"ALTER SYSTEM SET max_connections = 40",
|
||||
} {
|
||||
// Sized to the machine's memory, as one it shares with the gateway:
|
||||
// a gateway swapped out to give the database cache is worse than a
|
||||
// smaller cache, but a knowledge base's vector index is searched
|
||||
// well only from memory. Only a PostgreSQL init installed itself:
|
||||
// one that was already here may serve something else that needs its
|
||||
// connections.
|
||||
meminfo, _ := os.ReadFile("/proc/meminfo")
|
||||
total, ok := pgha.MemTotal(string(meminfo))
|
||||
if !ok {
|
||||
total = 1 << 30
|
||||
}
|
||||
mem := pgha.MemorySettings(total, true)
|
||||
qs := []string{"ALTER SYSTEM SET max_connections = 40"}
|
||||
for _, k := range []string{"shared_buffers", "effective_cache_size", "maintenance_work_mem", "work_mem"} {
|
||||
qs = append(qs, "ALTER SYSTEM SET "+k+" = '"+mem[k]+"'")
|
||||
}
|
||||
for _, q := range qs {
|
||||
if _, err := in.psql(ctx, q); err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -367,7 +376,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
|
||||
if err := in.waitForPostgres(ctx); err != nil {
|
||||
return "", err
|
||||
}
|
||||
in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
|
||||
in.done("PostgreSQL sized for a machine it shares, with %d MB of memory (shared_buffers %s, work_mem %s, 40 connections)",
|
||||
total>>20, mem["shared_buffers"], mem["work_mem"])
|
||||
}
|
||||
|
||||
in.setupPgvector(ctx, discovery.DiscoverHost())
|
||||
|
||||
@@ -52,6 +52,8 @@ func TestModelStatusSeparatesWaitingFromBroken(t *testing.T) {
|
||||
{"nothing could be placed", adminapi.ModelView{Min: 1, Ready: 0, Total: 0}, "not placed"},
|
||||
{"deliberately not running", adminapi.ModelView{Min: 0, Ready: 0, Total: 0}, "stopped"},
|
||||
{"short but rebuilding", adminapi.ModelView{Min: 2, Ready: 1, Total: 2}, "starting"},
|
||||
{"woken by a request", adminapi.ModelView{Min: 0, Ready: 1, Total: 1}, "ready (on demand)"},
|
||||
{"waking", adminapi.ModelView{Min: 0, Ready: 0, Total: 1}, "starting"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := modelStatus(tc.view); got != tc.want {
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "train",
|
||||
Short: "Train LoRA adapters on the nodes, on question/answer pairs",
|
||||
Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
|
||||
"writes them from a knowledge base; read them through first). It trains with\n" +
|
||||
"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
|
||||
"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
|
||||
"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
|
||||
"as the model's adapter.\n\n" +
|
||||
"An adapter learns how answers sound — words, format, tone — far better than\n" +
|
||||
"what they say. Facts belong to the knowledge base.",
|
||||
}
|
||||
|
||||
var model, base, rev, node, data, owner string
|
||||
var public bool
|
||||
var gpuMem int
|
||||
var epochs, rank int
|
||||
var lr float64
|
||||
var noGGUF bool
|
||||
start := &cobra.Command{
|
||||
Use: "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
|
||||
Short: "Queue an adapter's training on a node",
|
||||
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
|
||||
" --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
b, err := os.ReadFile(data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
|
||||
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
|
||||
if cmd.Flags().Changed("epochs") {
|
||||
req.Params["epochs"] = epochs
|
||||
}
|
||||
if cmd.Flags().Changed("rank") {
|
||||
req.Params["rank"] = rank
|
||||
}
|
||||
if cmd.Flags().Changed("lr") {
|
||||
req.Params["lr"] = lr
|
||||
}
|
||||
if noGGUF {
|
||||
f := false
|
||||
req.GGUF = &f
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var v adminapi.TrainJobView
|
||||
if err := client().patient(2*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs", json.RawMessage(body), &v); err != nil {
|
||||
return err
|
||||
}
|
||||
who := "every caller of " + v.Model
|
||||
if v.Owner != "" {
|
||||
who = v.Owner + " only"
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
|
||||
v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := start.Flags()
|
||||
f.StringVar(&model, "model", "", "the catalog model the adapter is for")
|
||||
f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
|
||||
f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
|
||||
f.StringVar(&node, "node", "", "the node that trains")
|
||||
f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
|
||||
f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
|
||||
f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
|
||||
f.IntVar(&rank, "rank", 16, "the adapter's rank")
|
||||
f.Float64Var(&lr, "lr", 2e-4, "the learning rate")
|
||||
f.BoolVar(&noGGUF, "no-gguf", false, "skip the llama.cpp GGUF, even for a model llama.cpp serves")
|
||||
f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
|
||||
"(default: the knowledge base's owner, when the pairs come from one)")
|
||||
f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
|
||||
for _, req := range []string{"model", "base-repo", "node", "data"} {
|
||||
_ = start.MarkFlagRequired(req)
|
||||
}
|
||||
c.AddCommand(start)
|
||||
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Training jobs, newest first",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var jobs []adminapi.TrainJobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/train/jobs", &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No training jobs (llmgw ctl train start).")
|
||||
return nil
|
||||
}
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "JOB\tADAPTER\tMODEL\tOWNER\tNODE\tSTATE\tPROGRESS\tLOSS\tPEAK\tCREATED")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", j.ID, j.Name, j.Model, or(j.Owner, "everyone"), j.Node, j.State,
|
||||
trainProgress(j), trainLoss(j), trainPeak(j), j.CreatedAt.Local().Format("01-02 15:04"))
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
})
|
||||
|
||||
var follow bool
|
||||
show := &cobra.Command{
|
||||
Use: "show JOB",
|
||||
Short: "How a training job goes, and what it made",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
out := cmd.OutOrStdout()
|
||||
last := ""
|
||||
for {
|
||||
var j adminapi.TrainJobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/train/jobs/"+args[0], &j); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(out).Encode(j)
|
||||
}
|
||||
line := fmt.Sprintf("%s %s %s loss %s peak %s %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
|
||||
if !follow {
|
||||
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
|
||||
or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
|
||||
fmt.Fprintln(out, line)
|
||||
if len(j.Artifacts) > 0 {
|
||||
fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if line != last {
|
||||
fmt.Fprintf(out, "%s %s\n", time.Now().Format("15:04:05"), line)
|
||||
last = line
|
||||
}
|
||||
if store.TrainOver(j.State) {
|
||||
if j.State != store.TrainDone {
|
||||
return fmt.Errorf("%s %s", j.ID, j.State)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
select {
|
||||
case <-cmd.Context().Done():
|
||||
return nil
|
||||
case <-time.After(3 * time.Second):
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
show.Flags().BoolVarP(&follow, "follow", "f", false, "follow it until it ends")
|
||||
c.AddCommand(show)
|
||||
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "cancel JOB",
|
||||
Short: "Stop a training job",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := client().do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/cancel", nil, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s cancelled\n", args[0])
|
||||
return nil
|
||||
},
|
||||
})
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "rm JOB",
|
||||
Short: "Remove an ended training job and what it made",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := client().do(cmd.Context(), "DELETE", "/api/v1/train/jobs/"+args[0], nil, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
|
||||
return nil
|
||||
},
|
||||
})
|
||||
|
||||
var restart bool
|
||||
publish := &cobra.Command{
|
||||
Use: "publish JOB",
|
||||
Short: "Make what a finished job made its model's adapter, kept on the mirror",
|
||||
Long: "The GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves, are kept\n" +
|
||||
"on the mirror and added to the model's adapters; a request then asks for the\n" +
|
||||
"adapter by its name, as a model. llama.cpp loads adapters when a replica starts:\n" +
|
||||
"--restart replaces the model's replicas, one at a time, to load it.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var v adminapi.TrainPublishView
|
||||
if err := client().patient(15*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/publish", nil, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "%s is %s's adapter, from %s\n", v.Adapter, v.Model, v.Source)
|
||||
if v.Owner != "" {
|
||||
fmt.Fprintf(out, "only %s sees it; to anyone else it is a model that does not exist\n", v.Owner)
|
||||
}
|
||||
if restart && strings.Contains(v.Note, "restart") {
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Models: []string{v.Model}})
|
||||
var res adminapi.RestartResult
|
||||
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%d replica(s) of %s restarting to load it; then ask for model %s\n", len(res.Queued), v.Model, v.Adapter)
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintln(out, v.Note)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
publish.Flags().BoolVar(&restart, "restart", false, "restart the model's replicas to load it (llama.cpp)")
|
||||
c.AddCommand(publish)
|
||||
|
||||
var kind, outFile string
|
||||
download := &cobra.Command{
|
||||
Use: "download JOB",
|
||||
Short: "Save what a training job made: the PEFT adapter (peft) or the GGUF (gguf)",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if outFile == "" {
|
||||
outFile = args[0] + ".tar.gz"
|
||||
if kind == "gguf" {
|
||||
outFile = args[0] + ".gguf"
|
||||
}
|
||||
}
|
||||
f, err := os.Create(outFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
cw := &countWriter{w: f}
|
||||
if err := client().patient(10*time.Minute).do(cmd.Context(), "GET",
|
||||
"/api/v1/train/jobs/"+args[0]+"/artifacts/"+kind, nil, io.Writer(cw)); err != nil {
|
||||
_ = os.Remove(outFile)
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", outFile, humanBytes(cw.n))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
download.Flags().StringVar(&kind, "kind", "peft", "peft (a gzipped tar of the PEFT adapter) or gguf")
|
||||
download.Flags().StringVarP(&outFile, "out", "o", "", "the file to write")
|
||||
c.AddCommand(download)
|
||||
return c
|
||||
}
|
||||
|
||||
type countWriter struct {
|
||||
w io.Writer
|
||||
n int64
|
||||
}
|
||||
|
||||
func (c *countWriter) Write(p []byte) (int, error) {
|
||||
n, err := c.w.Write(p)
|
||||
c.n += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
func trainProgress(j adminapi.TrainJobView) string {
|
||||
if j.Steps == 0 {
|
||||
return "-"
|
||||
}
|
||||
return fmt.Sprintf("%d/%d", j.Step, j.Steps)
|
||||
}
|
||||
|
||||
func trainLoss(j adminapi.TrainJobView) string {
|
||||
if j.Loss == 0 {
|
||||
return "-"
|
||||
}
|
||||
return fmt.Sprintf("%.3f", j.Loss)
|
||||
}
|
||||
|
||||
func trainPeak(j adminapi.TrainJobView) string {
|
||||
if j.PeakMiB == 0 {
|
||||
return "-"
|
||||
}
|
||||
return fmt.Sprintf("%d MiB", j.PeakMiB)
|
||||
}
|
||||
@@ -0,0 +1,717 @@
|
||||
// Package comfy has images made by ComfyUI: a workflow filled in with what
|
||||
// was asked for, queued, waited on, and its pictures fetched.
|
||||
//
|
||||
// ComfyUI speaks no OpenAI. A workflow is a graph of nodes that is queued with
|
||||
// POST /prompt, is done when GET /history/{id} lists it, and leaves its
|
||||
// pictures to be read one by one from GET /view. What the graph is depends on
|
||||
// the model — FLUX schnell is not FLUX dev, and neither is SDXL — so the graph
|
||||
// is a template the catalog names, with placeholders where a request's
|
||||
// prompt, size and seed go.
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"embed"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/big"
|
||||
"mime/multipart"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"path"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed workflows/*.json
|
||||
var builtin embed.FS
|
||||
|
||||
// DefaultWorkflow is the template used when the catalog names none.
|
||||
const DefaultWorkflow = "flux-schnell"
|
||||
|
||||
// Builtins are the workflows that come with llmgw, by name.
|
||||
func Builtins() []string {
|
||||
entries, _ := builtin.ReadDir("workflows")
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Template is a workflow in ComfyUI's API format, with placeholders.
|
||||
//
|
||||
// A placeholder is a string value "{{name}}", or "{{name|default}}" for one a
|
||||
// request may leave out. One that is the whole value becomes the value
|
||||
// itself, a number where the node wants a number; one inside a longer string
|
||||
// is written into it. The names are those of Params.
|
||||
type Template struct {
|
||||
graph map[string]any
|
||||
}
|
||||
|
||||
// Load reads a template: a built-in's name, or the workflow's JSON itself.
|
||||
func Load(nameOrJSON string) (*Template, error) {
|
||||
s := strings.TrimSpace(nameOrJSON)
|
||||
if s == "" {
|
||||
s = DefaultWorkflow
|
||||
}
|
||||
raw := []byte(s)
|
||||
if !strings.HasPrefix(s, "{") {
|
||||
b, err := builtin.ReadFile("workflows/" + s + ".json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("no built-in workflow %q (there are %s)", s, strings.Join(Builtins(), ", "))
|
||||
}
|
||||
raw = b
|
||||
}
|
||||
var g map[string]any
|
||||
if err := json.Unmarshal(raw, &g); err != nil {
|
||||
return nil, fmt.Errorf("the workflow is not JSON: %w", err)
|
||||
}
|
||||
if len(g) == 0 {
|
||||
return nil, errors.New("the workflow has no nodes")
|
||||
}
|
||||
for id, n := range g {
|
||||
node, ok := n.(map[string]any)
|
||||
if !ok || node["class_type"] == nil {
|
||||
// The UI's own save format has a "nodes" list instead, and is
|
||||
// not something /prompt accepts.
|
||||
return nil, fmt.Errorf("node %q has no class_type: the workflow must be saved in API format", id)
|
||||
}
|
||||
}
|
||||
return &Template{graph: g}, nil
|
||||
}
|
||||
|
||||
// Params are what a request decides. A zero number is one not given, which
|
||||
// leaves it to the template's default.
|
||||
type Params struct {
|
||||
Checkpoint string
|
||||
Prompt string
|
||||
NegativePrompt string
|
||||
Width, Height int
|
||||
N int
|
||||
Seed int64
|
||||
Steps int
|
||||
Guidance float64
|
||||
// Image is the input picture, by the name Upload gave it in ComfyUI's
|
||||
// input directory: {{image}}.
|
||||
Image string
|
||||
// Extra are a workflow's own placeholders, by name: {{resolution}}.
|
||||
Extra map[string]any
|
||||
}
|
||||
|
||||
func (p Params) value(name string) (any, bool) {
|
||||
switch name {
|
||||
case "checkpoint":
|
||||
return p.Checkpoint, p.Checkpoint != ""
|
||||
case "prompt":
|
||||
return p.Prompt, true
|
||||
case "negative_prompt":
|
||||
return p.NegativePrompt, true
|
||||
case "width":
|
||||
return p.Width, p.Width > 0
|
||||
case "height":
|
||||
return p.Height, p.Height > 0
|
||||
case "n":
|
||||
return max(p.N, 1), true
|
||||
case "seed":
|
||||
return p.Seed, true
|
||||
case "steps":
|
||||
return p.Steps, p.Steps > 0
|
||||
case "guidance":
|
||||
return p.Guidance, p.Guidance > 0
|
||||
case "image":
|
||||
return p.Image, p.Image != ""
|
||||
}
|
||||
v, ok := p.Extra[name]
|
||||
return v, ok
|
||||
}
|
||||
|
||||
// Fill is the workflow for one request.
|
||||
func (t *Template) Fill(p Params) (map[string]any, error) {
|
||||
var missing []string
|
||||
var fill func(v any) any
|
||||
fill = func(v any) any {
|
||||
switch v := v.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for k, x := range v {
|
||||
out[k] = fill(x)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(v))
|
||||
for i, x := range v {
|
||||
out[i] = fill(x)
|
||||
}
|
||||
return out
|
||||
case string:
|
||||
return fillString(v, p, &missing)
|
||||
}
|
||||
return v
|
||||
}
|
||||
g := fill(t.graph).(map[string]any)
|
||||
if len(missing) > 0 {
|
||||
slices.Sort(missing)
|
||||
return nil, fmt.Errorf("the workflow wants %s, which the request did not give and the workflow has no default for",
|
||||
strings.Join(slices.Compact(missing), ", "))
|
||||
}
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func fillString(s string, p Params, missing *[]string) any {
|
||||
if !strings.Contains(s, "{{") {
|
||||
return s
|
||||
}
|
||||
resolve := func(ph string) (any, bool) {
|
||||
name, def, hasDef := strings.Cut(ph, "|")
|
||||
name = strings.TrimSpace(name)
|
||||
if v, ok := p.value(name); ok {
|
||||
return v, true
|
||||
}
|
||||
if hasDef {
|
||||
def = strings.TrimSpace(def)
|
||||
if f, err := strconv.ParseFloat(def, 64); err == nil {
|
||||
return f, true
|
||||
}
|
||||
return def, true
|
||||
}
|
||||
*missing = append(*missing, name)
|
||||
return nil, false
|
||||
}
|
||||
// The whole value: what the node gets is the value, not its text.
|
||||
if strings.HasPrefix(s, "{{") && strings.HasSuffix(s, "}}") && strings.Count(s, "{{") == 1 {
|
||||
v, _ := resolve(s[2 : len(s)-2])
|
||||
return v
|
||||
}
|
||||
var b strings.Builder
|
||||
for {
|
||||
i := strings.Index(s, "{{")
|
||||
j := strings.Index(s, "}}")
|
||||
if i < 0 || j < i {
|
||||
b.WriteString(s)
|
||||
return b.String()
|
||||
}
|
||||
b.WriteString(s[:i])
|
||||
if v, ok := resolve(s[i+2 : j]); ok {
|
||||
fmt.Fprint(&b, v)
|
||||
}
|
||||
s = s[j+2:]
|
||||
}
|
||||
}
|
||||
|
||||
// RandomSeed is a seed for a request that named none. ComfyUI's are 64-bit,
|
||||
// but JSON numbers are read as doubles on the way: kept to 2^53.
|
||||
func RandomSeed() int64 {
|
||||
n, err := rand.Int(rand.Reader, big.NewInt(1<<53))
|
||||
if err != nil {
|
||||
return time.Now().UnixNano() & (1<<53 - 1)
|
||||
}
|
||||
return n.Int64()
|
||||
}
|
||||
|
||||
// Image is one file a workflow made: a picture, a mesh.
|
||||
type Image struct {
|
||||
Data []byte
|
||||
ContentType string
|
||||
// Name is the file's own, as ComfyUI saved it.
|
||||
Name string
|
||||
}
|
||||
|
||||
// Error is a failure, said as what it means for the request.
|
||||
type Error struct {
|
||||
Msg string
|
||||
// Retryable is a failure of this server, which another one might not
|
||||
// have: it could not be reached, or it answered 5xx.
|
||||
Retryable bool
|
||||
}
|
||||
|
||||
func (e *Error) Error() string { return e.Msg }
|
||||
|
||||
// Client queues workflows and waits for them.
|
||||
type Client struct {
|
||||
HTTP *http.Client
|
||||
// Poll is how often a queued workflow is asked after.
|
||||
Poll time.Duration
|
||||
}
|
||||
|
||||
// Generate runs a workflow on the ComfyUI at base and returns its pictures,
|
||||
// in the order of the nodes that made them. A context that ends takes the
|
||||
// workflow out of the queue, or interrupts it if it is running, so a client
|
||||
// that went away does not leave the card working for nobody.
|
||||
func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map[string]any) ([]Image, error) {
|
||||
return c.GenerateWatch(ctx, base, apiKey, workflow, nil)
|
||||
}
|
||||
|
||||
// progressEvery is how often a workflow's progress is asked after, beside
|
||||
// its history: a few requests a second is for the end, not for the news.
|
||||
const progressEvery = time.Second
|
||||
|
||||
// GenerateWatch is Generate, saying how far along the workflow is as it
|
||||
// changes: queued and how many ahead, then the node running and its step.
|
||||
func (c *Client) GenerateWatch(ctx context.Context, base, apiKey string, workflow map[string]any,
|
||||
onProgress func(Progress)) ([]Image, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
client := make([]byte, 8)
|
||||
_, _ = rand.Read(client)
|
||||
body, err := json.Marshal(map[string]any{"prompt": workflow, "client_id": "llmgw-" + hex.EncodeToString(client)})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var queued struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Error json.RawMessage `json:"error"`
|
||||
NodeErrors json.RawMessage `json:"node_errors"`
|
||||
}
|
||||
status, err := c.do(ctx, http.MethodPost, base+"/prompt", apiKey, body, &queued)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if status != http.StatusOK || queued.PromptID == "" {
|
||||
// A workflow ComfyUI will not run: a checkpoint it does not have, a
|
||||
// node it does not know. The catalog's to fix, not the client's, and
|
||||
// no other replica set up the same way would take it either.
|
||||
return nil, &Error{Msg: "ComfyUI refused the workflow: " + refusal(queued.Error, queued.NodeErrors)}
|
||||
}
|
||||
id := queued.PromptID
|
||||
|
||||
poll := c.Poll
|
||||
if poll <= 0 {
|
||||
poll = 250 * time.Millisecond
|
||||
}
|
||||
t := time.NewTicker(poll)
|
||||
defer t.Stop()
|
||||
var last Progress
|
||||
var asked time.Time
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.cancel(context.WithoutCancel(ctx), base, apiKey, id)
|
||||
return nil, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
if onProgress != nil && time.Since(asked) >= progressEvery {
|
||||
asked = time.Now()
|
||||
if p, ok := c.progress(ctx, base, apiKey, id, workflow); ok && p != last {
|
||||
last = p
|
||||
onProgress(p)
|
||||
}
|
||||
}
|
||||
var hist map[string]historyEntry
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/history/"+url.PathEscape(id), apiKey, nil, &hist); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
continue // said by the select
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
h, ok := hist[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if h.Status.StatusStr == "error" {
|
||||
return nil, &Error{Msg: "the workflow failed in ComfyUI: " + h.failure()}
|
||||
}
|
||||
if !h.Status.Completed {
|
||||
continue
|
||||
}
|
||||
return c.fetch(ctx, base, apiKey, h)
|
||||
}
|
||||
}
|
||||
|
||||
type historyEntry struct {
|
||||
Status struct {
|
||||
StatusStr string `json:"status_str"`
|
||||
Completed bool `json:"completed"`
|
||||
Messages [][]json.RawMessage `json:"messages"`
|
||||
} `json:"status"`
|
||||
// Outputs are each node's, by kind: "images" for a picture, "3d" for
|
||||
// the mesh a 3D node saves, and whatever else a custom node names.
|
||||
Outputs map[string]map[string]json.RawMessage `json:"outputs"`
|
||||
}
|
||||
|
||||
// fileRef is a file an output names, to be read from /view.
|
||||
type fileRef struct {
|
||||
Filename string `json:"filename"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
Type string `json:"type"`
|
||||
}
|
||||
|
||||
// failure is what ComfyUI said went wrong, from its execution_error.
|
||||
func (h historyEntry) failure() string {
|
||||
for _, m := range h.Status.Messages {
|
||||
if len(m) < 2 || string(m[0]) != `"execution_error"` {
|
||||
continue
|
||||
}
|
||||
var e struct {
|
||||
NodeType string `json:"node_type"`
|
||||
Message string `json:"exception_message"`
|
||||
}
|
||||
if json.Unmarshal(m[1], &e) == nil && e.Message != "" {
|
||||
return strings.TrimSpace(e.NodeType + ": " + strings.TrimSpace(e.Message))
|
||||
}
|
||||
}
|
||||
return "no reason given"
|
||||
}
|
||||
|
||||
func (c *Client) fetch(ctx context.Context, base, apiKey string, h historyEntry) ([]Image, error) {
|
||||
nodes := make([]string, 0, len(h.Outputs))
|
||||
for id := range h.Outputs {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
// Numerically where they are numbers, which is how ComfyUI names them.
|
||||
slices.SortFunc(nodes, func(a, b string) int {
|
||||
x, ea := strconv.Atoi(a)
|
||||
y, eb := strconv.Atoi(b)
|
||||
if ea == nil && eb == nil {
|
||||
return x - y
|
||||
}
|
||||
return strings.Compare(a, b)
|
||||
})
|
||||
var out []Image
|
||||
for _, n := range nodes {
|
||||
kinds := make([]string, 0, len(h.Outputs[n]))
|
||||
for k := range h.Outputs[n] {
|
||||
kinds = append(kinds, k)
|
||||
}
|
||||
slices.Sort(kinds)
|
||||
for _, k := range kinds {
|
||||
// Any list of files, whatever the node calls it; text and
|
||||
// numbers a node reports beside them are not files.
|
||||
var refs []fileRef
|
||||
if json.Unmarshal(h.Outputs[n][k], &refs) != nil {
|
||||
continue
|
||||
}
|
||||
for _, f := range refs {
|
||||
if f.Filename == "" {
|
||||
continue
|
||||
}
|
||||
q := url.Values{"filename": {f.Filename}, "subfolder": {f.Subfolder}, "type": {f.Type}}
|
||||
img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
img.Name = f.Filename
|
||||
if t := typeByName(f.Filename); t != "" && (img.ContentType == "" ||
|
||||
strings.HasPrefix(img.ContentType, "application/octet-stream") || strings.HasPrefix(img.ContentType, "text/plain")) {
|
||||
img.ContentType = t
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, &Error{Msg: "the workflow finished without a file: it has no node that saves one"}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// cancel takes a workflow out of the queue, or stops it if it is the one
|
||||
// running. Best effort: the server may be gone, which also stops it.
|
||||
func (c *Client) cancel(ctx context.Context, base, apiKey, id string) {
|
||||
ctx, done := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer done()
|
||||
body, _ := json.Marshal(map[string]any{"delete": []string{id}})
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/queue", apiKey, body, nil)
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return
|
||||
}
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == strconv.Quote(id) {
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/interrupt", apiKey, []byte("{}"), nil)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) client() *http.Client {
|
||||
if c.HTTP != nil {
|
||||
return c.HTTP
|
||||
}
|
||||
return http.DefaultClient
|
||||
}
|
||||
|
||||
func (c *Client) request(ctx context.Context, method, u, apiKey string, body []byte) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, method, u, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return nil, &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// do sends a request and reads a JSON answer into out. A 5xx is an error;
|
||||
// any other status is returned with whatever body could be read.
|
||||
func (c *Client) do(ctx context.Context, method, u, apiKey string, body []byte, out any) (int, error) {
|
||||
resp, err := c.request(ctx, method, u, apiKey, body)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
|
||||
if err != nil {
|
||||
return 0, &Error{Msg: "reading ComfyUI's answer: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode >= 500 {
|
||||
return resp.StatusCode, &Error{Msg: fmt.Sprintf("ComfyUI answered %d: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
if out != nil && len(b) > 0 {
|
||||
if err := json.Unmarshal(b, out); err != nil && resp.StatusCode == http.StatusOK {
|
||||
return resp.StatusCode, &Error{Msg: "ComfyUI's answer is not JSON: " + snippet(b)}
|
||||
}
|
||||
}
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
func (c *Client) get(ctx context.Context, u, apiKey string) (Image, error) {
|
||||
resp, err := c.request(ctx, http.MethodGet, u, apiKey, nil)
|
||||
if err != nil {
|
||||
return Image{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, maxFile))
|
||||
if err != nil {
|
||||
return Image{}, &Error{Msg: "reading a file from ComfyUI: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a file it made", resp.StatusCode)}
|
||||
}
|
||||
ct := resp.Header.Get("Content-Type")
|
||||
if ct == "" || ct == "application/octet-stream" {
|
||||
ct = http.DetectContentType(b)
|
||||
}
|
||||
if ct == "image/png" {
|
||||
b = withoutPNGText(b)
|
||||
}
|
||||
return Image{Data: b, ContentType: ct}, nil
|
||||
}
|
||||
|
||||
// withoutPNGText is a PNG without its text chunks. ComfyUI writes the whole
|
||||
// workflow into every picture it saves — the prompt, and the checkpoint and
|
||||
// nodes behind it — and what a picture says about how it was made is the
|
||||
// operator's to publish, not something to hand every client. A file that is
|
||||
// not a well-formed PNG is returned as it came.
|
||||
func withoutPNGText(b []byte) []byte {
|
||||
const sig = "\x89PNG\r\n\x1a\n"
|
||||
if len(b) < len(sig) || string(b[:len(sig)]) != sig {
|
||||
return b
|
||||
}
|
||||
out := make([]byte, 0, len(b))
|
||||
out = append(out, b[:len(sig)]...)
|
||||
for i := len(sig); i < len(b); {
|
||||
if i+12 > len(b) {
|
||||
return b
|
||||
}
|
||||
n := int(binary.BigEndian.Uint32(b[i:]))
|
||||
end := i + 12 + n
|
||||
if n < 0 || end > len(b) {
|
||||
return b
|
||||
}
|
||||
switch string(b[i+4 : i+8]) {
|
||||
case "tEXt", "iTXt", "zTXt":
|
||||
default:
|
||||
out = append(out, b[i:end]...)
|
||||
}
|
||||
i = end
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// refusal is what /prompt said when it would not queue a workflow.
|
||||
func refusal(e, nodes json.RawMessage) string {
|
||||
var top struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
}
|
||||
msg := snippet(e)
|
||||
if json.Unmarshal(e, &top) == nil && top.Message != "" {
|
||||
msg = top.Message
|
||||
if top.Details != "" {
|
||||
msg += ": " + top.Details
|
||||
}
|
||||
}
|
||||
var perNode map[string]struct {
|
||||
ClassType string `json:"class_type"`
|
||||
Errors []struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
} `json:"errors"`
|
||||
}
|
||||
if json.Unmarshal(nodes, &perNode) == nil {
|
||||
var parts []string
|
||||
for _, n := range perNode {
|
||||
for _, er := range n.Errors {
|
||||
parts = append(parts, strings.TrimSpace(n.ClassType+": "+er.Message+" "+er.Details))
|
||||
}
|
||||
}
|
||||
slices.Sort(parts)
|
||||
if len(parts) > 0 {
|
||||
msg += " (" + strings.Join(parts, "; ") + ")"
|
||||
}
|
||||
}
|
||||
if msg == "" {
|
||||
msg = "no reason given"
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
func snippet(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// maxFile bounds one file read from ComfyUI: a mesh with its textures runs
|
||||
// to tens of megabytes, a picture to a few.
|
||||
const maxFile = 256 << 20
|
||||
|
||||
// typeByName is a file's type by its extension, for those a server sends
|
||||
// as octet-stream: the meshes a 3D node saves.
|
||||
func typeByName(name string) string {
|
||||
switch strings.ToLower(path.Ext(name)) {
|
||||
case ".glb":
|
||||
return "model/gltf-binary"
|
||||
case ".gltf":
|
||||
return "model/gltf+json"
|
||||
case ".obj":
|
||||
return "model/obj"
|
||||
case ".stl":
|
||||
return "model/stl"
|
||||
case ".ply":
|
||||
return "application/ply"
|
||||
case ".png":
|
||||
return "image/png"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// Upload puts a picture in ComfyUI's input directory, where a LoadImage node
|
||||
// reads it, and returns the name it was given there.
|
||||
func (c *Client) Upload(ctx context.Context, base, apiKey, name string, data []byte) (string, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
var body bytes.Buffer
|
||||
mw := multipart.NewWriter(&body)
|
||||
fw, err := mw.CreateFormFile("image", name)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := fw.Write(data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_ = mw.WriteField("type", "input")
|
||||
_ = mw.WriteField("overwrite", "true")
|
||||
if err := mw.Close(); err != nil {
|
||||
return "", err
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPost, base+"/upload/image", &body)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
req.Header.Set("Content-Type", mw.FormDataContentType())
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return "", ctx.Err()
|
||||
}
|
||||
return "", &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if resp.StatusCode >= 500 {
|
||||
return "", &Error{Msg: fmt.Sprintf("ComfyUI answered %d to the picture: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
var out struct {
|
||||
Name string `json:"name"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK || json.Unmarshal(b, &out) != nil || out.Name == "" {
|
||||
return "", &Error{Msg: fmt.Sprintf("ComfyUI did not take the picture (%d): %s", resp.StatusCode, snippet(b))}
|
||||
}
|
||||
if out.Subfolder != "" {
|
||||
return out.Subfolder + "/" + out.Name, nil
|
||||
}
|
||||
return out.Name, nil
|
||||
}
|
||||
|
||||
// progress is a workflow's: its place in the queue, from ComfyUI, and the
|
||||
// node running and its step, from the gate when there is one — a ComfyUI
|
||||
// outside the cluster has no gate, and says only where it is queued.
|
||||
func (c *Client) progress(ctx context.Context, base, apiKey, id string, workflow map[string]any) (Progress, bool) {
|
||||
p := Progress{PromptID: id, State: "queued"}
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
Pending [][]json.RawMessage `json:"queue_pending"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return p, false
|
||||
}
|
||||
quoted := strconv.Quote(id)
|
||||
queued := false
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == quoted {
|
||||
p.State = "running"
|
||||
}
|
||||
}
|
||||
if p.State != "running" {
|
||||
// Pending in the order ComfyUI will take them: by number.
|
||||
mine := -1.0
|
||||
var numbers []float64
|
||||
for _, r := range q.Pending {
|
||||
if len(r) < 2 {
|
||||
continue
|
||||
}
|
||||
var n float64
|
||||
_ = json.Unmarshal(r[0], &n)
|
||||
if string(r[1]) == quoted {
|
||||
mine, queued = n, true
|
||||
continue
|
||||
}
|
||||
numbers = append(numbers, n)
|
||||
}
|
||||
if queued {
|
||||
p.Ahead = len(q.Running)
|
||||
for _, n := range numbers {
|
||||
if n < mine {
|
||||
p.Ahead++
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
}
|
||||
var g Progress
|
||||
if st, err := c.do(ctx, http.MethodGet, base+"/progress/"+url.PathEscape(id), apiKey, nil, &g); err == nil &&
|
||||
st == http.StatusOK && g.State != "" {
|
||||
p = g
|
||||
p.PromptID = id
|
||||
if node, ok := workflow[p.Node].(map[string]any); ok {
|
||||
p.NodeClass, _ = node["class_type"].(string)
|
||||
}
|
||||
}
|
||||
return p, true
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
|
||||
for _, name := range Builtins() {
|
||||
tpl, err := Load(name)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "flux.safetensors", Prompt: `a "red" fox`, Width: 768, Height: 512, N: 2, Seed: 42})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
b, _ := json.Marshal(g)
|
||||
s := string(b)
|
||||
for _, want := range []string{`"ckpt_name":"flux.safetensors"`, `"text":"a \"red\" fox"`, `"width":768`,
|
||||
`"height":512`, `"batch_size":2`, `"seed":42`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("%s: %s not in %s", name, want, s)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "{{") {
|
||||
t.Errorf("%s: a placeholder was left: %s", name, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStepCountGivenReplacesTheDefault(t *testing.T) {
|
||||
tpl, _ := Load("flux-schnell")
|
||||
g, _ := tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 4.0 {
|
||||
t.Errorf("default steps = %v, want 4", steps)
|
||||
}
|
||||
g, _ = tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512, Steps: 8})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 8 {
|
||||
t.Errorf("steps = %v, want 8", steps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPlaceholderWithoutAValueOrADefaultIsSaid(t *testing.T) {
|
||||
tpl, err := Load(`{"1":{"class_type":"X","inputs":{"a":"{{checkpoint}}","b":"size {{width}}x{{height}}"}}}`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tpl.Fill(Params{Width: 5}); err == nil || !strings.Contains(err.Error(), "checkpoint, height") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "c", Width: 5, Height: 6})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b := g["1"].(map[string]any)["inputs"].(map[string]any)["b"]; b != "size 5x6" {
|
||||
t.Errorf("b = %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowInTheUIsFormatIsRefused(t *testing.T) {
|
||||
if _, err := Load(`{"nodes":[{"id":1}],"links":[]}`); err == nil || !strings.Contains(err.Error(), "API format") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
if _, err := Load("sdxl-turbo"); err == nil {
|
||||
t.Fatal("an unknown built-in was loaded")
|
||||
}
|
||||
}
|
||||
|
||||
// fakeComfy is just enough of ComfyUI: /prompt queues, /history answers
|
||||
// after a few asks, /view serves a picture.
|
||||
type fakeComfy struct {
|
||||
mu sync.Mutex
|
||||
asked int
|
||||
fail string
|
||||
refuse bool
|
||||
deleted []string
|
||||
queued map[string]any
|
||||
readyAt int
|
||||
srv *httptest.Server
|
||||
interrupted bool
|
||||
}
|
||||
|
||||
func newFakeComfy(t *testing.T) *fakeComfy {
|
||||
f := &fakeComfy{readyAt: 2}
|
||||
f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
switch {
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/prompt":
|
||||
if f.refuse {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
fmt.Fprint(w, `{"error":{"message":"Prompt outputs failed validation","details":""},"node_errors":{"1":{"class_type":"CheckpointLoaderSimple","errors":[{"message":"Value not in list","details":"ckpt_name: 'x' not in []"}]}}}`)
|
||||
return
|
||||
}
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.queued = b.Prompt
|
||||
fmt.Fprint(w, `{"prompt_id":"p1","number":1,"node_errors":{}}`)
|
||||
case r.URL.Path == "/history/p1":
|
||||
f.asked++
|
||||
if f.asked < f.readyAt {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
if f.fail != "" {
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"error","completed":false,"messages":[["execution_start",{}],["execution_error",{"node_type":"KSampler","exception_message":%q}]]},"outputs":{}}}`, f.fail)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true,"messages":[]},"outputs":{"7":{"images":[{"filename":"a.png","subfolder":"","type":"temp"},{"filename":"b.png","subfolder":"","type":"temp"}]}}}}`)
|
||||
case r.URL.Path == "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "PNG:"+r.URL.Query().Get("filename"))
|
||||
case r.URL.Path == "/queue" && r.Method == http.MethodPost:
|
||||
var b struct {
|
||||
Delete []string `json:"delete"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.deleted = append(f.deleted, b.Delete...)
|
||||
case r.URL.Path == "/queue":
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"p1",{},{},[]]],"queue_pending":[]}`)
|
||||
case r.URL.Path == "/interrupt":
|
||||
f.interrupted = true
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(f.srv.Close)
|
||||
return f
|
||||
}
|
||||
|
||||
func TestGenerateWaitsForTheWorkflowAndFetchesItsPictures(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
images, err := c.Generate(t.Context(), f.srv.URL, "", map[string]any{"1": map[string]any{"class_type": "X"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(images) != 2 || string(images[0].Data) != "PNG:a.png" || images[1].ContentType != "image/png" {
|
||||
t.Fatalf("images = %+v", images)
|
||||
}
|
||||
if f.queued["1"] == nil {
|
||||
t.Error("the workflow was not what was queued")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowThatFailsSaysWhy(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.fail = "CUDA out of memory"
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if err == nil || !strings.Contains(err.Error(), "KSampler: CUDA out of memory") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowComfyUIRefusesIsNotRetried(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.refuse = true
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
ce, ok := err.(*Error)
|
||||
if !ok || ce.Retryable || !strings.Contains(ce.Msg, "Value not in list") {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIThatCannotBeReachedIsRetryable(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.srv.Close()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if ce, ok := err.(*Error); !ok || !ce.Retryable {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientThatLeavesTakesItsWorkflowOffTheCard(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.readyAt = 1 << 30
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(ctx, f.srv.URL, "", map[string]any{})
|
||||
if err == nil {
|
||||
t.Fatal("no error")
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if len(f.deleted) != 1 || f.deleted[0] != "p1" || !f.interrupted {
|
||||
t.Errorf("deleted %v, interrupted %v", f.deleted, f.interrupted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureLosesTheWorkflowComfyUIWroteIntoIt(t *testing.T) {
|
||||
chunk := func(kind, data string) []byte {
|
||||
b := make([]byte, 0, 12+len(data))
|
||||
b = binary.BigEndian.AppendUint32(b, uint32(len(data)))
|
||||
b = append(b, kind...)
|
||||
b = append(b, data...)
|
||||
return append(b, 0, 0, 0, 0) // a CRC nobody here checks
|
||||
}
|
||||
png := []byte("\x89PNG\r\n\x1a\n")
|
||||
png = append(png, chunk("IHDR", "0123456789abc")...)
|
||||
png = append(png, chunk("tEXt", "prompt\x00{\"1\":...}")...)
|
||||
png = append(png, chunk("IDAT", "pixels")...)
|
||||
png = append(png, chunk("iTXt", "workflow\x00...")...)
|
||||
png = append(png, chunk("IEND", "")...)
|
||||
got := string(withoutPNGText(png))
|
||||
if strings.Contains(got, "prompt") || strings.Contains(got, "workflow") {
|
||||
t.Errorf("text left in: %q", got)
|
||||
}
|
||||
for _, kind := range []string{"IHDR", "IDAT", "pixels", "IEND"} {
|
||||
if !strings.Contains(got, kind) {
|
||||
t.Errorf("%s gone: %q", kind, got)
|
||||
}
|
||||
}
|
||||
if broken := png[:30]; string(withoutPNGText(broken)) != string(broken) {
|
||||
t.Error("a truncated PNG was changed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The gate is how a node runs ComfyUI as one of llmgw's engines.
|
||||
//
|
||||
// ComfyUI asks nobody for a key: whoever reaches its port can queue work,
|
||||
// upload files and install nodes. Every other engine here demands the key
|
||||
// the control plane made for its replica, so ComfyUI is put behind a gate
|
||||
// that does: ComfyUI listens on the loopback only, and the gate on the
|
||||
// replica's port, letting through only the requests the router makes. The
|
||||
// gate also answers what the agent asks every engine — /health once
|
||||
// ComfyUI is up, and /v1/models naming the model — and lays the weights out
|
||||
// where ComfyUI looks for them.
|
||||
//
|
||||
// ComfyUI runs as the gate's child, in its process group, so stopping the
|
||||
// replica stops both, the card memory ComfyUI holds is counted as the
|
||||
// replica's, and either one exiting ends the other.
|
||||
|
||||
// GateConfig is one replica's gate.
|
||||
type GateConfig struct {
|
||||
// Listen is the replica's address, the one the router is given.
|
||||
Listen string
|
||||
// Model is the name the agent asks /v1/models for.
|
||||
Model string
|
||||
// Key is what every request must carry as a bearer token; empty lets
|
||||
// every request through, as an engine started without one does.
|
||||
Key string
|
||||
// Weights is the model as the agent fetched it: a file, or a directory
|
||||
// of files.
|
||||
Weights string
|
||||
// State is the replica's own directory: ComfyUI's base directory, with
|
||||
// its output, temp and user directories, and the models it is shown.
|
||||
State string
|
||||
// Python runs ComfyUI's main.py, found in Dir.
|
||||
Python, Dir string
|
||||
// Args are passed on to main.py, after the gate's own.
|
||||
Args []string
|
||||
// Output is where ComfyUI's output goes.
|
||||
Output io.Writer
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// The requests the router makes of ComfyUI: queueing, waiting, fetching and
|
||||
// cancelling. Nothing that uploads, installs or changes ComfyUI.
|
||||
var gatePaths = []string{"/prompt", "/history/", "/view", "/queue", "/interrupt", "/system_stats", "/object_info",
|
||||
// The input picture of a 3D model: into ComfyUI's input directory,
|
||||
// which is the replica's own.
|
||||
"/upload/image"}
|
||||
|
||||
// modelFolders are the kinds of file ComfyUI's loaders look for. Each is
|
||||
// shown the same directory: a workflow names its files, so a file being
|
||||
// listed where it is not wanted costs nothing.
|
||||
var modelFolders = []string{"checkpoints", "diffusion_models", "text_encoders", "vae", "loras",
|
||||
"clip_vision", "controlnet", "upscale_models", "embeddings", "style_models"}
|
||||
|
||||
// RunGate lays out the replica, starts ComfyUI and serves the gate until
|
||||
// ctx ends or ComfyUI exits.
|
||||
func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
log := cfg.Log
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
state, err := filepath.Abs(cfg.State)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// What ComfyUI expects to find in its base directory, and dies
|
||||
// without: custom_nodes is listed before anything else is done.
|
||||
for _, d := range []string{"custom_nodes", "input", "output", "temp", "user"} {
|
||||
if err := os.MkdirAll(filepath.Join(state, d), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
out := cfg.Output
|
||||
if out == nil {
|
||||
out = os.Stdout
|
||||
}
|
||||
nodes, err := parseNodes(os.Getenv(NodesEnv))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
site, err := installNodes(ctx, state, cfg.Python, nodes, findUV(), func(s string) { fmt.Fprintln(out, "[gate] "+s) })
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
models := filepath.Join(state, "models")
|
||||
if err := LinkWeights(cfg.Weights, models); err != nil {
|
||||
return err
|
||||
}
|
||||
extra := filepath.Join(state, "extra_model_paths.yaml")
|
||||
if err := os.WriteFile(extra, []byte(extraModelPaths(models)), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return err
|
||||
}
|
||||
|
||||
inner, err := freePort()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
args := append([]string{filepath.Join(cfg.Dir, "main.py"),
|
||||
"--listen", "127.0.0.1", "--port", strconv.Itoa(inner),
|
||||
"--base-directory", state, "--extra-model-paths-config", extra,
|
||||
"--disable-auto-launch"}, cfg.Args...)
|
||||
child := exec.Command(cfg.Python, args...) //nolint:gosec // the agent's own configuration
|
||||
child.Dir = cfg.Dir
|
||||
child.Env = os.Environ()
|
||||
if site != "" {
|
||||
pp := site
|
||||
if old := os.Getenv("PYTHONPATH"); old != "" {
|
||||
pp += string(os.PathListSeparator) + old
|
||||
}
|
||||
child.Env = append(child.Env, "PYTHONPATH="+pp)
|
||||
}
|
||||
// ComfyUI writes to a pipe of the gate's, never to what the gate was
|
||||
// given. That is the agent's, and an agent that restarts and adopts the
|
||||
// replica leaves it with no reader: ComfyUI's next progress bar would
|
||||
// fail with a broken pipe, and take the workflow it belongs to with it.
|
||||
// The gate reads the pipe for as long as ComfyUI writes, and passes on
|
||||
// what it can.
|
||||
signal.Ignore(syscall.SIGPIPE)
|
||||
pr, err := child.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
child.Stderr = child.Stdout
|
||||
if err := child.Start(); err != nil {
|
||||
return fmt.Errorf("starting ComfyUI: %w", err)
|
||||
}
|
||||
relayed := make(chan struct{})
|
||||
go func() { relay(pr, out); close(relayed) }()
|
||||
exited := make(chan error, 1)
|
||||
go func() {
|
||||
// Wait closes the pipe; what is in it is read first.
|
||||
<-relayed
|
||||
exited <- child.Wait()
|
||||
}()
|
||||
|
||||
upstream := &url.URL{Scheme: "http", Host: net.JoinHostPort("127.0.0.1", strconv.Itoa(inner))}
|
||||
// The workflows' progress, heard on ComfyUI's websocket under an id of
|
||||
// the gate's own.
|
||||
book := newProgressBook()
|
||||
cid := make([]byte, 8)
|
||||
_, _ = rand.Read(cid)
|
||||
clientID := "llmgw-gate-" + hex.EncodeToString(cid)
|
||||
go book.watch(ctx, upstream, clientID, log)
|
||||
srv := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: newGate(upstream, cfg.Key, cfg.Model, book, clientID),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
served := make(chan error, 1)
|
||||
go func() { served <- srv.ListenAndServe() }()
|
||||
log.Info("ComfyUI gate up", "listen", cfg.Listen, "comfyui", upstream.Host, "weights", cfg.Weights)
|
||||
|
||||
stop := func() {
|
||||
_ = child.Process.Signal(syscall.SIGTERM)
|
||||
select {
|
||||
case <-exited:
|
||||
case <-time.After(20 * time.Second):
|
||||
_ = child.Process.Kill()
|
||||
<-exited
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
shutdown(srv)
|
||||
stop()
|
||||
return nil
|
||||
case err := <-exited:
|
||||
shutdown(srv)
|
||||
if err == nil {
|
||||
err = errors.New("exited")
|
||||
}
|
||||
return fmt.Errorf("ComfyUI %w", err)
|
||||
case err := <-served:
|
||||
stop()
|
||||
return fmt.Errorf("the gate stopped serving: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
func shutdown(srv *http.Server) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = srv.Shutdown(ctx)
|
||||
}
|
||||
|
||||
// NewGate is the gate's handler, in front of the ComfyUI at upstream.
|
||||
func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
return newGate(upstream, key, model, newProgressBook(), "")
|
||||
}
|
||||
|
||||
// newGate is the gate with the book of its workflows' progress, kept under
|
||||
// clientID; an empty one leaves the workflows' own client ids alone.
|
||||
func newGate(upstream *url.URL, key, model string, book *progressBook, clientID string) http.Handler {
|
||||
proxy := httputil.NewSingleHostReverseProxy(upstream)
|
||||
base := proxy.Director
|
||||
proxy.Director = func(r *http.Request) {
|
||||
base(r)
|
||||
r.Header.Del("Authorization")
|
||||
// ComfyUI refuses a request whose Origin is not its own host; the
|
||||
// router sends none, and nothing else gets this far.
|
||||
r.Header.Del("Origin")
|
||||
r.Host = upstream.Host
|
||||
}
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
up := func(ctx context.Context) error {
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, upstream.String()+"/system_stats", nil)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("ComfyUI answered %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if key != "" {
|
||||
got, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
|
||||
if !ok || subtle.ConstantTimeCompare([]byte(got), []byte(key)) != 1 {
|
||||
http.Error(w, "this engine wants its key", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
}
|
||||
switch r.URL.Path {
|
||||
case "/health":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
case "/v1/models":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprintf(w, `{"object":"list","data":[{"id":%q,"object":"model","owned_by":"comfyui"}]}`, model)
|
||||
return
|
||||
}
|
||||
if id, ok := strings.CutPrefix(r.URL.Path, "/progress/"); ok && r.Method == http.MethodGet {
|
||||
p, found := book.get(id)
|
||||
if !found {
|
||||
http.Error(w, "no word of that workflow", http.StatusNotFound)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(p)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/prompt" && r.Method == http.MethodPost && clientID != "" {
|
||||
book.promptThroughGate(proxy, clientID)(w, r)
|
||||
return
|
||||
}
|
||||
for _, p := range gatePaths {
|
||||
if r.URL.Path == p || strings.HasSuffix(p, "/") && strings.HasPrefix(r.URL.Path, p) {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// cachedName is a file the agent fetched by itself: its cache names it by
|
||||
// the source's hash, then the name it had.
|
||||
var cachedName = regexp.MustCompile(`^[0-9a-f]{16}-`)
|
||||
|
||||
// FileName is what a weights file is called where ComfyUI sees it: its own
|
||||
// name, without the prefix the agent's cache gives it.
|
||||
func FileName(path string) string {
|
||||
return cachedName.ReplaceAllString(filepath.Base(path), "")
|
||||
}
|
||||
|
||||
// LinkWeights shows ComfyUI the weights: a link in dir to the file, or to
|
||||
// every file of the directory. Links made before are taken away first, so
|
||||
// a replica started again on other weights sees only those.
|
||||
func LinkWeights(weights, dir string) error {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
old, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range old {
|
||||
if e.Type()&os.ModeSymlink != 0 {
|
||||
_ = os.Remove(filepath.Join(dir, e.Name()))
|
||||
}
|
||||
}
|
||||
weights, err = filepath.Abs(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
info, err := os.Stat(weights)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the weights: %w", err)
|
||||
}
|
||||
files := []string{weights}
|
||||
if info.IsDir() {
|
||||
entries, err := os.ReadDir(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files = files[:0]
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() && !strings.HasPrefix(e.Name(), ".") {
|
||||
files = append(files, filepath.Join(weights, e.Name()))
|
||||
}
|
||||
}
|
||||
if len(files) == 0 {
|
||||
return fmt.Errorf("the weights directory %s has no files", weights)
|
||||
}
|
||||
}
|
||||
for _, f := range files {
|
||||
if err := os.Symlink(f, filepath.Join(dir, FileName(f))); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func extraModelPaths(models string) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "llmgw:\n base_path: %q\n is_default: true\n", models)
|
||||
for _, f := range modelFolders {
|
||||
fmt.Fprintf(&b, " %s: .\n", f)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer l.Close()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
// relay copies r to w for as long as r has anything, and goes on reading
|
||||
// after w fails: a writer that stops reading must not stop the one writing.
|
||||
func relay(r io.Reader, w io.Writer) {
|
||||
buf := make([]byte, 32<<10)
|
||||
broken := false
|
||||
for {
|
||||
n, err := r.Read(buf)
|
||||
if n > 0 && !broken {
|
||||
if _, werr := w.Write(buf[:n]); werr != nil {
|
||||
broken = true
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheGateWantsTheKeyAndLetsThroughOnlyTheRoutersRequests(t *testing.T) {
|
||||
var got []string
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = append(got, r.Method+" "+r.URL.Path+" auth="+r.Header.Get("Authorization"))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
gate := httptest.NewServer(NewGate(u, "k", "flux"))
|
||||
defer gate.Close()
|
||||
|
||||
do := func(method, path, key string) (int, string) {
|
||||
req, _ := http.NewRequest(method, gate.URL+path, strings.NewReader("{}"))
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, string(b)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", ""); code != http.StatusUnauthorized {
|
||||
t.Errorf("no key: %d", code)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", "wrong"); code != http.StatusUnauthorized {
|
||||
t.Errorf("wrong key: %d", code)
|
||||
}
|
||||
for _, p := range []string{"/upload/mask", "/api/manager/queue/install", "/userdata/x", "/internal/logs"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusNotFound {
|
||||
t.Errorf("%s: %d, want it refused", p, code)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"/prompt", "/history/p1", "/view?filename=a.png", "/queue", "/interrupt", "/upload/image"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusOK {
|
||||
t.Errorf("%s: %d", p, code)
|
||||
}
|
||||
}
|
||||
if code, body := do("GET", "/v1/models", "k"); code != 200 || !strings.Contains(body, `"id":"flux"`) {
|
||||
t.Errorf("/v1/models: %d %s", code, body)
|
||||
}
|
||||
if code, _ := do("GET", "/health", "k"); code != 200 {
|
||||
t.Errorf("/health: %d", code)
|
||||
}
|
||||
for _, g := range got {
|
||||
if strings.Contains(g, "auth=Bearer") {
|
||||
t.Errorf("the key reached ComfyUI: %s", g)
|
||||
}
|
||||
}
|
||||
comfy.Close()
|
||||
if code, _ := do("GET", "/health", "k"); code != http.StatusServiceUnavailable {
|
||||
t.Errorf("/health with ComfyUI down: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheWeightsAreShownByTheirOwnNames(t *testing.T) {
|
||||
cache := t.TempDir()
|
||||
one := filepath.Join(cache, "0123456789abcdef-flux1-schnell-fp8.safetensors")
|
||||
_ = os.WriteFile(one, []byte("w"), 0o644)
|
||||
models := filepath.Join(t.TempDir(), "models")
|
||||
if err := LinkWeights(one, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(models, "flux1-schnell-fp8.safetensors")); err != nil || target != one {
|
||||
t.Fatalf("%q %v", target, err)
|
||||
}
|
||||
|
||||
set := t.TempDir()
|
||||
for _, n := range []string{"flux1-dev.safetensors", "ae.safetensors", "t5xxl_fp8.safetensors", ".complete"} {
|
||||
_ = os.WriteFile(filepath.Join(set, n), []byte("w"), 0o644)
|
||||
}
|
||||
if err := LinkWeights(set, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(models)
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
if strings.Join(names, ",") != "ae.safetensors,flux1-dev.safetensors,t5xxl_fp8.safetensors" {
|
||||
t.Errorf("shown %v: the old weights' link should be gone and the marker not shown", names)
|
||||
}
|
||||
if !strings.Contains(extraModelPaths(models), "checkpoints: .") {
|
||||
t.Error(extraModelPaths(models))
|
||||
}
|
||||
}
|
||||
|
||||
// TestMain lets the test binary stand in for ComfyUI: started by the gate
|
||||
// as `python main.py --listen … --port …`, it answers /system_stats and
|
||||
// says what it was given.
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI") == "1" {
|
||||
fakeComfyUI()
|
||||
return
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func fakeComfyUI() {
|
||||
var port, base string
|
||||
for i, a := range os.Args {
|
||||
switch a {
|
||||
case "--port":
|
||||
port = os.Args[i+1]
|
||||
case "--base-directory":
|
||||
base = os.Args[i+1]
|
||||
}
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(base, "args"), []byte(strings.Join(os.Args[1:], " ")+"\nkey="+os.Getenv("LLMGW_ENGINE_KEY")), 0o644)
|
||||
http.HandleFunc("/system_stats", func(w http.ResponseWriter, _ *http.Request) {
|
||||
// A progress bar's worth of output with every answer: a write that
|
||||
// failed would end this process, as it fails ComfyUI's workflow.
|
||||
if _, err := os.Stdout.Write([]byte(strings.Repeat("progress ", 20000) + "\n")); err != nil {
|
||||
os.Exit(4)
|
||||
}
|
||||
_, _ = io.WriteString(w, "{}")
|
||||
})
|
||||
http.HandleFunc("/quit", func(w http.ResponseWriter, _ *http.Request) { os.Exit(3) })
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI_EXIT") == "1" {
|
||||
time.AfterFunc(200*time.Millisecond, func() { os.Exit(3) })
|
||||
}
|
||||
_ = http.ListenAndServe("127.0.0.1:"+port, nil)
|
||||
}
|
||||
|
||||
func TestTheGateRunsComfyUIAndEndsWithIt(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "flux.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
state := t.TempDir()
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Model: "flux", Weights: weights, State: state,
|
||||
Python: os.Args[0], Dir: t.TempDir(), Args: []string{"--disable-smart-memory"}, Output: io.Discard})
|
||||
}()
|
||||
get := func(path string) int {
|
||||
resp, err := http.Get("http://" + listen + path)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for get("/health") != http.StatusOK {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the gate never said ComfyUI was up")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "args"))
|
||||
for _, want := range []string{"main.py --listen 127.0.0.1 --port", "--base-directory " + state,
|
||||
"--extra-model-paths-config " + filepath.Join(state, "extra_model_paths.yaml"), "--disable-smart-memory"} {
|
||||
if !strings.Contains(string(b), want) {
|
||||
t.Errorf("%q not in what ComfyUI was given: %s", want, b)
|
||||
}
|
||||
}
|
||||
if _, err := os.Lstat(filepath.Join(state, "models", "flux.safetensors")); err != nil {
|
||||
t.Error("the weights were not shown to ComfyUI:", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(state, "custom_nodes")); err != nil {
|
||||
t.Error("ComfyUI lists custom_nodes in its base directory first thing, and it is not there:", err)
|
||||
}
|
||||
if get("/quit") != http.StatusNotFound {
|
||||
t.Error("a path the router never asks for was let through")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Errorf("stopped, the gate said %v", err)
|
||||
}
|
||||
case <-time.After(25 * time.Second):
|
||||
t.Fatal("the gate did not stop")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIExitingEndsTheGate(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI_EXIT", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
err := RunGate(t.Context(), GateConfig{Listen: "127.0.0.1:0", Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: io.Discard})
|
||||
if err == nil || !strings.Contains(err.Error(), "exit status 3") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type brokenWriter struct{}
|
||||
|
||||
func (brokenWriter) Write([]byte) (int, error) { return 0, errors.New("broken pipe") }
|
||||
|
||||
func TestComfyUIKeepsWritingWhenWhatTheGateWritesToIsGone(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: brokenWriter{}})
|
||||
}()
|
||||
ok := 0
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for ok < 20 && time.Now().Before(deadline) {
|
||||
resp, err := http.Get("http://" + listen + "/health")
|
||||
if err == nil {
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
ok++
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
select {
|
||||
case err := <-done:
|
||||
t.Fatalf("the gate ended: %v", err)
|
||||
default:
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if ok < 20 {
|
||||
t.Fatalf("ComfyUI answered %d times; output it could not hand on stopped it", ok)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Custom nodes, put in a replica's own base directory at the commit the
|
||||
// catalog names, their packages installed beside the replica rather than
|
||||
// into ComfyUI's environment: one model's node cannot break another
|
||||
// model's replicas, or the pictures every replica makes.
|
||||
|
||||
// Node is a custom node as the catalog gives it.
|
||||
type Node struct {
|
||||
Repo string `json:"repo"`
|
||||
Commit string `json:"commit"`
|
||||
Requirements []string `json:"requirements,omitempty"`
|
||||
}
|
||||
|
||||
// NodesEnv is where the agent hands the gate a model's custom nodes.
|
||||
const NodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
|
||||
var nodeDir = regexp.MustCompile(`[^A-Za-z0-9._-]`)
|
||||
|
||||
// installNodes puts each node in state/custom_nodes at its commit and its
|
||||
// packages in state/site, and returns the PYTHONPATH ComfyUI is started
|
||||
// with: empty when no node wants a package. Run as the engines' user, in
|
||||
// the replica's own directory, before ComfyUI starts.
|
||||
func installNodes(ctx context.Context, state, python string, nodes []Node, uv string, out func(string)) (string, error) {
|
||||
if len(nodes) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 20*time.Minute)
|
||||
defer cancel()
|
||||
var reqs []string
|
||||
for _, n := range nodes {
|
||||
name := nodeDir.ReplaceAllString(strings.TrimSuffix(path.Base(n.Repo), ".git"), "_")
|
||||
dir := filepath.Join(state, "custom_nodes", name)
|
||||
if head, err := gitOutput(ctx, dir, "rev-parse", "HEAD"); err == nil && head == n.Commit {
|
||||
reqs = append(reqs, n.Requirements...)
|
||||
continue
|
||||
}
|
||||
out(fmt.Sprintf("custom node %s at %s", n.Repo, n.Commit[:12]))
|
||||
_ = os.RemoveAll(dir)
|
||||
for _, args := range [][]string{
|
||||
{"init", "-q", dir},
|
||||
{"-C", dir, "remote", "add", "origin", n.Repo},
|
||||
{"-C", dir, "fetch", "-q", "--depth", "1", "origin", n.Commit},
|
||||
{"-C", dir, "checkout", "-q", "FETCH_HEAD"},
|
||||
} {
|
||||
if b, err := exec.CommandContext(ctx, "git", args...).CombinedOutput(); err != nil { //nolint:gosec // the catalog's repository, checked as https
|
||||
return "", fmt.Errorf("custom node %s at %s: git %s: %v: %s", n.Repo, n.Commit, args[len(args)-2], err, lastLines(b))
|
||||
}
|
||||
}
|
||||
reqs = append(reqs, n.Requirements...)
|
||||
}
|
||||
if len(reqs) == 0 {
|
||||
return "", nil
|
||||
}
|
||||
if uv == "" {
|
||||
return "", fmt.Errorf("custom nodes need %s, and no uv is installed to install them with", strings.Join(reqs, ", "))
|
||||
}
|
||||
site := filepath.Join(state, "site")
|
||||
marker := filepath.Join(site, ".llmgw-requirements")
|
||||
want := strings.Join(reqs, "\n")
|
||||
if b, err := os.ReadFile(marker); err == nil && string(b) == want {
|
||||
return site, nil
|
||||
}
|
||||
out("installing " + strings.Join(reqs, " ") + " for the custom nodes")
|
||||
// What ComfyUI's environment has, as constraints: a package both want is
|
||||
// installed at the version ComfyUI runs, so the replica's copy and
|
||||
// ComfyUI's never disagree.
|
||||
freeze, err := exec.CommandContext(ctx, uv, "pip", "freeze", "--python", python).Output() //nolint:gosec // the agent's own uv
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("reading ComfyUI's packages: %w", err)
|
||||
}
|
||||
constraints := filepath.Join(state, "constraints.txt")
|
||||
if err := os.WriteFile(constraints, withoutNamed(freeze, reqs), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
_ = os.RemoveAll(site)
|
||||
args := append([]string{"pip", "install", "--python", python, "--target", site, "-c", constraints, "--quiet"}, reqs...)
|
||||
cmd := exec.CommandContext(ctx, uv, args...) //nolint:gosec // requirements checked by the catalog as package specs
|
||||
cmd.Env = append(os.Environ(), "UV_NO_CONFIG=1", "UV_CACHE_DIR="+filepath.Join(state, ".uv-cache"))
|
||||
if b, err := cmd.CombinedOutput(); err != nil {
|
||||
return "", fmt.Errorf("installing the custom nodes' packages (%s): %v: %s", strings.Join(reqs, " "), err, lastLines(b))
|
||||
}
|
||||
// The replica's copies come first on ComfyUI's path, but of anything
|
||||
// both have they are the same version, so which is found is the same.
|
||||
if err := os.WriteFile(marker, []byte(want), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return "", err
|
||||
}
|
||||
return site, nil
|
||||
}
|
||||
|
||||
// parseNodes reads the nodes the agent handed over.
|
||||
func parseNodes(s string) ([]Node, error) {
|
||||
if strings.TrimSpace(s) == "" {
|
||||
return nil, nil
|
||||
}
|
||||
var nodes []Node
|
||||
if err := json.Unmarshal([]byte(s), &nodes); err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", NodesEnv, err)
|
||||
}
|
||||
return nodes, nil
|
||||
}
|
||||
|
||||
// findUV is the uv the agent installed engines with.
|
||||
func findUV() string {
|
||||
if m, _ := filepath.Glob("/opt/llmgw/engines/uv/*/*/uv"); len(m) > 0 {
|
||||
return m[len(m)-1]
|
||||
}
|
||||
if p, err := exec.LookPath("uv"); err == nil {
|
||||
return p
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func gitOutput(ctx context.Context, dir string, args ...string) (string, error) {
|
||||
b, err := exec.CommandContext(ctx, "git", append([]string{"-C", dir}, args...)...).Output() //nolint:gosec // fixed arguments
|
||||
return strings.TrimSpace(string(b)), err
|
||||
}
|
||||
|
||||
func lastLines(b []byte) string {
|
||||
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
|
||||
if len(lines) > 4 {
|
||||
lines = lines[len(lines)-4:]
|
||||
}
|
||||
return strings.Join(lines, " | ")
|
||||
}
|
||||
|
||||
// withoutNamed is ComfyUI's packages less those a node names itself: of a
|
||||
// package a node asks for by name, the node's range decides — TripoSR's
|
||||
// checkpoint loads with transformers 4 and not 5 — and of everything else,
|
||||
// what ComfyUI runs. A node that names a package must name a range ComfyUI
|
||||
// can run with too: the copy it gets is first on ComfyUI's path.
|
||||
func withoutNamed(freeze []byte, reqs []string) []byte {
|
||||
named := map[string]bool{}
|
||||
for _, r := range reqs {
|
||||
named[packageName(r)] = true
|
||||
}
|
||||
var out []string
|
||||
for _, line := range strings.Split(string(freeze), "\n") {
|
||||
if strings.TrimSpace(line) == "" || named[packageName(line)] {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return []byte(strings.Join(out, "\n") + "\n")
|
||||
}
|
||||
|
||||
// packageName is a requirement's package, as PyPI compares them.
|
||||
func packageName(spec string) string {
|
||||
s := strings.TrimSpace(spec)
|
||||
if i := strings.IndexAny(s, "<>=!~;[ @"); i >= 0 {
|
||||
s = s[:i]
|
||||
}
|
||||
return strings.NewReplacer("_", "-", ".", "-").Replace(strings.ToLower(s))
|
||||
}
|
||||
@@ -0,0 +1,223 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/net/websocket"
|
||||
)
|
||||
|
||||
// How far along a workflow is, which ComfyUI says only on its websocket and
|
||||
// only to the client that queued the workflow. The gate keeps one websocket
|
||||
// open to its ComfyUI, under a client id of its own, puts that id on every
|
||||
// workflow it lets through, and keeps the last word on each: the node
|
||||
// running and its step out of how many. The router asks at
|
||||
// /progress/{prompt_id} while it waits, and a job says it.
|
||||
|
||||
// Progress is a workflow's state as the gate last heard it.
|
||||
type Progress struct {
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
// State is queued, running, done or failed.
|
||||
State string `json:"state"`
|
||||
// Ahead is how many workflows wait before it, while it is queued.
|
||||
Ahead int `json:"ahead,omitempty"`
|
||||
// Node is the workflow's node running, by its id; NodeClass its kind.
|
||||
Node string `json:"node,omitempty"`
|
||||
NodeClass string `json:"node_class,omitempty"`
|
||||
// Step of Steps is the running node's own count: a sampler's steps.
|
||||
Step int `json:"step,omitempty"`
|
||||
Steps int `json:"steps,omitempty"`
|
||||
// Percent is Step of Steps, where a node counts.
|
||||
Percent int `json:"percent,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
// progressBook is the last word on each of the most recent workflows.
|
||||
type progressBook struct {
|
||||
mu sync.Mutex
|
||||
by map[string]*Progress
|
||||
order []string
|
||||
}
|
||||
|
||||
// keptPrompts bounds the book: a gate serves one replica, which runs one
|
||||
// workflow at a time and queues a few.
|
||||
const keptPrompts = 256
|
||||
|
||||
func newProgressBook() *progressBook { return &progressBook{by: map[string]*Progress{}} }
|
||||
|
||||
func (b *progressBook) update(id string, f func(*Progress)) {
|
||||
if id == "" {
|
||||
return
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
p, ok := b.by[id]
|
||||
if !ok {
|
||||
p = &Progress{PromptID: id, State: "queued"}
|
||||
b.by[id] = p
|
||||
b.order = append(b.order, id)
|
||||
for len(b.order) > keptPrompts {
|
||||
delete(b.by, b.order[0])
|
||||
b.order = b.order[1:]
|
||||
}
|
||||
}
|
||||
f(p)
|
||||
}
|
||||
|
||||
func (b *progressBook) get(id string) (Progress, bool) {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
p, ok := b.by[id]
|
||||
if !ok {
|
||||
return Progress{}, false
|
||||
}
|
||||
return *p, true
|
||||
}
|
||||
|
||||
// read takes one message off ComfyUI's websocket into the book.
|
||||
func (b *progressBook) read(msg []byte) {
|
||||
var m struct {
|
||||
Type string `json:"type"`
|
||||
Data struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Node json.RawMessage `json:"node"`
|
||||
Value int `json:"value"`
|
||||
Max int `json:"max"`
|
||||
Message string `json:"exception_message"`
|
||||
NodeType string `json:"node_type"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if json.Unmarshal(msg, &m) != nil {
|
||||
return
|
||||
}
|
||||
d := m.Data
|
||||
var node string
|
||||
_ = json.Unmarshal(d.Node, &node)
|
||||
switch m.Type {
|
||||
case "execution_start":
|
||||
b.update(d.PromptID, func(p *Progress) { p.State, p.Ahead = "running", 0 })
|
||||
case "executing":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
if node == "" {
|
||||
// No node: the workflow is through.
|
||||
p.State, p.Node, p.Step, p.Steps, p.Percent = "done", "", 0, 0, 100
|
||||
return
|
||||
}
|
||||
p.State, p.Ahead, p.Node, p.Step, p.Steps, p.Percent = "running", 0, node, 0, 0, 0
|
||||
})
|
||||
case "progress":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
p.State, p.Step, p.Steps = "running", d.Value, d.Max
|
||||
if node != "" {
|
||||
p.Node = node
|
||||
}
|
||||
if d.Max > 0 {
|
||||
p.Percent = min(100, d.Value*100/d.Max)
|
||||
}
|
||||
})
|
||||
case "execution_success":
|
||||
b.update(d.PromptID, func(p *Progress) { p.State, p.Percent = "done", 100 })
|
||||
case "execution_error", "execution_interrupted":
|
||||
b.update(d.PromptID, func(p *Progress) {
|
||||
p.State = "failed"
|
||||
p.Error = strings.TrimSpace(d.NodeType + ": " + d.Message)
|
||||
if m.Type == "execution_interrupted" {
|
||||
p.Error = "interrupted"
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// watch keeps a websocket open to the ComfyUI at upstream under clientID,
|
||||
// reading into the book, until ctx ends: opened again when it closes, as it
|
||||
// does whenever ComfyUI restarts.
|
||||
func (b *progressBook) watch(ctx context.Context, upstream *url.URL, clientID string, log *slog.Logger) {
|
||||
ws := "ws://" + upstream.Host + "/ws?clientId=" + url.QueryEscape(clientID)
|
||||
wait := time.Second
|
||||
for ctx.Err() == nil {
|
||||
conn, err := websocket.Dial(ws, "", "http://"+upstream.Host)
|
||||
if err == nil {
|
||||
wait = time.Second
|
||||
go func() { <-ctx.Done(); conn.Close() }()
|
||||
for {
|
||||
var msg []byte
|
||||
if err := websocket.Message.Receive(conn, &msg); err != nil {
|
||||
break
|
||||
}
|
||||
// Binary frames are previews of pictures; only text is news.
|
||||
if len(msg) > 0 && msg[0] == '{' {
|
||||
b.read(msg)
|
||||
}
|
||||
}
|
||||
conn.Close()
|
||||
} else if log != nil {
|
||||
log.Debug("ComfyUI's websocket is not open yet", "err", err)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(wait):
|
||||
}
|
||||
wait = min(wait*2, 30*time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
// withClient is a /prompt body with the gate's client id in it, so the
|
||||
// progress of the workflow it queues comes to the gate's websocket.
|
||||
func withClient(body []byte, clientID string) []byte {
|
||||
var m map[string]json.RawMessage
|
||||
if json.Unmarshal(body, &m) != nil {
|
||||
return body
|
||||
}
|
||||
id, _ := json.Marshal(clientID)
|
||||
m["client_id"] = id
|
||||
out, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return body
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// promptThroughGate forwards a /prompt with the gate's client id, and books
|
||||
// the workflow it queued as queued.
|
||||
func (b *progressBook) promptThroughGate(proxy http.Handler, clientID string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(io.LimitReader(r.Body, 64<<20))
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
body = withClient(body, clientID)
|
||||
r.Body = io.NopCloser(bytes.NewReader(body))
|
||||
r.ContentLength = int64(len(body))
|
||||
rec := &captured{ResponseWriter: w}
|
||||
proxy.ServeHTTP(rec, r)
|
||||
var out struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
}
|
||||
if json.Unmarshal(rec.body.Bytes(), &out) == nil && out.PromptID != "" {
|
||||
b.update(out.PromptID, func(*Progress) {})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// captured passes a response on and keeps a copy of the first of it.
|
||||
type captured struct {
|
||||
http.ResponseWriter
|
||||
body bytes.Buffer
|
||||
}
|
||||
|
||||
func (c *captured) Write(p []byte) (int, error) {
|
||||
if c.body.Len() < 64<<10 {
|
||||
c.body.Write(p)
|
||||
}
|
||||
return c.ResponseWriter.Write(p)
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBookReadsComfyUIsWebsocket(t *testing.T) {
|
||||
b := newProgressBook()
|
||||
for _, m := range []string{
|
||||
`{"type":"status","data":{"status":{"exec_info":{"queue_remaining":1}}}}`,
|
||||
`{"type":"execution_start","data":{"prompt_id":"p1"}}`,
|
||||
`{"type":"executing","data":{"node":"5","prompt_id":"p1"}}`,
|
||||
`{"type":"progress","data":{"value":12,"max":30,"prompt_id":"p1","node":"5"}}`,
|
||||
} {
|
||||
b.read([]byte(m))
|
||||
}
|
||||
p, _ := b.get("p1")
|
||||
if p.State != "running" || p.Node != "5" || p.Step != 12 || p.Steps != 30 || p.Percent != 40 {
|
||||
t.Errorf("%+v", p)
|
||||
}
|
||||
b.read([]byte(`{"type":"executing","data":{"node":null,"prompt_id":"p1"}}`))
|
||||
if p, _ := b.get("p1"); p.State != "done" || p.Percent != 100 {
|
||||
t.Errorf("finished: %+v", p)
|
||||
}
|
||||
b.read([]byte(`{"type":"execution_error","data":{"prompt_id":"p2","node_type":"KSampler","exception_message":"CUDA out of memory"}}`))
|
||||
if p, _ := b.get("p2"); p.State != "failed" || p.Error != "KSampler: CUDA out of memory" {
|
||||
t.Errorf("failed: %+v", p)
|
||||
}
|
||||
for i := range keptPrompts + 10 {
|
||||
b.update(fmt.Sprint("x", i), func(*Progress) {})
|
||||
}
|
||||
if _, ok := b.get("p1"); ok || len(b.by) != keptPrompts {
|
||||
t.Errorf("the book kept %d", len(b.by))
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheGateHearsTheWorkflowsItLetsThrough(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var sent map[string]any
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/prompt" {
|
||||
mu.Lock()
|
||||
_ = json.NewDecoder(r.Body).Decode(&sent)
|
||||
mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"p9"}`)
|
||||
}
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
book := newProgressBook()
|
||||
gate := httptest.NewServer(newGate(u, "", "m", book, "llmgw-gate-1"))
|
||||
defer gate.Close()
|
||||
resp, err := http.Post(gate.URL+"/prompt", "application/json", strings.NewReader(`{"prompt":{},"client_id":"router"}`))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if !strings.Contains(string(b), "p9") {
|
||||
t.Fatalf("the answer did not come through: %s", b)
|
||||
}
|
||||
mu.Lock()
|
||||
if sent["client_id"] != "llmgw-gate-1" {
|
||||
t.Errorf("ComfyUI was sent client %v: its progress would go to nobody listening", sent["client_id"])
|
||||
}
|
||||
mu.Unlock()
|
||||
book.read([]byte(`{"type":"progress","data":{"value":3,"max":4,"prompt_id":"p9","node":"2"}}`))
|
||||
resp, _ = http.Get(gate.URL + "/progress/p9")
|
||||
var p Progress
|
||||
_ = json.NewDecoder(resp.Body).Decode(&p)
|
||||
resp.Body.Close()
|
||||
if p.State != "running" || p.Percent != 75 {
|
||||
t.Errorf("/progress: %+v", p)
|
||||
}
|
||||
if resp, _ := http.Get(gate.URL + "/progress/unknown"); resp.StatusCode != http.StatusNotFound {
|
||||
t.Errorf("an unknown workflow: %d", resp.StatusCode)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWaitingWorkflowSaysHowManyAreAheadAndThenHowFarAlong(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
stage := 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
s := stage
|
||||
mu.Unlock()
|
||||
switch r.URL.Path {
|
||||
case "/prompt":
|
||||
fmt.Fprint(w, `{"prompt_id":"me"}`)
|
||||
case "/queue":
|
||||
if s == 0 {
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"a"]],"queue_pending":[[3,"me"],[2,"b"],[4,"c"]]}`)
|
||||
} else {
|
||||
fmt.Fprint(w, `{"queue_running":[[3,"me"]],"queue_pending":[]}`)
|
||||
}
|
||||
case "/progress/me":
|
||||
fmt.Fprint(w, `{"state":"running","node":"5","step":6,"steps":12,"percent":50}`)
|
||||
case "/history/me":
|
||||
if s < 2 {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"me":{"status":{"status_str":"success","completed":true},"outputs":{"9":{"images":[{"filename":"a.png","type":"temp"}]}}}}`)
|
||||
case "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png")
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
var seen []Progress
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
_, err := c.GenerateWatch(t.Context(), srv.URL, "", map[string]any{"5": map[string]any{"class_type": "KSampler"}},
|
||||
func(p Progress) {
|
||||
mu.Lock()
|
||||
seen = append(seen, p)
|
||||
stage++
|
||||
mu.Unlock()
|
||||
})
|
||||
done <- err
|
||||
}()
|
||||
if err := <-done; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(seen) < 2 || seen[0].State != "queued" || seen[0].Ahead != 2 {
|
||||
t.Fatalf("%+v: queued behind the one running and the one numbered before it, wanted first", seen)
|
||||
}
|
||||
if seen[1].State != "running" || seen[1].Percent != 50 || seen[1].NodeClass != "KSampler" {
|
||||
t.Errorf("then %+v", seen[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodeIsPutAtItsCommit(t *testing.T) {
|
||||
if _, err := exec.LookPath("git"); err != nil {
|
||||
t.Skip("no git")
|
||||
}
|
||||
repo := t.TempDir()
|
||||
run := func(args ...string) string {
|
||||
cmd := exec.Command("git", args...)
|
||||
cmd.Dir = repo
|
||||
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@t", "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@t")
|
||||
b, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("git %v: %v %s", args, err, b)
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
run("init", "-q")
|
||||
run("config", "uploadpack.allowReachableSHA1InWant", "true")
|
||||
_ = os.WriteFile(filepath.Join(repo, "__init__.py"), []byte("v1\n"), 0o644)
|
||||
run("add", ".")
|
||||
run("commit", "-qm", "one")
|
||||
first := run("rev-parse", "HEAD")
|
||||
_ = os.WriteFile(filepath.Join(repo, "__init__.py"), []byte("v2\n"), 0o644)
|
||||
run("commit", "-qam", "two")
|
||||
|
||||
state := t.TempDir()
|
||||
site, err := installNodes(t.Context(), state, "python3", []Node{{Repo: "file://" + repo, Commit: first}}, "", func(string) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "custom_nodes", filepath.Base(repo), "__init__.py"))
|
||||
if string(b) != "v1\n" || site != "" {
|
||||
t.Errorf("the node is %q, site %q: wanted the commit named, not the branch's head", b, site)
|
||||
}
|
||||
if _, err := installNodes(t.Context(), state, "python3", []Node{{Repo: "file://" + repo, Commit: first,
|
||||
Requirements: []string{"trimesh"}}}, "", func(string) {}); err == nil || !strings.Contains(err.Error(), "no uv") {
|
||||
t.Errorf("packages without a uv: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestANodesOwnRangeDecidesThePackagesItNames(t *testing.T) {
|
||||
freeze := []byte("numpy==2.3.1\ntransformers==5.19.0\nhuggingface-hub==1.33.0\nPillow==12.3.0\n")
|
||||
got := string(withoutNamed(freeze, []string{"transformers>=4.50.3,<5", "Huggingface_Hub<1", "trimesh"}))
|
||||
if strings.Contains(got, "transformers") || strings.Contains(got, "huggingface") ||
|
||||
!strings.Contains(got, "numpy==2.3.1") || !strings.Contains(got, "Pillow==12.3.0") {
|
||||
t.Errorf("constraints %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"8": {"class_type": "FluxGuidance", "inputs": {"guidance": "{{guidance|3.5}}", "conditioning": ["2", 0]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|20}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["8", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|4}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -493,6 +493,11 @@ type Personal struct {
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
// PerMesh is the cost of one 3D model (/v1/3d/generations).
|
||||
PerMesh int64 `koanf:"per_mesh"`
|
||||
}
|
||||
|
||||
// Control configures the control plane.
|
||||
@@ -686,6 +691,10 @@ type Band struct {
|
||||
Name string `koanf:"name"`
|
||||
Priority int `koanf:"priority"`
|
||||
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||
// QueueTTL is how long this band's requests may wait for room; the
|
||||
// admission's queue_ttl unless set. Long for a band of bulk work, which
|
||||
// had rather wait behind people than be turned away.
|
||||
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||
}
|
||||
|
||||
// Timeouts bounds each phase of an upstream request.
|
||||
@@ -723,7 +732,10 @@ type Endpoint struct {
|
||||
Models []string `koanf:"models"`
|
||||
// Engine names the inference engine, which decides what its metrics are
|
||||
// called. Empty means vLLM's names.
|
||||
Engine string `koanf:"engine"`
|
||||
Engine string `koanf:"engine"`
|
||||
// Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
// name or its JSON; flux-schnell unless set.
|
||||
Workflow string `koanf:"workflow"`
|
||||
Pool string `koanf:"pool"`
|
||||
Weight float64 `koanf:"weight"`
|
||||
Priority int `koanf:"priority"`
|
||||
|
||||
@@ -70,7 +70,11 @@ type API struct {
|
||||
// guard is the guardrail policies' library and attachments.
|
||||
guard *GuardAdmin
|
||||
// kb is knowledge bases; nil without a store.
|
||||
kb *KBAdmin
|
||||
kb *KBAdmin
|
||||
// tenancy is organizations, keys and people; nil without a store.
|
||||
tenancy TenancyStore
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
settings *Settings
|
||||
// restartAgents tells every connected agent to start again.
|
||||
restartAgents func(reason string) []string
|
||||
@@ -117,6 +121,8 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
a.registerKB(mux)
|
||||
a.registerTenancy(mux)
|
||||
a.registerTrain(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
|
||||
@@ -197,6 +203,9 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/origin", a.setOrigin)
|
||||
mux.HandleFunc("GET /api/v1/plan", a.planModel)
|
||||
mux.HandleFunc("POST /api/v1/metrics/range", a.metricsBatch)
|
||||
mux.HandleFunc("GET /api/v1/client-jobs", a.listJobs)
|
||||
mux.HandleFunc("GET /api/v1/client-jobs/{id}", a.getJob)
|
||||
mux.HandleFunc("POST /api/v1/client-jobs/{id}/cancel", a.cancelJob)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}", a.request)
|
||||
mux.HandleFunc("GET /api/v1/requests/{id}/body", a.requestBody)
|
||||
mux.HandleFunc("POST /api/v1/requests/{id}/replay", a.replayRequest)
|
||||
@@ -1605,6 +1614,8 @@ type RequestView struct {
|
||||
TTFTMs int64 `json:"ttft_ms"`
|
||||
PromptTokens int64 `json:"prompt_tokens"`
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
Images int64 `json:"images,omitempty"`
|
||||
Meshes int64 `json:"meshes,omitempty"`
|
||||
CostMicros int64 `json:"cost_micros"`
|
||||
// What the guardrails did, by name.
|
||||
GuardAction string `json:"guard_action,omitempty"`
|
||||
@@ -1638,7 +1649,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
|
||||
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
|
||||
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, Images: u.Images, Meshes: u.Meshes, CostMicros: u.CostMicros,
|
||||
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
|
||||
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
|
||||
})
|
||||
@@ -1963,32 +1974,9 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "no model "+name)
|
||||
return
|
||||
}
|
||||
base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
|
||||
have := map[string]bool{}
|
||||
for _, ad := range m.Adapters {
|
||||
have[ad.Source] = true
|
||||
}
|
||||
// The file is read only where llama.cpp will load it: a vLLM adapter is
|
||||
// a PEFT directory, which has no header to read, and vLLM itself refuses
|
||||
// one that does not fit the model when it is loaded.
|
||||
ggufs := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
|
||||
for _, ad := range req.Adapters {
|
||||
if have[ad.Source] || !ggufs {
|
||||
continue // already checked when it was attached
|
||||
}
|
||||
h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
|
||||
switch {
|
||||
case err != nil:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
|
||||
return
|
||||
case !h.IsAdapter():
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
|
||||
return
|
||||
case baseErr == nil && h.Architecture != base.Architecture:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
|
||||
ad.Name, h.Architecture, m.Name, base.Architecture))
|
||||
return
|
||||
}
|
||||
if err := checkAdapters(r.Context(), m, req.Adapters); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
|
||||
if err != nil {
|
||||
@@ -2070,3 +2058,36 @@ func multiNodeReadiness(n inventory.Node) string {
|
||||
}
|
||||
return "ready"
|
||||
}
|
||||
|
||||
// checkAdapters reads each adapter a model does not have yet, where
|
||||
// llama.cpp will load it, to refuse one that is not a LoRA adapter for the
|
||||
// model's architecture: such an adapter is a replica that fails to start,
|
||||
// found minutes later on a node.
|
||||
func checkAdapters(ctx context.Context, m catalog.Model, adapters []catalog.Adapter) error {
|
||||
// A vLLM adapter is a PEFT directory, which has no header to read, and
|
||||
// vLLM itself refuses one that does not fit the model when it is loaded.
|
||||
if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
return nil
|
||||
}
|
||||
client := safedial.Client(safedial.ControlPlane, 30*time.Second)
|
||||
base, baseErr := gguf.KindFromURL(ctx, client, m.Source)
|
||||
have := map[string]bool{}
|
||||
for _, ad := range m.Adapters {
|
||||
have[ad.Source] = true
|
||||
}
|
||||
for _, ad := range adapters {
|
||||
if have[ad.Source] {
|
||||
continue // checked when it was attached
|
||||
}
|
||||
h, err := gguf.KindFromURL(ctx, client, ad.Source)
|
||||
switch {
|
||||
case err != nil:
|
||||
return fmt.Errorf("adapter %s: %v", ad.Name, err)
|
||||
case !h.IsAdapter():
|
||||
return fmt.Errorf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source)
|
||||
case baseErr == nil && h.Architecture != base.Architecture:
|
||||
return fmt.Errorf("adapter %s is for %s models and %s is %s", ad.Name, h.Architecture, m.Name, base.Architecture)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -75,11 +75,11 @@ func (a *API) usageExport(w http.ResponseWriter, r *http.Request) {
|
||||
func WriteExportCSV(w io.Writer, rows []store.ExportRow) {
|
||||
c := csv.NewWriter(w)
|
||||
_ = c.Write([]string{"period_start", "org", "team", "project", "user", "key", "model",
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost"})
|
||||
"requests", "errors", "limited", "prompt_tokens", "completion_tokens", "cost_micros", "cost", "images", "meshes"})
|
||||
for _, r := range rows {
|
||||
_ = c.Write([]string{r.Period.Format(time.RFC3339), safe(r.Org), safe(r.Team), safe(r.Project), safe(r.User),
|
||||
r.Key, safe(r.Model), i(r.Requests), i(r.Errors), i(r.Limited), i(r.Prompt), i(r.Completion),
|
||||
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64)})
|
||||
i(r.CostMicros), strconv.FormatFloat(float64(r.CostMicros)/1e6, 'f', 6, 64), i(r.Images), i(r.Meshes)})
|
||||
}
|
||||
c.Flush()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Jobs — 3D models asked for and asked after — as an operator sees them:
|
||||
// every owner's, which a client sees only of its own.
|
||||
|
||||
// jobStore is what jobs need of the store; asked of the usage source by
|
||||
// type assertion, as requestFinder is.
|
||||
type jobStore interface {
|
||||
AllJobs(ctx context.Context, status string, limit int) ([]store.Job, error)
|
||||
JobByID(ctx context.Context, id string) (store.Job, error)
|
||||
CancelJob(ctx context.Context, id string, now time.Time) (store.Job, error)
|
||||
}
|
||||
|
||||
// JobView is one job.
|
||||
type JobView struct {
|
||||
ID string `json:"id"`
|
||||
Owner string `json:"owner"`
|
||||
KeyID string `json:"key_id"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
Model string `json:"model"`
|
||||
Status string `json:"status"`
|
||||
Progress json.RawMessage `json:"progress,omitempty"`
|
||||
OutputFileID string `json:"output_file_id,omitempty"`
|
||||
ContentType string `json:"content_type,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt *time.Time `json:"started_at,omitempty"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
|
||||
func jobView(j store.Job) JobView {
|
||||
v := JobView{ID: j.ID, Owner: j.Owner, KeyID: j.KeyID, Endpoint: j.Endpoint, Model: j.Model, Status: j.Status,
|
||||
OutputFileID: j.OutputFileID, ContentType: j.ContentType, Error: j.Error, CreatedAt: j.CreatedAt}
|
||||
if j.Progress != "" && json.Valid([]byte(j.Progress)) {
|
||||
v.Progress = json.RawMessage(j.Progress)
|
||||
}
|
||||
if !j.StartedAt.IsZero() {
|
||||
v.StartedAt = &j.StartedAt
|
||||
}
|
||||
if !j.FinishedAt.IsZero() {
|
||||
v.FinishedAt = &j.FinishedAt
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (a *API) jobs() (jobStore, bool) {
|
||||
s, ok := a.usage.(jobStore)
|
||||
return s, ok
|
||||
}
|
||||
|
||||
func (a *API) listJobs(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
limit, _ := strconv.Atoi(r.URL.Query().Get("limit"))
|
||||
js, err := s.AllJobs(r.Context(), r.URL.Query().Get("status"), limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := make([]JobView, 0, len(js))
|
||||
for _, j := range js {
|
||||
out = append(out, jobView(j))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) getJob(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
j, err := s.JobByID(r.Context(), r.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no job "+r.PathValue("id"))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, jobView(j))
|
||||
}
|
||||
|
||||
func (a *API) cancelJob(w http.ResponseWriter, r *http.Request) {
|
||||
s, ok := a.jobs()
|
||||
if !ok {
|
||||
writeError(w, http.StatusNotImplemented, "jobs need the store, which this control plane runs without")
|
||||
return
|
||||
}
|
||||
if _, err := s.JobByID(r.Context(), r.PathValue("id")); errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no job "+r.PathValue("id"))
|
||||
return
|
||||
}
|
||||
j, err := s.CancelJob(r.Context(), r.PathValue("id"), time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, jobView(j))
|
||||
}
|
||||
@@ -30,6 +30,9 @@ type KBStore interface {
|
||||
DeleteDocument(ctx context.Context, collection, name string) (bool, error)
|
||||
SearchChunks(ctx context.Context, q store.KBSearch) ([]store.KBHit, error)
|
||||
SetKBDefault(ctx context.Context, d store.KBDefault) error
|
||||
CollectionChunks(ctx context.Context, c store.KBCollection, limit int) ([]store.KBHit, error)
|
||||
CollectionsFor(ctx context.Context, e store.Effective, write bool) ([]store.KBCollection, error)
|
||||
EffectiveFor(ctx context.Context, scope store.Scope, id, org string) (store.Effective, error)
|
||||
KBDefaults(ctx context.Context) ([]store.KBDefault, error)
|
||||
}
|
||||
|
||||
@@ -161,6 +164,7 @@ func (a *API) registerKB(mux Mux) {
|
||||
mux.HandleFunc("DELETE /api/v1/kb/grants", a.kbGrant(false))
|
||||
mux.HandleFunc("POST /api/v1/kb/search", a.kbSearch)
|
||||
mux.HandleFunc("POST /api/v1/kb/defaults", a.kbSetDefault)
|
||||
mux.HandleFunc("POST /api/v1/kb/collections/{id}/dataset", a.kbDataset)
|
||||
}
|
||||
|
||||
// kbReady answers for a gateway without knowledge bases, and says whether
|
||||
@@ -495,7 +499,11 @@ func (a *API) kbSearch(w http.ResponseWriter, r *http.Request) {
|
||||
// Said, not swallowed: a search that quietly lost its meaning half
|
||||
// looks like a collection that does not know the answer.
|
||||
if a.kb.Embed != nil {
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, in.Query)
|
||||
question := in.Query
|
||||
if m, ok := a.ctl.Catalog().Resolve(c.EmbedModel); ok {
|
||||
question = m.QueryPrefix + question
|
||||
}
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, question)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, c.EmbedModel+" did not embed the question: "+err.Error())
|
||||
return
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Training data from a collection (BACKLOG F4): questions a person might
|
||||
// ask of each chunk and the answers it holds, written by a model of the
|
||||
// cluster, for people to read through before an adapter is trained on them.
|
||||
// What an adapter learns well is how the organisation's answers sound; what
|
||||
// they say is still the knowledge base's to give.
|
||||
|
||||
// KBDatasetRequest asks for training data from a collection.
|
||||
type KBDatasetRequest struct {
|
||||
// Model writes the questions and answers: a chat model of the catalog.
|
||||
Model string `json:"model"`
|
||||
// PerChunk is how many pairs each chunk gives; 8 when 0.
|
||||
PerChunk int `json:"per_chunk,omitempty"`
|
||||
// MaxChunks bounds how many chunks are read; 500 when 0.
|
||||
MaxChunks int `json:"max_chunks,omitempty"`
|
||||
}
|
||||
|
||||
// KBDatasetLine is one line of the answer: a pair, or why a chunk gave none.
|
||||
type KBDatasetLine struct {
|
||||
Q string `json:"q,omitempty"`
|
||||
A string `json:"a,omitempty"`
|
||||
Doc string `json:"doc"`
|
||||
Ord int `json:"ord"`
|
||||
// Collection is where the pair came from: an adapter trained on it
|
||||
// belongs to a part of the tree that may read it.
|
||||
Collection string `json:"collection,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
const kbDatasetPrompt = `From the passage of an organisation's document below, write %d different questions an employee might ask, each with a short, correct answer.
|
||||
Rules: use only what the passage says. Phrase the questions differently from each other (formal, everyday, short, indirect). Answers are one or two sentences and keep the passage's numbers exactly. Leave out e-mail addresses and people's names. Write in the passage's own language.
|
||||
Output: one JSON object per line, {"q": "...", "a": "..."}, and nothing else.
|
||||
|
||||
DOCUMENT %s:
|
||||
%s`
|
||||
|
||||
var thinkBlock = regexp.MustCompile(`(?s)<think>.*?</think>`)
|
||||
|
||||
func (a *API) kbDataset(w http.ResponseWriter, r *http.Request) {
|
||||
c, ok := a.kbCollection(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if a.replay == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no router here to ask a model")
|
||||
return
|
||||
}
|
||||
var in KBDatasetRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
m, found := a.ctl.Catalog().Resolve(in.Model)
|
||||
if !found {
|
||||
writeError(w, http.StatusBadRequest, "no model "+in.Model+" in the catalog")
|
||||
return
|
||||
}
|
||||
if m.Capabilities.Embeddings {
|
||||
writeError(w, http.StatusBadRequest, m.Name+" embeds; questions are written by a chat model")
|
||||
return
|
||||
}
|
||||
per := in.PerChunk
|
||||
if per <= 0 || per > 30 {
|
||||
per = 8
|
||||
}
|
||||
limit := in.MaxChunks
|
||||
if limit <= 0 || limit > 5000 {
|
||||
limit = 500
|
||||
}
|
||||
chunks, err := a.kb.Store.CollectionChunks(r.Context(), c, limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/x-ndjson")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
enc := json.NewEncoder(w)
|
||||
flush := func() {
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
}
|
||||
for _, ch := range chunks {
|
||||
if r.Context().Err() != nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"model": m.Name, "temperature": 0.7, "max_tokens": 200 * per,
|
||||
"messages": []map[string]string{{"role": "user", "content": fmt.Sprintf(kbDatasetPrompt, per, ch.Document, ch.Body)}}})
|
||||
status, out, _, err := a.replay(r.Context(), "/v1/chat/completions", m.Name, body)
|
||||
line := KBDatasetLine{Doc: ch.Document, Ord: ch.Ord, Collection: c.ID}
|
||||
if err == nil && status != http.StatusOK {
|
||||
err = fmt.Errorf("%d: %s", status, strings.TrimSpace(string(out)))
|
||||
}
|
||||
if err != nil {
|
||||
line.Error = err.Error()
|
||||
_ = enc.Encode(line)
|
||||
flush()
|
||||
continue
|
||||
}
|
||||
pairs := datasetPairs(out)
|
||||
if len(pairs) == 0 {
|
||||
line.Error = "the model gave no pairs it could read"
|
||||
_ = enc.Encode(line)
|
||||
}
|
||||
for _, p := range pairs {
|
||||
p.Doc, p.Ord, p.Collection = ch.Document, ch.Ord, c.ID
|
||||
_ = enc.Encode(p)
|
||||
}
|
||||
flush()
|
||||
}
|
||||
}
|
||||
|
||||
// datasetPairs reads the {"q","a"} lines out of a chat completion.
|
||||
func datasetPairs(completion []byte) []KBDatasetLine {
|
||||
var res struct {
|
||||
Choices []struct {
|
||||
Message struct {
|
||||
Content string `json:"content"`
|
||||
} `json:"message"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
if json.Unmarshal(completion, &res) != nil || len(res.Choices) == 0 {
|
||||
return nil
|
||||
}
|
||||
text := thinkBlock.ReplaceAllString(res.Choices[0].Message.Content, "")
|
||||
var out []KBDatasetLine
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line = strings.TrimSuffix(strings.TrimSpace(line), ",")
|
||||
if !strings.HasPrefix(line, "{") || !strings.HasSuffix(line, "}") {
|
||||
continue
|
||||
}
|
||||
var p struct{ Q, A string }
|
||||
if json.Unmarshal([]byte(line), &p) == nil && strings.TrimSpace(p.Q) != "" && strings.TrimSpace(p.A) != "" {
|
||||
out = append(out, KBDatasetLine{Q: strings.TrimSpace(p.Q), A: strings.TrimSpace(p.A)})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestPairsAreReadOutOfWhatAModelWrote(t *testing.T) {
|
||||
completion := `{"choices":[{"message":{"content":"<think>let me see</think>\n{\"q\": \"Babalık izni kaç gün?\", \"a\": \"10 iş günü.\"},\nnot json\n{\"q\": \"\", \"a\": \"empty question\"}\n{\"q\": \"Ne zaman?\", \"a\": \"Doğumdan sonra.\"}"}}]}`
|
||||
got := datasetPairs([]byte(completion))
|
||||
if len(got) != 2 || got[0].Q != "Babalık izni kaç gün?" || got[1].A != "Doğumdan sonra." {
|
||||
t.Errorf("pairs %+v", got)
|
||||
}
|
||||
if datasetPairs([]byte(`{"choices":[]}`)) != nil {
|
||||
t.Error("pairs from no choice")
|
||||
}
|
||||
}
|
||||
|
||||
func TestADatasetIsWrittenChunkByChunk(t *testing.T) {
|
||||
dsn := os.Getenv("LLMGW_TEST_POSTGRES")
|
||||
if dsn == "" {
|
||||
t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
|
||||
}
|
||||
ctx := context.Background()
|
||||
s, err := store.OpenPostgres(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
if s.KnowledgeAvailable() != nil {
|
||||
t.Skip("no pgvector")
|
||||
}
|
||||
models := append(kbModels(), catalog.Model{Name: "writer", Source: "/w",
|
||||
Profiles: []catalog.Profile{{Name: "cpu", GPUClass: "h100-80g", TP: 1}}})
|
||||
a, _ := setup(t, models)
|
||||
a.SetKB(&KBAdmin{Store: s})
|
||||
asked := 0
|
||||
a.SetReplayer(func(_ context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) {
|
||||
asked++
|
||||
if path != "/v1/chat/completions" || model != "writer" || !strings.Contains(string(body), "Write in the passage") {
|
||||
t.Errorf("asked %s %s %s", path, model, body)
|
||||
}
|
||||
if strings.Contains(string(body), "bozuk") {
|
||||
return 200, []byte(`{"choices":[{"message":{"content":"sorry"}}]}`), 0, nil
|
||||
}
|
||||
return 200, []byte(`{"choices":[{"message":{"content":"{\"q\":\"Soru?\",\"a\":\"Cevap.\"}"}}]}`), 0, nil
|
||||
})
|
||||
org, _ := s.CreateOrg(ctx, "ds-"+strings.ToLower(store.NewID("")))
|
||||
c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeOrg, OwnerID: org, Name: "d", EmbedModel: "bge-m3", Dims: 3})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, d := range []string{"a.md", "b.md"} {
|
||||
body := "iyi metin"
|
||||
if d == "b.md" {
|
||||
body = "bozuk metin"
|
||||
}
|
||||
if _, _, err := s.PutDocument(ctx, c, store.KBDocument{Name: d, Digest: d}, []store.KBChunk{{Body: body, Embedding: []float32{1, 0, 0}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"writer","per_chunk":3}`)))
|
||||
var lines []KBDatasetLine
|
||||
for _, l := range strings.Split(strings.TrimSpace(rec.Body.String()), "\n") {
|
||||
var x KBDatasetLine
|
||||
_ = json.Unmarshal([]byte(l), &x)
|
||||
lines = append(lines, x)
|
||||
}
|
||||
if rec.Code != 200 || asked != 2 || len(lines) != 2 || lines[0].Q != "Soru?" || lines[0].Doc != "a.md" ||
|
||||
lines[1].Doc != "b.md" || lines[1].Error == "" {
|
||||
t.Errorf("%d asked %d: %+v", rec.Code, asked, lines)
|
||||
}
|
||||
// An embedding model writes no questions.
|
||||
rec = httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"bge-m3"}`)))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("an embedding model: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -62,11 +62,18 @@ var PathParams = map[string]string{
|
||||
"jobs/{id}": "The job's ID, as starting it answered.",
|
||||
"runs/{id}": "The quality run's ID, as starting it answered.",
|
||||
"requests/{id}": "The request's ID: the x-request-id a client was answered with.",
|
||||
"client-jobs/{id}": "The job's ID, job_….",
|
||||
"fabrics/{id}": "The physical fabric's ID, as GET /api/v1/fabrics lists it.",
|
||||
"quotas/{scope}": "The scope: org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT, or their IDs after the colon.",
|
||||
"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
|
||||
"holds/{domain}": "The domain as level:name, rack:r12-07, as `llmgw ctl facility` lists it.",
|
||||
"collections/{id}": "The collection's ID, its name when that is unique, or ORG/NAME.",
|
||||
"orgs/{id}": "The organization's ID or name.",
|
||||
"teams/{id}": "The team's ID.",
|
||||
"projects/{id}": "The project's ID.",
|
||||
"keys/{id}": "The key's ID, as listed (not the key itself).",
|
||||
"users/{id}": "The person's ID or email.",
|
||||
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
|
||||
}
|
||||
|
||||
// Routes is every endpoint Register mounts.
|
||||
@@ -288,10 +295,77 @@ var Routes = []Route{
|
||||
Description: "On an organization, team, project or key; the narrowest one with a default wins. No collections clears it. " +
|
||||
"A request's own X-Llmgw-Knowledge overrides it, and X-Llmgw-Knowledge: none turns it off.",
|
||||
Request: KBDefaultRequest{}, Response: KBDefaultView{}},
|
||||
{Method: "POST", Path: "/api/v1/kb/collections/{id}/dataset", Summary: "Training data from a collection: questions and answers a chat model writes from each chunk",
|
||||
Description: "NDJSON, a line per pair as each chunk is read, or a line with error for a chunk that gave none. For people to correct before an adapter is trained on it.",
|
||||
Request: KBDatasetRequest{}, Response: KBDatasetLine{}},
|
||||
{Method: "POST", Path: "/api/v1/kb/search", Summary: "Try a search of a collection, as an administrator",
|
||||
Description: "No tenancy applies: this is for checking what a collection holds and finds.",
|
||||
Request: KBSearchRequest{}, Response: []KBHitView{}},
|
||||
|
||||
// LoRA adapters trained on the nodes.
|
||||
// Organizations, teams, projects, keys and people.
|
||||
{Method: "GET", Path: "/api/v1/orgs", Summary: "The tenancy tree: every organization, its teams and their projects",
|
||||
Response: []OrgView{}},
|
||||
{Method: "POST", Path: "/api/v1/orgs", Summary: "Add an organization",
|
||||
Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
|
||||
{Method: "PATCH", Path: "/api/v1/orgs/{id}", Summary: "Rename an organization",
|
||||
Request: TenancyChangeRequest{}, Response: TenancyItem{}},
|
||||
{Method: "DELETE", Path: "/api/v1/orgs/{id}", Summary: "Remove an organization that has no teams",
|
||||
Response: map[string]any{}},
|
||||
{Method: "POST", Path: "/api/v1/teams", Summary: "Add a team to an organization",
|
||||
Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
|
||||
{Method: "PATCH", Path: "/api/v1/teams/{id}", Summary: "Rename a team, or move it to another organization",
|
||||
Description: "Everything under it moves too, and from then on inherits the new organization's limits.",
|
||||
Request: TenancyChangeRequest{}, Response: TenancyItem{}},
|
||||
{Method: "DELETE", Path: "/api/v1/teams/{id}", Summary: "Remove a team that has no projects",
|
||||
Response: map[string]any{}},
|
||||
{Method: "POST", Path: "/api/v1/projects", Summary: "Add a project to a team",
|
||||
Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
|
||||
{Method: "PATCH", Path: "/api/v1/projects/{id}", Summary: "Rename a project, or move it to another team",
|
||||
Request: TenancyChangeRequest{}, Response: TenancyItem{}},
|
||||
{Method: "DELETE", Path: "/api/v1/projects/{id}", Summary: "Remove a project with no people, keys or collections",
|
||||
Description: "A project that ever had a key keeps it, revoked, because usage refers to it; rename such a project instead.",
|
||||
Response: map[string]any{}},
|
||||
{Method: "GET", Path: "/api/v1/keys", Summary: "API keys, with their project and owner; never the keys themselves",
|
||||
Query: []Param{{Name: "project", Description: "Only this project's keys: ORG/TEAM/PROJECT or its ID."}},
|
||||
Response: []KeyView{}},
|
||||
{Method: "POST", Path: "/api/v1/keys", Summary: "Issue an API key",
|
||||
Description: "The answer holds the key, once: only its hash is kept. A lifetime longer than a scope above it allows is refused.",
|
||||
Request: KeyCreateRequest{}, Response: KeyCreated{}, Status: http.StatusCreated},
|
||||
{Method: "GET", Path: "/api/v1/keys/{id}", Summary: "One API key", Response: KeyView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/keys/{id}", Summary: "Revoke an API key",
|
||||
Description: "A soft delete: usage refers to it. A gateway that has it cached accepts it until auth.cache_ttl.",
|
||||
Response: map[string]any{}},
|
||||
{Method: "GET", Path: "/api/v1/users", Summary: "People, with where each is a member",
|
||||
Query: []Param{{Name: "state", Description: "Only people in this state: pending, active or disabled."}},
|
||||
Response: []UserView{}},
|
||||
{Method: "GET", Path: "/api/v1/users/{id}", Summary: "One person, with where they are a member", Response: UserView{}},
|
||||
{Method: "POST", Path: "/api/v1/users/{id}/memberships", Summary: "Put a person in an organization, team or project",
|
||||
Description: "Already there, only the role changes.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/users/{id}/memberships", Summary: "Take a person out of an organization, team or project",
|
||||
Description: "Their keys in projects they no longer reach are revoked.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
|
||||
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
|
||||
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
|
||||
Request: TrainRequest{}, Response: TrainJobView{}, Status: http.StatusCreated},
|
||||
{Method: "GET", Path: "/api/v1/train/jobs", Summary: "Training jobs, newest first",
|
||||
Query: []Param{{Name: "limit", Description: "How many; 50 when absent.", Type: "integer"}},
|
||||
Response: []TrainJobView{}},
|
||||
{Method: "GET", Path: "/api/v1/train/jobs/{id}", Summary: "One training job: how it goes, and what it made",
|
||||
Response: TrainJobView{}},
|
||||
{Method: "POST", Path: "/api/v1/train/jobs/{id}/cancel", Summary: "Stop a training job",
|
||||
Response: TrainJobView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/train/jobs/{id}", Summary: "Remove an ended training job and what it made",
|
||||
Response: map[string]any{}},
|
||||
{Method: "POST", Path: "/api/v1/train/jobs/{id}/publish", Summary: "Make what a finished job made the model's adapter",
|
||||
Description: "Kept on the mirror — the GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves — and added to the model's adapters after the same check as POST /api/v1/models/{name}/adapters.",
|
||||
Response: TrainPublishView{}},
|
||||
{Method: "GET", Path: "/api/v1/train/jobs/{id}/artifacts/{kind}", Summary: "What a training job made: peft (a gzipped tar) or gguf",
|
||||
Response: map[string]any{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/agents/restart", Summary: "Have every node's agent start again; their engines keep serving",
|
||||
Response: AgentRestartView{}},
|
||||
|
||||
@@ -324,6 +398,14 @@ var Routes = []Route{
|
||||
{Name: "format", Description: "json for JSON; CSV otherwise."},
|
||||
},
|
||||
Response: []store.ExportRow{}, CSV: true},
|
||||
{Method: "GET", Path: "/api/v1/client-jobs", Summary: "Jobs clients asked for — 3D models — of every owner, newest first",
|
||||
Query: []Param{
|
||||
{Name: "status", Description: "Only jobs in this status: queued, running, cancelling, completed, failed, cancelled, expired."},
|
||||
{Name: "limit", Type: "integer", Description: "The most jobs to answer with; 50 when absent."},
|
||||
},
|
||||
Response: []JobView{}},
|
||||
{Method: "GET", Path: "/api/v1/client-jobs/{id}", Summary: "One job a client asked for", Response: JobView{}},
|
||||
{Method: "POST", Path: "/api/v1/client-jobs/{id}/cancel", Summary: "Stop a job, in the engine's queue too", Response: JobView{}},
|
||||
{Method: "GET", Path: "/api/v1/requests/{id}", Summary: "What one request went through",
|
||||
Response: RequestView{}},
|
||||
{Method: "GET", Path: "/api/v1/requests/{id}/body", Summary: "A kept request and its answer",
|
||||
|
||||
@@ -0,0 +1,658 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The tenancy tree, keys and people's memberships, administered over the
|
||||
// API as `llmgw org`, `keys` and `users` do against the store: so that
|
||||
// they can be run from elsewhere, and kept in Terraform.
|
||||
|
||||
// TenancyStore is the part of the store the tenancy endpoints need.
|
||||
type TenancyStore interface {
|
||||
Tree(ctx context.Context) ([]store.TreeOrg, error)
|
||||
CreateOrg(ctx context.Context, name string) (string, error)
|
||||
CreateTeam(ctx context.Context, orgID, name string) (string, error)
|
||||
CreateProject(ctx context.Context, teamID, name string) (string, error)
|
||||
RenameOrg(ctx context.Context, id, name string) error
|
||||
RenameTeam(ctx context.Context, id, name string) error
|
||||
RenameProject(ctx context.Context, id, name string) error
|
||||
MoveTeam(ctx context.Context, teamID, orgID string) error
|
||||
MoveProject(ctx context.Context, projectID, teamID string) error
|
||||
DeleteOrg(ctx context.Context, id string) error
|
||||
DeleteTeam(ctx context.Context, id string) error
|
||||
DeleteProject(ctx context.Context, id string) error
|
||||
|
||||
AllKeys(ctx context.Context) ([]store.KeyRow, error)
|
||||
KeyByID(ctx context.Context, id string) (store.Key, error)
|
||||
CreateKey(ctx context.Context, k store.Key) error
|
||||
CreateUserKey(ctx context.Context, userID string, k store.Key) error
|
||||
RevokeKey(ctx context.Context, id string, at time.Time) error
|
||||
|
||||
Users(ctx context.Context, state store.UserState) ([]store.User, error)
|
||||
UserByID(ctx context.Context, id string) (store.User, error)
|
||||
UserByEmail(ctx context.Context, email string) (store.User, error)
|
||||
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
|
||||
AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
|
||||
}
|
||||
|
||||
// SetTenancy enables the tenancy endpoints. The quota store must be set
|
||||
// too: names are resolved through it.
|
||||
func (a *API) SetTenancy(t TenancyStore) { a.tenancy = t }
|
||||
|
||||
// OrgView is an organization with what is in it.
|
||||
type OrgView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Members int `json:"members"`
|
||||
Teams []TeamView `json:"teams"`
|
||||
}
|
||||
|
||||
// TeamView is a team with its projects.
|
||||
type TeamView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Org string `json:"org"`
|
||||
Members int `json:"members"`
|
||||
Projects []ProjectView `json:"projects"`
|
||||
}
|
||||
|
||||
// ProjectView is a project, with how many people and keys are in it.
|
||||
type ProjectView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Team string `json:"team"`
|
||||
// Path is ORG/TEAM/PROJECT by name.
|
||||
Path string `json:"path"`
|
||||
Users int `json:"users"`
|
||||
Keys int `json:"keys"`
|
||||
ActiveKeys int `json:"active_keys"`
|
||||
}
|
||||
|
||||
// TenancyCreateRequest adds an organization, a team or a project.
|
||||
type TenancyCreateRequest struct {
|
||||
Name string `json:"name"`
|
||||
// In is the organization a team goes in, or the team a project goes in
|
||||
// (ORG/TEAM), by name or ID; nothing for an organization.
|
||||
In string `json:"in,omitempty"`
|
||||
}
|
||||
|
||||
// TenancyChangeRequest renames an organization, team or project, or moves
|
||||
// a team to another organization or a project to another team.
|
||||
type TenancyChangeRequest struct {
|
||||
Name string `json:"name,omitempty"`
|
||||
// In is where it moves to, by name or ID; not for an organization.
|
||||
In string `json:"in,omitempty"`
|
||||
}
|
||||
|
||||
// TenancyItem is an organization, team or project added or changed.
|
||||
type TenancyItem struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name,omitempty"`
|
||||
// In is the organization a team is in, or the team a project is in, by
|
||||
// ID.
|
||||
In string `json:"in,omitempty"`
|
||||
}
|
||||
|
||||
// KeyView is a key as listed: never the key itself, which is not kept.
|
||||
type KeyView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
// Prefix is the key's first characters, to tell it apart.
|
||||
Prefix string `json:"prefix"`
|
||||
Project string `json:"project,omitempty"`
|
||||
// ProjectPath is ORG/TEAM/PROJECT, or empty for a personal key.
|
||||
ProjectPath string `json:"project_path,omitempty"`
|
||||
User string `json:"user,omitempty"`
|
||||
UserEmail string `json:"user_email,omitempty"`
|
||||
Band string `json:"band,omitempty"`
|
||||
RoutingProfile string `json:"routing_profile,omitempty"`
|
||||
State string `json:"state"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
ExpiresAt time.Time `json:"expires_at,omitzero"`
|
||||
RevokedAt time.Time `json:"revoked_at,omitzero"`
|
||||
LastUsedAt time.Time `json:"last_used_at,omitzero"`
|
||||
}
|
||||
|
||||
// KeyCreateRequest issues a key: a project's, a person's in a project they
|
||||
// are a member of, or a person's own with no project.
|
||||
type KeyCreateRequest struct {
|
||||
Name string `json:"name"`
|
||||
// Project is ORG/TEAM/PROJECT or its ID.
|
||||
Project string `json:"project,omitempty"`
|
||||
// User is the person it belongs to, by ID or email.
|
||||
User string `json:"user,omitempty"`
|
||||
Band string `json:"band,omitempty"`
|
||||
RoutingProfile string `json:"routing_profile,omitempty"`
|
||||
// ExpiresIn is its lifetime, as 720h; none never expires, unless a
|
||||
// scope above it limits how long keys live.
|
||||
ExpiresIn string `json:"expires_in,omitempty"`
|
||||
}
|
||||
|
||||
// KeyCreated is a new key, with the key itself: this answer is the only
|
||||
// time it is given.
|
||||
type KeyCreated struct {
|
||||
KeyView
|
||||
Secret string `json:"secret"`
|
||||
}
|
||||
|
||||
// UserView is a person, with where they are a member.
|
||||
type UserView struct {
|
||||
ID string `json:"id"`
|
||||
Email string `json:"email"`
|
||||
Name string `json:"name,omitempty"`
|
||||
State string `json:"state"`
|
||||
// Role is the person's part in running the gateway, if any: viewer,
|
||||
// operator, billing or admin.
|
||||
Role string `json:"role,omitempty"`
|
||||
Memberships []MembershipView `json:"memberships"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastLoginAt time.Time `json:"last_login_at,omitzero"`
|
||||
}
|
||||
|
||||
// MembershipView is one place a person belongs.
|
||||
type MembershipView struct {
|
||||
Target string `json:"target"`
|
||||
Path string `json:"path,omitempty"`
|
||||
Role string `json:"role"`
|
||||
At time.Time `json:"at"`
|
||||
By string `json:"by,omitempty"`
|
||||
}
|
||||
|
||||
// MembershipRequest puts a person in an organization, team or project, or
|
||||
// with DELETE takes them out.
|
||||
type MembershipRequest struct {
|
||||
// Target is org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT, or the
|
||||
// ID after the colon.
|
||||
Target string `json:"target"`
|
||||
// Role is member or manager; member when absent. Ignored by DELETE.
|
||||
Role string `json:"role,omitempty"`
|
||||
}
|
||||
|
||||
func (a *API) registerTenancy(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/orgs", a.tenancyTree)
|
||||
for _, level := range []store.Scope{store.ScopeOrg, store.ScopeTeam, store.ScopeProject} {
|
||||
path := "/api/v1/" + string(level) + "s"
|
||||
mux.HandleFunc("POST "+path, a.tenancyCreate(level))
|
||||
mux.HandleFunc("PATCH "+path+"/{id}", a.tenancyChange(level))
|
||||
mux.HandleFunc("DELETE "+path+"/{id}", a.tenancyDelete(level))
|
||||
}
|
||||
mux.HandleFunc("GET /api/v1/keys", a.keyList)
|
||||
mux.HandleFunc("POST /api/v1/keys", a.keyCreate)
|
||||
mux.HandleFunc("GET /api/v1/keys/{id}", a.keyGet)
|
||||
mux.HandleFunc("DELETE /api/v1/keys/{id}", a.keyRevoke)
|
||||
mux.HandleFunc("GET /api/v1/users", a.userList)
|
||||
mux.HandleFunc("GET /api/v1/users/{id}", a.userGet)
|
||||
mux.HandleFunc("POST /api/v1/users/{id}/memberships", a.membership(true))
|
||||
mux.HandleFunc("DELETE /api/v1/users/{id}/memberships", a.membership(false))
|
||||
}
|
||||
|
||||
// tenancyReady answers for a control plane without the store, and says
|
||||
// whether it did.
|
||||
func (a *API) tenancyReady(w http.ResponseWriter) bool {
|
||||
if a.tenancy == nil || a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "organizations, keys and people need the store, which this control plane runs without")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// tenancyStatus is the status a store error answers with.
|
||||
func tenancyStatus(err error) int {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
return http.StatusNotFound
|
||||
case errors.Is(err, store.ErrDuplicateName), errors.Is(err, store.ErrNotEmpty):
|
||||
return http.StatusConflict
|
||||
}
|
||||
return http.StatusBadRequest
|
||||
}
|
||||
|
||||
func (a *API) tenancyTree(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
tree, err := a.tenancy.Tree(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []OrgView{}
|
||||
for _, o := range tree {
|
||||
ov := OrgView{ID: o.ID, Name: o.Name, Members: o.Members, Teams: []TeamView{}}
|
||||
for _, t := range o.Teams {
|
||||
tv := TeamView{ID: t.ID, Name: t.Name, Org: o.ID, Members: t.Members, Projects: []ProjectView{}}
|
||||
for _, p := range t.Projects {
|
||||
tv.Projects = append(tv.Projects, ProjectView{ID: p.ID, Name: p.Name, Team: t.ID,
|
||||
Path: o.Name + "/" + t.Name + "/" + p.Name, Users: p.Users, Keys: p.Keys, ActiveKeys: p.ActiveKeys})
|
||||
}
|
||||
ov.Teams = append(ov.Teams, tv)
|
||||
}
|
||||
out = append(out, ov)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// parentOf is the level something at level sits in.
|
||||
func parentOf(level store.Scope) store.Scope {
|
||||
if level == store.ScopeProject {
|
||||
return store.ScopeTeam
|
||||
}
|
||||
return store.ScopeOrg
|
||||
}
|
||||
|
||||
// levelNoun is a level as a sentence says it.
|
||||
func levelNoun(level store.Scope) string {
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
return "an organization"
|
||||
case store.ScopeTeam:
|
||||
return "a team"
|
||||
}
|
||||
return "a project"
|
||||
}
|
||||
|
||||
func (a *API) tenancyCreate(level store.Scope) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in TenancyCreateRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
in.Name = strings.TrimSpace(in.Name)
|
||||
if in.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "a name is needed")
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
var id, parent string
|
||||
var err error
|
||||
if level != store.ScopeOrg {
|
||||
if in.In == "" {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s goes in %s: give in", levelNoun(level), levelNoun(parentOf(level))))
|
||||
return
|
||||
}
|
||||
if _, parent, _, err = a.resolveScope(ctx, string(parentOf(level))+":"+in.In); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
id, err = a.tenancy.CreateOrg(ctx, in.Name)
|
||||
case store.ScopeTeam:
|
||||
id, err = a.tenancy.CreateTeam(ctx, parent, in.Name)
|
||||
default:
|
||||
id, err = a.tenancy.CreateProject(ctx, parent, in.Name)
|
||||
}
|
||||
if errors.Is(err, store.ErrDuplicateName) {
|
||||
writeError(w, http.StatusConflict, fmt.Sprintf("there is already %s named %q there", levelNoun(level), in.Name))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, TenancyItem{ID: id, Name: in.Name, In: parent})
|
||||
}
|
||||
}
|
||||
|
||||
func (a *API) tenancyChange(level store.Scope) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in TenancyChangeRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
if in.Name == "" && in.In == "" {
|
||||
writeError(w, http.StatusBadRequest, "nothing to change: give name, in or both")
|
||||
return
|
||||
}
|
||||
out := TenancyItem{ID: id}
|
||||
if in.In != "" {
|
||||
if level == store.ScopeOrg {
|
||||
writeError(w, http.StatusBadRequest, "an organization is in nothing; only a team or a project moves")
|
||||
return
|
||||
}
|
||||
_, to, _, err := a.resolveScope(ctx, string(parentOf(level))+":"+in.In)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
if level == store.ScopeTeam {
|
||||
err = a.tenancy.MoveTeam(ctx, id, to)
|
||||
} else {
|
||||
err = a.tenancy.MoveProject(ctx, id, to)
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
out.In = to
|
||||
}
|
||||
if name := strings.TrimSpace(in.Name); name != "" {
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
err = a.tenancy.RenameOrg(ctx, id, name)
|
||||
case store.ScopeTeam:
|
||||
err = a.tenancy.RenameTeam(ctx, id, name)
|
||||
default:
|
||||
err = a.tenancy.RenameProject(ctx, id, name)
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
out.Name = name
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *API) tenancyDelete(level store.Scope) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
err = a.tenancy.DeleteOrg(ctx, id)
|
||||
case store.ScopeTeam:
|
||||
err = a.tenancy.DeleteTeam(ctx, id)
|
||||
default:
|
||||
err = a.tenancy.DeleteProject(ctx, id)
|
||||
}
|
||||
if err != nil {
|
||||
// Only what is empty goes: the store says what is still in it.
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"id": id, "deleted": true})
|
||||
}
|
||||
}
|
||||
|
||||
// keyState is a key's state at an instant.
|
||||
func keyState(k store.Key, now time.Time) string {
|
||||
switch {
|
||||
case !k.RevokedAt.IsZero():
|
||||
return "revoked"
|
||||
case !k.ExpiresAt.IsZero() && !now.Before(k.ExpiresAt):
|
||||
return "expired"
|
||||
}
|
||||
return "active"
|
||||
}
|
||||
|
||||
func keyView(k store.Key, path, email string, now time.Time) KeyView {
|
||||
// The store says "Acme / ml / search"; a path given to the API is
|
||||
// written Acme/ml/search, and is answered the same way.
|
||||
path = strings.ReplaceAll(path, " / ", "/")
|
||||
return KeyView{ID: k.ID, Name: k.Name, Prefix: k.Prefix, Project: k.ProjectID, ProjectPath: path,
|
||||
User: k.UserID, UserEmail: email, Band: k.Band, RoutingProfile: k.RoutingProfile, State: keyState(k, now),
|
||||
CreatedAt: k.CreatedAt, ExpiresAt: k.ExpiresAt, RevokedAt: k.RevokedAt, LastUsedAt: k.LastUsedAt}
|
||||
}
|
||||
|
||||
func (a *API) keyList(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
project := ""
|
||||
if p := r.URL.Query().Get("project"); p != "" {
|
||||
var err error
|
||||
if _, project, _, err = a.resolveScope(r.Context(), "project:"+p); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
rows, err := a.tenancy.AllKeys(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
out := []KeyView{}
|
||||
for _, k := range rows {
|
||||
if project == "" || k.ProjectID == project {
|
||||
out = append(out, keyView(k.Key, k.ProjectPath, k.OwnerEmail, now))
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) keyGet(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
rows, err := a.tenancy.AllKeys(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
for _, k := range rows {
|
||||
if k.ID == r.PathValue("id") {
|
||||
writeJSON(w, http.StatusOK, keyView(k.Key, k.ProjectPath, k.OwnerEmail, time.Now()))
|
||||
return
|
||||
}
|
||||
}
|
||||
writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
|
||||
}
|
||||
|
||||
// findUser reads a person by ID or email.
|
||||
func (a *API) findUser(ctx context.Context, ref string) (store.User, error) {
|
||||
ref = strings.TrimSpace(ref)
|
||||
if u, err := a.tenancy.UserByID(ctx, ref); err == nil {
|
||||
return u, nil
|
||||
}
|
||||
if strings.Contains(ref, "@") {
|
||||
if u, err := a.tenancy.UserByEmail(ctx, ref); err == nil {
|
||||
return u, nil
|
||||
}
|
||||
}
|
||||
return store.User{}, fmt.Errorf("%w: no person %q", store.ErrNotFound, ref)
|
||||
}
|
||||
|
||||
func (a *API) keyCreate(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in KeyCreateRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if strings.TrimSpace(in.Name) == "" {
|
||||
writeError(w, http.StatusBadRequest, "a key needs a name")
|
||||
return
|
||||
}
|
||||
if in.Project == "" && in.User == "" {
|
||||
writeError(w, http.StatusBadRequest, "a key belongs to a project, a person, or a person in a project: give project, user or both")
|
||||
return
|
||||
}
|
||||
var owner store.User
|
||||
var err error
|
||||
if in.User != "" {
|
||||
if owner, err = a.findUser(ctx, in.User); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
project, path := "", ""
|
||||
if in.Project != "" {
|
||||
if _, project, path, err = a.resolveScope(ctx, "project:"+in.Project); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
var expires time.Duration
|
||||
if in.ExpiresIn != "" {
|
||||
if expires, err = time.ParseDuration(in.ExpiresIn); err != nil || expires <= 0 {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("expires_in %q is not a positive duration such as 720h", in.ExpiresIn))
|
||||
return
|
||||
}
|
||||
}
|
||||
secret, k, err := store.NewKey(store.NewID("key-"), strings.TrimSpace(in.Name), project)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
k.CreatedAt = time.Now()
|
||||
k.Band, k.RoutingProfile = in.Band, in.RoutingProfile
|
||||
if expires > 0 {
|
||||
k.ExpiresAt = k.CreatedAt.Add(expires)
|
||||
}
|
||||
if owner.ID != "" {
|
||||
err = a.tenancy.CreateUserKey(ctx, owner.ID, k)
|
||||
} else {
|
||||
err = a.tenancy.CreateKey(ctx, k)
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotMember):
|
||||
writeError(w, http.StatusConflict, fmt.Sprintf("%s is not a member of %s; add the membership first", owner.Email, path))
|
||||
return
|
||||
case errors.Is(err, store.ErrKeyTTL):
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
case err != nil:
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
k.UserID = owner.ID
|
||||
writeJSON(w, http.StatusCreated, KeyCreated{KeyView: keyView(k, path, owner.Email, time.Now()), Secret: secret})
|
||||
}
|
||||
|
||||
func (a *API) keyRevoke(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if _, err := a.tenancy.KeyByID(r.Context(), id); err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.RevokeKey(r.Context(), id, time.Now()); err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
// A key held in a gateway's cache keeps working until auth.cache_ttl.
|
||||
writeJSON(w, http.StatusOK, map[string]any{"id": id, "revoked": true})
|
||||
}
|
||||
|
||||
func (a *API) userView(ctx context.Context, u store.User) UserView {
|
||||
role := u.Role
|
||||
if u.IsAdmin {
|
||||
role = "admin"
|
||||
}
|
||||
v := UserView{ID: u.ID, Email: u.Email, Name: u.Name, State: string(u.State), Role: role,
|
||||
Memberships: []MembershipView{}, CreatedAt: u.CreatedAt, LastLoginAt: u.LastLoginAt}
|
||||
if ms, err := a.tenancy.MembershipsOf(ctx, u.ID); err == nil {
|
||||
for _, m := range ms {
|
||||
v.Memberships = append(v.Memberships, MembershipView{Target: string(m.Scope) + ":" + m.ScopeID,
|
||||
Path: m.Path, Role: m.Role, At: m.AddedAt, By: m.AddedBy})
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (a *API) userList(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
us, err := a.tenancy.Users(r.Context(), store.UserState(r.URL.Query().Get("state")))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []UserView{}
|
||||
for _, u := range us {
|
||||
out = append(out, a.userView(r.Context(), u))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) userGet(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
u, err := a.findUser(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.userView(r.Context(), u))
|
||||
}
|
||||
|
||||
func (a *API) membership(add bool) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in MembershipRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
u, err := a.findUser(ctx, r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(ctx, in.Target)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if add {
|
||||
role := in.Role
|
||||
if role == "" {
|
||||
role = store.RoleMember
|
||||
}
|
||||
if role != store.RoleMember && role != store.RoleManager {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
} else {
|
||||
// The person's keys in projects they no longer reach are
|
||||
// revoked with it, by the store.
|
||||
if _, err := a.tenancy.RemoveMembership(ctx, u.ID, scope, id, time.Now()); err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
err = errors.New(u.Email + " is not a member there")
|
||||
}
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.userView(ctx, u))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestTheTenancyTreeKeysAndMembershipsAreAdministered(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, _ := setup(t, testModels())
|
||||
s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
a.SetQuotas(s)
|
||||
a.SetTenancy(s)
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
do := func(method, path, body string, out any) int {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest(method, path, strings.NewReader(body)))
|
||||
if out != nil && rec.Code < 300 {
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
|
||||
t.Fatalf("%s %s: %v (%s)", method, path, err, rec.Body.String())
|
||||
}
|
||||
}
|
||||
if rec.Code >= 300 {
|
||||
t.Logf("%s %s: %d %s", method, path, rec.Code, strings.TrimSpace(rec.Body.String()))
|
||||
}
|
||||
return rec.Code
|
||||
}
|
||||
var made TenancyItem
|
||||
if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, &made); c != http.StatusCreated {
|
||||
t.Fatalf("adding an organization: %d", c)
|
||||
}
|
||||
acme := made.ID
|
||||
if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, nil); c != http.StatusConflict {
|
||||
t.Errorf("a second Acme: %d", c)
|
||||
}
|
||||
if c := do("POST", "/api/v1/teams", `{"name":"hr"}`, nil); c != http.StatusBadRequest {
|
||||
t.Errorf("a team in nothing: %d", c)
|
||||
}
|
||||
do("POST", "/api/v1/teams", `{"name":"hr","in":"Acme"}`, &made)
|
||||
hr := made.ID
|
||||
if c := do("POST", "/api/v1/projects", `{"name":"crm","in":"Acme/hr"}`, &made); c != http.StatusCreated || made.In != hr {
|
||||
t.Fatalf("adding a project: %d %v", c, made)
|
||||
}
|
||||
crm := made.ID
|
||||
do("POST", "/api/v1/orgs", `{"name":"Other"}`, &made)
|
||||
other := made.ID
|
||||
|
||||
// Renamed, and moved with what is in it.
|
||||
if c := do("PATCH", "/api/v1/projects/"+crm, `{"name":"crm2"}`, nil); c != http.StatusOK {
|
||||
t.Errorf("renaming: %d", c)
|
||||
}
|
||||
if c := do("PATCH", "/api/v1/teams/"+hr, `{"in":"Other"}`, nil); c != http.StatusOK {
|
||||
t.Errorf("moving: %d", c)
|
||||
}
|
||||
if c := do("PATCH", "/api/v1/orgs/"+acme, `{"in":"Other"}`, nil); c != http.StatusBadRequest {
|
||||
t.Errorf("moving an organization: %d", c)
|
||||
}
|
||||
var tree []OrgView
|
||||
do("GET", "/api/v1/orgs", "", &tree)
|
||||
if len(tree) != 2 || tree[1].ID != other || len(tree[1].Teams) != 1 ||
|
||||
tree[1].Teams[0].Projects[0].Path != "Other/hr/crm2" {
|
||||
t.Fatalf("the tree is %+v", tree)
|
||||
}
|
||||
|
||||
// A project's key: given once, listed without it.
|
||||
var key KeyCreated
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
|
||||
strings.NewReader(`{"name":"svc","project":"Other/hr/crm2","expires_in":"24h"}`)))
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("issuing a key: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &key)
|
||||
if !strings.HasPrefix(key.Secret, "sk-llmgw-") || key.Project != crm || key.ExpiresAt.IsZero() {
|
||||
t.Errorf("the key issued: %+v", key)
|
||||
}
|
||||
if k, err := s.KeyByHash(ctx, store.HashKey(key.Secret)); err != nil || k.ID != key.ID {
|
||||
t.Errorf("the key does not authenticate: %v", err)
|
||||
}
|
||||
var keys []KeyView
|
||||
do("GET", "/api/v1/keys?project="+crm, "", &keys)
|
||||
if len(keys) != 1 || keys[0].ID != key.ID || keys[0].State != "active" || keys[0].ProjectPath != "Other/hr/crm2" {
|
||||
t.Errorf("the project's keys: %+v", keys)
|
||||
}
|
||||
raw := httptest.NewRecorder()
|
||||
mux.ServeHTTP(raw, httptest.NewRequest("GET", "/api/v1/keys", nil))
|
||||
if strings.Contains(raw.Body.String(), key.Secret) || strings.Contains(raw.Body.String(), "secret") {
|
||||
t.Error("a listing holds the key")
|
||||
}
|
||||
if c := do("POST", "/api/v1/keys", `{"name":"x","project":"Other/hr/crm2","expires_in":"soon"}`, nil); c != http.StatusBadRequest {
|
||||
t.Errorf("a lifetime that is not one: %d", c)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/projects/"+crm, "", nil); c != http.StatusConflict {
|
||||
t.Errorf("removing a project with a key: %d", c)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/keys/"+key.ID, "", nil); c != http.StatusOK {
|
||||
t.Errorf("revoking: %d", c)
|
||||
}
|
||||
var one KeyView
|
||||
do("GET", "/api/v1/keys/"+key.ID, "", &one)
|
||||
if one.State != "revoked" {
|
||||
t.Errorf("after revoking, %+v", one)
|
||||
}
|
||||
|
||||
// A person: a key in a project only as its member, revoked when they
|
||||
// leave it.
|
||||
if err := s.CreateUser(ctx, store.User{ID: "u1", Email: "ayse@example.com", PasswordHash: "x",
|
||||
State: store.UserActive, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c := do("POST", "/api/v1/keys", `{"name":"mine","project":"`+crm+`","user":"ayse@example.com"}`, nil); c != http.StatusConflict {
|
||||
t.Errorf("a key where its person is no member: %d", c)
|
||||
}
|
||||
var u UserView
|
||||
if c := do("POST", "/api/v1/users/ayse@example.com/memberships", `{"target":"project:Other/hr/crm2"}`, &u); c != http.StatusOK ||
|
||||
len(u.Memberships) != 1 || u.Memberships[0].Target != "project:"+crm || u.Memberships[0].Role != "member" {
|
||||
t.Fatalf("joining: %d %+v", c, u)
|
||||
}
|
||||
var mine KeyCreated
|
||||
rec = httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
|
||||
strings.NewReader(`{"name":"mine","project":"`+crm+`","user":"u1"}`)))
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &mine)
|
||||
if rec.Code != http.StatusCreated || mine.User != "u1" {
|
||||
t.Fatalf("a member's key: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, &u); c != http.StatusOK || len(u.Memberships) != 0 {
|
||||
t.Errorf("leaving: %d %+v", c, u)
|
||||
}
|
||||
do("GET", "/api/v1/keys/"+mine.ID, "", &one)
|
||||
if one.State != "revoked" {
|
||||
t.Errorf("the key of a person who left: %+v", one)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, nil); c != http.StatusNotFound {
|
||||
t.Errorf("leaving twice: %d", c)
|
||||
}
|
||||
var people []UserView
|
||||
do("GET", "/api/v1/users", "", &people)
|
||||
if len(people) != 1 || people[0].Email != "ayse@example.com" {
|
||||
t.Errorf("people: %+v", people)
|
||||
}
|
||||
|
||||
// Only what is empty is removed.
|
||||
if c := do("DELETE", "/api/v1/orgs/Acme", "", nil); c != http.StatusOK {
|
||||
t.Errorf("removing the empty Acme by name: %d", c)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/orgs/"+other, "", nil); c != http.StatusConflict {
|
||||
t.Errorf("removing an organization with a team: %d", c)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTenancyNeedsTheStore(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
if c := call(t, a, "GET", "/api/v1/orgs", "", nil); c != http.StatusNotImplemented {
|
||||
t.Errorf("without a store: %d", c)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,559 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Training LoRA adapters (BACKLOG F4), administered: a job names the model
|
||||
// the adapter is for, the weights to train on, the node and how much of a
|
||||
// card it needs, and the pairs; the node trains when it can, and what it
|
||||
// made is kept here to download or publish.
|
||||
|
||||
// TrainStore is where jobs are kept.
|
||||
type TrainStore interface {
|
||||
CreateTrainJob(ctx context.Context, j store.TrainJob) error
|
||||
TrainJobs(ctx context.Context, limit int) ([]store.TrainJob, error)
|
||||
TrainJob(ctx context.Context, id string) (store.TrainJob, error)
|
||||
TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
|
||||
TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
|
||||
DeleteTrainJob(ctx context.Context, id string) error
|
||||
}
|
||||
|
||||
// TrainAdmin is what the training endpoints need.
|
||||
type TrainAdmin struct {
|
||||
Store TrainStore
|
||||
Service *train.Service
|
||||
// Mirror keeps what a job made where the nodes fetch weights; nil
|
||||
// leaves publishing off.
|
||||
Mirror TrainMirror
|
||||
}
|
||||
|
||||
// TrainMirror is the model mirror, as publishing needs it.
|
||||
type TrainMirror interface {
|
||||
Upload(ctx context.Context, model, version, file string, body []byte, source string) (mirror.FetchResponse, error)
|
||||
}
|
||||
|
||||
// TrainPublishView is an adapter published: where the nodes fetch it from,
|
||||
// and what to do for replicas to load it.
|
||||
type TrainPublishView struct {
|
||||
Adapter string `json:"adapter"`
|
||||
Model string `json:"model"`
|
||||
Source string `json:"source"`
|
||||
// Owner is who sees it; empty is every caller of the model.
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// SetTrain enables the training endpoints.
|
||||
func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
|
||||
|
||||
// SetTrainMirror gives publishing the mirror, which is set up after the
|
||||
// training endpoints are.
|
||||
func (a *API) SetTrainMirror(m TrainMirror) {
|
||||
if a.train != nil {
|
||||
a.train.Mirror = m
|
||||
}
|
||||
}
|
||||
|
||||
// TrainRequest queues a job.
|
||||
type TrainRequest struct {
|
||||
// Name is the adapter's: what requests will ask for it by.
|
||||
Name string `json:"name"`
|
||||
// Model is the catalog model the adapter is for.
|
||||
Model string `json:"model"`
|
||||
// BaseRepo and BaseRevision are the weights trained on: the model's own
|
||||
// on its registry, in safetensors (Hugging Face), not a GGUF.
|
||||
BaseRepo string `json:"base_repo"`
|
||||
BaseRevision string `json:"base_revision,omitempty"`
|
||||
// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
|
||||
// what an engine there leaves free.
|
||||
Node string `json:"node"`
|
||||
GPUMemoryMiB int `json:"gpu_memory_mib"`
|
||||
// Owner is who the adapter will belong to: org:, team:, project: (by
|
||||
// path or ID), user: or key:. Only callers there see it. When the
|
||||
// pairs come from one knowledge base, it defaults to that collection's
|
||||
// owner; it must be able to read every collection they come from.
|
||||
Owner string `json:"owner,omitempty"`
|
||||
// Public makes the adapter every caller's its model allows. Refused for
|
||||
// pairs from a knowledge base.
|
||||
Public bool `json:"public,omitempty"`
|
||||
// Data is the pairs, NDJSON of {"q", "a"} (llmgw ctl kb dataset).
|
||||
Data string `json:"data"`
|
||||
// Params tune the trainer: rank, alpha, dropout, epochs, lr, batch,
|
||||
// max_len, seed.
|
||||
Params map[string]any `json:"params,omitempty"`
|
||||
// GGUF also makes the adapter in llama.cpp's format; when absent, it
|
||||
// does when the model is served by llama.cpp.
|
||||
GGUF *bool `json:"gguf,omitempty"`
|
||||
}
|
||||
|
||||
// TrainJobView is a job as listed.
|
||||
type TrainJobView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model"`
|
||||
BaseRepo string `json:"base_repo"`
|
||||
BaseRevision string `json:"base_revision,omitempty"`
|
||||
Node string `json:"node"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
GPUMemoryMiB int `json:"gpu_memory_mib"`
|
||||
Params string `json:"params,omitempty"`
|
||||
GGUF bool `json:"gguf"`
|
||||
Pairs int `json:"pairs"`
|
||||
State string `json:"state"`
|
||||
Step int `json:"step"`
|
||||
Steps int `json:"steps"`
|
||||
Loss float64 `json:"loss"`
|
||||
Message string `json:"message,omitempty"`
|
||||
PeakMiB int `json:"peak_mib"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt time.Time `json:"started_at,omitzero"`
|
||||
FinishedAt time.Time `json:"finished_at,omitzero"`
|
||||
CreatedBy string `json:"created_by,omitempty"`
|
||||
// Artifacts are what it made: peft (a gzipped tar), gguf.
|
||||
Artifacts []string `json:"artifacts,omitempty"`
|
||||
}
|
||||
|
||||
func trainView(j store.TrainJob) TrainJobView {
|
||||
return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
|
||||
Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
|
||||
Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
|
||||
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
|
||||
}
|
||||
|
||||
func (a *API) registerTrain(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/train/jobs", a.trainCreate)
|
||||
mux.HandleFunc("GET /api/v1/train/jobs", a.trainList)
|
||||
mux.HandleFunc("GET /api/v1/train/jobs/{id}", a.trainGet)
|
||||
mux.HandleFunc("POST /api/v1/train/jobs/{id}/cancel", a.trainCancel)
|
||||
mux.HandleFunc("DELETE /api/v1/train/jobs/{id}", a.trainDelete)
|
||||
mux.HandleFunc("GET /api/v1/train/jobs/{id}/artifacts/{kind}", a.trainArtifact)
|
||||
mux.HandleFunc("POST /api/v1/train/jobs/{id}/publish", a.trainPublish)
|
||||
}
|
||||
|
||||
func (a *API) trainReady(w http.ResponseWriter) bool {
|
||||
if a.train == nil {
|
||||
writeError(w, http.StatusNotImplemented, "training needs the store and the nodes' agents, which this control plane runs without")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// trainMaxData bounds a job's pairs.
|
||||
const trainMaxData = 64 << 20
|
||||
|
||||
var adapterName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`)
|
||||
|
||||
// trainParams are the trainer's settings, with what each may be.
|
||||
var trainParams = map[string][2]float64{
|
||||
"rank": {1, 256}, "alpha": {1, 1024}, "dropout": {0, 0.9}, "epochs": {1, 100},
|
||||
"lr": {1e-7, 1e-2}, "batch": {1, 128}, "max_len": {16, 32768}, "seed": {0, 1 << 31},
|
||||
}
|
||||
|
||||
func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.trainReady(w) {
|
||||
return
|
||||
}
|
||||
var in TrainRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, trainMaxData+1<<20)).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
bad := func(format string, args ...any) {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
|
||||
}
|
||||
if !adapterName.MatchString(in.Name) {
|
||||
bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
|
||||
return
|
||||
}
|
||||
m, ok := a.ctl.Catalog().Resolve(in.Model)
|
||||
if !ok {
|
||||
bad("no model %s in the catalog", in.Model)
|
||||
return
|
||||
}
|
||||
if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
|
||||
bad("%s is already a model's or an adapter's name", in.Name)
|
||||
return
|
||||
}
|
||||
if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
|
||||
bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
|
||||
return
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
|
||||
bad("no node %s", in.Node)
|
||||
return
|
||||
}
|
||||
if in.GPUMemoryMiB <= 0 {
|
||||
bad("gpu_memory_mib is what the job needs of a card, in MiB")
|
||||
return
|
||||
}
|
||||
pairs, colls, err := countPairs(in.Data)
|
||||
if err != nil {
|
||||
bad("%v", err)
|
||||
return
|
||||
}
|
||||
owner, err := a.trainOwner(r.Context(), in, colls)
|
||||
if err != nil {
|
||||
bad("%v", err)
|
||||
return
|
||||
}
|
||||
for k, v := range in.Params {
|
||||
lim, known := trainParams[k]
|
||||
f, isNum := v.(float64)
|
||||
if !known || !isNum || f < lim[0] || f > lim[1] {
|
||||
bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
|
||||
return
|
||||
}
|
||||
}
|
||||
params := ""
|
||||
if len(in.Params) > 0 {
|
||||
b, _ := json.Marshal(in.Params)
|
||||
params = string(b)
|
||||
}
|
||||
gguf := false
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == "llamacpp" {
|
||||
gguf = true
|
||||
}
|
||||
}
|
||||
if in.GGUF != nil {
|
||||
gguf = *in.GGUF
|
||||
}
|
||||
j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
|
||||
BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
|
||||
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
|
||||
if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
// At once rather than at the next look, when the node is here.
|
||||
go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
|
||||
j.State = store.TrainQueued
|
||||
writeJSON(w, http.StatusCreated, trainView(j))
|
||||
}
|
||||
|
||||
// trainOwner is who a job's adapter will belong to, as scope:ID: one that
|
||||
// may read every knowledge base the pairs come from. Pairs from a knowledge
|
||||
// base never make an adapter everyone's.
|
||||
func (a *API) trainOwner(ctx context.Context, in TrainRequest, colls []string) (string, error) {
|
||||
if in.Public {
|
||||
if len(colls) > 0 {
|
||||
return "", errors.New("these pairs come from a knowledge base, and an adapter trained on them " +
|
||||
"carries some of what it says: it belongs to a part of the tree that may read it, not to everyone")
|
||||
}
|
||||
if in.Owner != "" {
|
||||
return "", errors.New("an adapter is public or has an owner, not both")
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
if len(colls) > 0 && a.kb == nil {
|
||||
return "", errors.New("the pairs name knowledge bases, which this control plane cannot read")
|
||||
}
|
||||
var found []store.KBCollection
|
||||
for _, id := range colls {
|
||||
c, err := a.kb.Store.Collection(ctx, id)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the pairs come from collection %s, which is not here: %w", id, err)
|
||||
}
|
||||
found = append(found, c)
|
||||
}
|
||||
if in.Owner == "" {
|
||||
if len(found) == 1 {
|
||||
return string(found[0].OwnerScope) + ":" + found[0].OwnerID, nil
|
||||
}
|
||||
return "", errors.New("name the adapter's owner (org:, team:, project:, user: or key:), or make it public " +
|
||||
"when its pairs are not a knowledge base's")
|
||||
}
|
||||
scope, id, err := a.kbTarget(ctx, in.Owner)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, c := range found {
|
||||
e, err := a.kb.Store.EffectiveFor(ctx, scope, id, c.OrgID)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s: %w", in.Owner, err)
|
||||
}
|
||||
readable, err := a.kb.Store.CollectionsFor(ctx, e, false)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !slices.ContainsFunc(readable, func(x store.KBCollection) bool { return x.ID == c.ID }) {
|
||||
return "", fmt.Errorf("%s may not read collection %s, which the pairs come from: "+
|
||||
"an adapter trained on it would tell its callers what it says", in.Owner, c.Name)
|
||||
}
|
||||
}
|
||||
return string(scope) + ":" + id, nil
|
||||
}
|
||||
|
||||
// countPairs checks a job's data: NDJSON of {"q", "a"}, at least one, and
|
||||
// says which knowledge bases the pairs come from.
|
||||
func countPairs(data string) (int, []string, error) {
|
||||
if len(data) > trainMaxData {
|
||||
return 0, nil, fmt.Errorf("the data is over %d MiB", trainMaxData>>20)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
var colls []string
|
||||
n := 0
|
||||
sc := bufio.NewScanner(strings.NewReader(data))
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
line := 0
|
||||
for sc.Scan() {
|
||||
line++
|
||||
b := bytes.TrimSpace(sc.Bytes())
|
||||
if len(b) == 0 {
|
||||
continue
|
||||
}
|
||||
var p struct{ Q, A, Collection string }
|
||||
if err := json.Unmarshal(b, &p); err != nil {
|
||||
return 0, nil, fmt.Errorf("data line %d is not JSON: %v", line, err)
|
||||
}
|
||||
if strings.TrimSpace(p.Q) == "" || strings.TrimSpace(p.A) == "" {
|
||||
return 0, nil, fmt.Errorf("data line %d has no q or no a", line)
|
||||
}
|
||||
if p.Collection != "" && !seen[p.Collection] {
|
||||
seen[p.Collection] = true
|
||||
colls = append(colls, p.Collection)
|
||||
}
|
||||
n++
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
if n == 0 {
|
||||
return 0, nil, errors.New("the data holds no pairs: NDJSON of {\"q\", \"a\"}, as llmgw ctl kb dataset writes")
|
||||
}
|
||||
return n, colls, nil
|
||||
}
|
||||
|
||||
func (a *API) trainList(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.trainReady(w) {
|
||||
return
|
||||
}
|
||||
limit := 50
|
||||
if n, err := strconv.Atoi(r.URL.Query().Get("limit")); err == nil && n > 0 && n <= 500 {
|
||||
limit = n
|
||||
}
|
||||
jobs, err := a.train.Store.TrainJobs(r.Context(), limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []TrainJobView{}
|
||||
for _, j := range jobs {
|
||||
out = append(out, trainView(j))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) trainJob(w http.ResponseWriter, r *http.Request) (store.TrainJob, bool) {
|
||||
if !a.trainReady(w) {
|
||||
return store.TrainJob{}, false
|
||||
}
|
||||
j, err := a.train.Store.TrainJob(r.Context(), r.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no training job "+r.PathValue("id"))
|
||||
return j, false
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return j, false
|
||||
}
|
||||
return j, true
|
||||
}
|
||||
|
||||
func (a *API) trainGet(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
v := trainView(j)
|
||||
v.Artifacts, _ = a.train.Store.TrainArtifactKinds(r.Context(), j.ID)
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
func (a *API) trainCancel(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := a.train.Service.Cancel(r.Context(), j.ID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
j, _ = a.train.Store.TrainJob(r.Context(), j.ID)
|
||||
writeJSON(w, http.StatusOK, trainView(j))
|
||||
}
|
||||
|
||||
func (a *API) trainDelete(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !store.TrainOver(j.State) {
|
||||
writeError(w, http.StatusConflict, "the job has not ended: cancel it first")
|
||||
return
|
||||
}
|
||||
if err := a.train.Store.DeleteTrainJob(r.Context(), j.ID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"id": j.ID, "deleted": true})
|
||||
}
|
||||
|
||||
func (a *API) trainArtifact(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
kind := r.PathValue("kind")
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, kind)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "the job made no "+kind)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
name := j.Name + ".tar.gz"
|
||||
w.Header().Set("Content-Type", "application/gzip")
|
||||
if kind == "gguf" {
|
||||
name = j.Name + ".gguf"
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
}
|
||||
w.Header().Set("Content-Disposition", `attachment; filename="`+name+`"`)
|
||||
w.Header().Set("Content-Length", strconv.Itoa(len(b)))
|
||||
_, _ = w.Write(b)
|
||||
}
|
||||
|
||||
// trainPublish puts what a job made on the mirror and makes it the model's
|
||||
// adapter: the GGUF for a model llama.cpp serves, the PEFT directory for
|
||||
// one vLLM serves.
|
||||
func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if a.train.Mirror == nil {
|
||||
writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
|
||||
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
|
||||
return
|
||||
}
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
|
||||
return
|
||||
}
|
||||
if j.State != store.TrainDone {
|
||||
writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
|
||||
return
|
||||
}
|
||||
m, found := a.ctl.Catalog().Resolve(j.Model)
|
||||
if !found {
|
||||
writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
|
||||
return
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
if ad.Name == j.Name {
|
||||
writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
|
||||
return
|
||||
}
|
||||
}
|
||||
llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
|
||||
source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
|
||||
var url string
|
||||
if llama {
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
|
||||
return
|
||||
}
|
||||
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
|
||||
return
|
||||
}
|
||||
url = res.URL
|
||||
} else {
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusConflict, "the job made no PEFT adapter")
|
||||
return
|
||||
}
|
||||
files, err := untarAll(b)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
for name, data := range files {
|
||||
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
|
||||
return
|
||||
}
|
||||
url = res.URL[:strings.LastIndex(res.URL, "/")+1]
|
||||
}
|
||||
}
|
||||
ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
|
||||
if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
|
||||
if llama {
|
||||
v.Note = "llama.cpp loads adapters when a replica starts: restart " + m.Name + "'s replicas (llmgw ctl restart " +
|
||||
m.Name + ") and ask for model " + j.Name
|
||||
} else {
|
||||
v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// untarAll reads a gzipped tar of a PEFT directory's files.
|
||||
func untarAll(b []byte) (map[string][]byte, error) {
|
||||
gz, err := gzip.NewReader(bytes.NewReader(b))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
out := map[string][]byte{}
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
return out, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if h.Typeflag != tar.TypeReg || strings.ContainsAny(h.Name, "/\\") {
|
||||
continue
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(tr, 4<<30))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[h.Name] = data
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type noAgents struct{}
|
||||
|
||||
func (noAgents) Send(string, *llmgwv1.ControlMessage) error { return nil }
|
||||
func (noAgents) Connected(string) bool { return false }
|
||||
|
||||
func TestATrainingJobIsCheckedBeforeItIsQueued(t *testing.T) {
|
||||
models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
|
||||
a, _ := setup(t, models, gpuNode("cuda01", 1))
|
||||
s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
post := func(body string) (int, string) {
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
|
||||
return rec.Code, rec.Body.String()
|
||||
}
|
||||
good := map[string]any{"name": "tone", "model": "small", "base_repo": "Qwen/Qwen2.5-0.5B-Instruct", "node": "cuda01",
|
||||
"gpu_memory_mib": 2300, "data": "{\"q\":\"a\",\"a\":\"b\"}\n{\"q\":\"c\",\"a\":\"d\"}\n", "params": map[string]any{"epochs": 3},
|
||||
"public": true}
|
||||
with := func(k string, v any) string {
|
||||
m := map[string]any{}
|
||||
for kk, vv := range good {
|
||||
m[kk] = vv
|
||||
}
|
||||
m[k] = v
|
||||
b, _ := json.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
for _, c := range []struct {
|
||||
field string
|
||||
value any
|
||||
says string
|
||||
}{
|
||||
{"name", "bad name", "letters, digits"},
|
||||
{"model", "nope", "no model nope"},
|
||||
{"name", "small", "already a model"},
|
||||
{"base_repo", "Qwen", "owner/name"},
|
||||
{"node", "gone", "no node gone"},
|
||||
{"data", "{\"q\":\"a\"}\n", "no q or no a"},
|
||||
{"data", "", "no pairs"},
|
||||
{"params", map[string]any{"epochs": 0}, "params.epochs"},
|
||||
{"params", map[string]any{"momentum": 0.9}, "params.momentum"},
|
||||
{"public", false, "name the adapter's owner"},
|
||||
{"data", "{\"q\":\"a\",\"a\":\"b\",\"collection\":\"kb_aaaaaaaaaaaaaaaa\"}\n", "from a knowledge base"},
|
||||
} {
|
||||
if code, body := post(with(c.field, c.value)); code != http.StatusBadRequest || !strings.Contains(body, c.says) {
|
||||
t.Errorf("%s=%v: %d %s", c.field, c.value, code, body)
|
||||
}
|
||||
}
|
||||
code, body := post(with("name", "tone"))
|
||||
var v TrainJobView
|
||||
_ = json.Unmarshal([]byte(body), &v)
|
||||
if code != http.StatusCreated || v.Pairs != 2 || !v.GGUF || v.State != "queued" || v.Params != `{"epochs":3}` {
|
||||
t.Fatalf("%d %s", code, body)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("DELETE", "/api/v1/train/jobs/"+v.ID, nil))
|
||||
if rec.Code != http.StatusConflict {
|
||||
t.Errorf("deleting a queued job: %d", rec.Code)
|
||||
}
|
||||
rec = httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/v1/train/jobs/"+v.ID+"/artifacts/peft", nil))
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Errorf("an artifact not made: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterTrainedOnAKnowledgeBaseBelongsToWhoMayReadIt(t *testing.T) {
|
||||
dsn := os.Getenv("LLMGW_TEST_POSTGRES")
|
||||
if dsn == "" {
|
||||
t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
|
||||
}
|
||||
ctx := context.Background()
|
||||
s, err := store.OpenPostgres(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
if s.KnowledgeAvailable() != nil {
|
||||
t.Skip("no pgvector")
|
||||
}
|
||||
models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
|
||||
a, _ := setup(t, models, gpuNode("cuda01", 1))
|
||||
a.SetQuotas(s)
|
||||
a.SetKB(&KBAdmin{Store: s})
|
||||
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
|
||||
sfx := strings.ToLower(store.NewID(""))
|
||||
org, _ := s.CreateOrg(ctx, "Tr"+sfx)
|
||||
hr, _ := s.CreateTeam(ctx, org, "hr")
|
||||
_, _ = s.CreateProject(ctx, hr, "docs")
|
||||
sales, _ := s.CreateTeam(ctx, org, "sales")
|
||||
_, _ = s.CreateProject(ctx, sales, "crm")
|
||||
c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeTeam, OwnerID: hr, Name: "p", EmbedModel: "bge-m3", Dims: 3})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
post := func(extra string) (int, TrainJobView, string) {
|
||||
body := `{"name":"tone-` + store.NewID("") + `","model":"small","base_repo":"Qwen/Q","node":"cuda01","gpu_memory_mib":2000,` +
|
||||
`"data":"{\"q\":\"a\",\"a\":\"b\",\"collection\":\"` + c.ID + `\"}\n"` + extra + `}`
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
|
||||
var v TrainJobView
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &v)
|
||||
return rec.Code, v, rec.Body.String()
|
||||
}
|
||||
if code, v, body := post(""); code != http.StatusCreated || v.Owner != "team:"+hr {
|
||||
t.Errorf("by default: %d %s", code, body)
|
||||
}
|
||||
if code, v, body := post(`,"owner":"project:Tr` + sfx + `/hr/docs"`); code != http.StatusCreated || !strings.HasPrefix(v.Owner, "project:") {
|
||||
t.Errorf("a project of the team: %d %s", code, body)
|
||||
}
|
||||
if code, _, body := post(`,"owner":"project:Tr` + sfx + `/sales/crm"`); code != http.StatusBadRequest || !strings.Contains(body, "may not read") {
|
||||
t.Errorf("a project that cannot read it: %d %s", code, body)
|
||||
}
|
||||
if code, _, body := post(`,"public":true`); code != http.StatusBadRequest || !strings.Contains(body, "not to everyone") {
|
||||
t.Errorf("public: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
@@ -30,7 +30,7 @@ type OrphanView struct {
|
||||
func (a *API) Orphans() []OrphanView {
|
||||
var wanted []string
|
||||
for _, m := range a.ctl.Catalog().Models() {
|
||||
wanted = append(wanted, m.Source)
|
||||
wanted = append(wanted, m.WeightSources()...)
|
||||
// A model's adapters are its weights too; collecting them would
|
||||
// leave replicas that cannot restart.
|
||||
for _, ad := range m.Adapters {
|
||||
|
||||
@@ -82,6 +82,8 @@ type Service struct {
|
||||
// agentSettings is what each node's agent is told to run on; see
|
||||
// SetAgentSettings.
|
||||
agentSettings func(node string) string
|
||||
// train is told how training jobs go; see SetTrain.
|
||||
train TrainHandler
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -368,6 +370,9 @@ func (s *Service) readLoop(ctx context.Context,
|
||||
msg, err := stream.Receive()
|
||||
if err != nil {
|
||||
c.close()
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnNodeGone(nodeID)
|
||||
}
|
||||
if ctx.Err() != nil || errors.Is(err, io.EOF) {
|
||||
s.log.Info("agent disconnected", "node", nodeID)
|
||||
return
|
||||
@@ -505,6 +510,16 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
ch <- m.Upgrade
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainStatus:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainStatus(nodeID, m.TrainStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainArtifact:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainArtifact(nodeID, m.TrainArtifact)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_Logs:
|
||||
s.logMu.Lock()
|
||||
ch, ok := s.logWaits[m.Logs.GetRequestId()]
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// TrainHandler is told how training jobs on the nodes go (BACKLOG F4).
|
||||
type TrainHandler interface {
|
||||
OnTrainStatus(node string, st *llmgwv1.TrainStatus)
|
||||
OnTrainArtifact(node string, a *llmgwv1.TrainArtifact)
|
||||
// OnNodeGone is a node's stream ending: a job running there ended with
|
||||
// it.
|
||||
OnNodeGone(node string)
|
||||
}
|
||||
|
||||
// SetTrain gives the service what training jobs' messages go to.
|
||||
func (s *Service) SetTrain(h TrainHandler) {
|
||||
s.mu.Lock()
|
||||
s.train = h
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) trainHandler() TrainHandler {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.train
|
||||
}
|
||||
|
||||
// Send gives a connected node's agent a command.
|
||||
func (s *Service) Send(node string, msg *llmgwv1.ControlMessage) error { return s.send(node, msg) }
|
||||
|
||||
// Connected is whether a node's agent holds a stream to this control plane.
|
||||
func (s *Service) Connected(node string) bool { return slices.Contains(s.ConnectedNodes(), node) }
|
||||
@@ -7,13 +7,17 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"maps"
|
||||
"net/url"
|
||||
"path"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
@@ -172,6 +176,12 @@ type Profile struct {
|
||||
// 0.10.2. Empty takes whatever version the node has.
|
||||
EngineVersion string `json:"engine_version,omitempty" koanf:"engine_version"`
|
||||
GPUClass string `json:"gpu_class,omitempty" koanf:"gpu_class"`
|
||||
// Source is the weights this profile loads, when not the model's: an
|
||||
// embedding model served by vLLM from its repository on a card and by
|
||||
// llama.cpp from a GGUF on CPUs is one model with two sources. Empty
|
||||
// loads the model's. Updates and canaries move the model's source; a
|
||||
// profile with its own keeps it.
|
||||
Source string `json:"source,omitempty" koanf:"source"`
|
||||
|
||||
TP int `json:"tp,omitempty" koanf:"tp"`
|
||||
PP int `json:"pp,omitempty" koanf:"pp"`
|
||||
@@ -321,6 +331,13 @@ type Model struct {
|
||||
// prompts leave the cluster, so which keys may name it is worth an
|
||||
// allowlist.
|
||||
External *External `json:"external,omitempty" koanf:"external"`
|
||||
// ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
// pictures: the workflow, and the file it loads.
|
||||
ComfyUI *ComfyUI `json:"comfyui,omitempty" koanf:"comfyui"`
|
||||
// License is the terms of what the model makes, for a model whose
|
||||
// output somebody ships: shown with every result and written into a
|
||||
// mesh.
|
||||
License *License `json:"license,omitempty" koanf:"license"`
|
||||
// Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
// for by its own name as though it were a model. Every replica loads
|
||||
// every adapter, applying one only to a request that names it — unless
|
||||
@@ -363,6 +380,13 @@ type Model struct {
|
||||
// Capabilities is what this model can do beyond plain completion.
|
||||
Capabilities Capabilities `json:"capabilities,omitempty" koanf:"capabilities"`
|
||||
|
||||
// QueryPrefix is put before a question an embedding model embeds for a
|
||||
// knowledge-base search; documents are embedded without it. Models
|
||||
// trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
// "Instruct: Given a web search query, retrieve relevant passages that
|
||||
// answer the query\nQuery: " for one.
|
||||
QueryPrefix string `json:"query_prefix,omitempty" koanf:"query_prefix"`
|
||||
|
||||
// Role "guard" is a model that judges the gateway's traffic for the
|
||||
// guardrails rather than serving clients: hidden from /v1/models and the
|
||||
// chat page, refused to any key that names it, critical by default, and
|
||||
@@ -450,6 +474,202 @@ type Capabilities struct {
|
||||
Tools bool `json:"tools,omitempty" koanf:"tools"`
|
||||
Vision bool `json:"vision,omitempty" koanf:"vision"`
|
||||
Embeddings bool `json:"embeddings,omitempty" koanf:"embeddings"`
|
||||
// Images is a model that makes pictures, asked for at
|
||||
// /v1/images/generations. Implied by an external model whose API is
|
||||
// ComfyUI's.
|
||||
Images bool `json:"images,omitempty" koanf:"images"`
|
||||
// Meshes is a model that makes 3D models from a picture, asked for as a
|
||||
// job at /v1/3d/generations. Implied by a ComfyUI model whose output is
|
||||
// meshes.
|
||||
Meshes bool `json:"meshes,omitempty" koanf:"meshes"`
|
||||
}
|
||||
|
||||
// The kinds of thing a ComfyUI model makes.
|
||||
const (
|
||||
OutputPictures = "pictures"
|
||||
OutputMeshes = "meshes"
|
||||
)
|
||||
|
||||
// comfyOutput is what a ComfyUI model makes: pictures unless it says.
|
||||
func (m Model) comfyOutput() string {
|
||||
out := ""
|
||||
switch {
|
||||
case m.External != nil:
|
||||
out = m.External.Output
|
||||
case m.ComfyUI != nil:
|
||||
out = m.ComfyUI.Output
|
||||
}
|
||||
if out == "" {
|
||||
return OutputPictures
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// MakesImages reports whether a model answers /v1/images/generations.
|
||||
func (m Model) MakesImages() bool {
|
||||
return m.Capabilities.Images || m.ServesComfyUI() && m.comfyOutput() == OutputPictures
|
||||
}
|
||||
|
||||
// MakesMeshes reports whether a model answers /v1/3d/generations.
|
||||
func (m Model) MakesMeshes() bool {
|
||||
return m.Capabilities.Meshes || m.ServesComfyUI() && m.comfyOutput() == OutputMeshes
|
||||
}
|
||||
|
||||
// License is the terms what a model makes comes under: said beside every
|
||||
// result, and written into a mesh, which travels without the API.
|
||||
type License struct {
|
||||
// Name is the licence's own: MIT, Apache-2.0.
|
||||
Name string `json:"name" koanf:"name"`
|
||||
URL string `json:"url,omitempty" koanf:"url"`
|
||||
// Notice is what must accompany what was made: attribution, terms on
|
||||
// the output, what the weights were trained on.
|
||||
Notice string `json:"notice,omitempty" koanf:"notice"`
|
||||
}
|
||||
|
||||
// EngineComfyUI is the engine that runs ComfyUI on a node.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// ServesComfyUI reports whether a model is made by ComfyUI, on the
|
||||
// cluster's own nodes or on a server outside it.
|
||||
func (m Model) ServesComfyUI() bool {
|
||||
if m.External != nil {
|
||||
return m.External.API == APIComfyUI
|
||||
}
|
||||
return len(m.Profiles) > 0 && m.Profiles[0].Engine == EngineComfyUI
|
||||
}
|
||||
|
||||
// ComfyUI is a ComfyUI model's workflow and checkpoint.
|
||||
type ComfyUI struct {
|
||||
// Workflow is a built-in's name (flux-schnell, the default, or
|
||||
// flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
// format.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
// source's own name unless set, which a source that is a set of files
|
||||
// must.
|
||||
Checkpoint string `json:"checkpoint,omitempty" koanf:"checkpoint"`
|
||||
// Output is what the workflow makes: pictures (the default), asked for
|
||||
// at /v1/images/generations, or meshes, asked for as a job at
|
||||
// /v1/3d/generations from a picture.
|
||||
Output string `json:"output,omitempty" koanf:"output"`
|
||||
// Nodes are custom nodes the workflow needs, put in each replica's own
|
||||
// custom_nodes at the commit named.
|
||||
Nodes []CustomNode `json:"nodes,omitempty" koanf:"nodes"`
|
||||
}
|
||||
|
||||
// CustomNode is a ComfyUI node from a git repository.
|
||||
type CustomNode struct {
|
||||
// Repo is the repository, https only.
|
||||
Repo string `json:"repo" koanf:"repo"`
|
||||
// Commit is the full commit the node is used at: a branch moves, and a
|
||||
// node that changed under a model is a model that changed unasked.
|
||||
Commit string `json:"commit" koanf:"commit"`
|
||||
// Requirements are the PyPI packages the node needs, by name and version
|
||||
// range only. They are installed beside the replica, never into
|
||||
// ComfyUI's environment: a package named here at the range named, and
|
||||
// every other one at the version ComfyUI already runs. One named here
|
||||
// is first on ComfyUI's path, so its range must suit ComfyUI too.
|
||||
Requirements []string `json:"requirements,omitempty" koanf:"requirements"`
|
||||
}
|
||||
|
||||
// NodesEnv carries a ComfyUI model's custom nodes to the agent, which hands
|
||||
// them to the gate.
|
||||
const NodesEnv = "LLMGW_COMFYUI_NODES"
|
||||
|
||||
var (
|
||||
fullCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
|
||||
packageSpec = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]*(\[[A-Za-z0-9,._-]+\])?\s*([<>=!~]=?\s*[A-Za-z0-9.*+!_-]+\s*,?\s*)*$`)
|
||||
)
|
||||
|
||||
func (n CustomNode) validate(model string) error {
|
||||
u, err := url.Parse(n.Repo)
|
||||
if err != nil || u.Scheme != "https" || u.Host == "" || u.User != nil {
|
||||
return fmt.Errorf("model %q: custom node %q is not an https repository", model, n.Repo)
|
||||
}
|
||||
if !fullCommit.MatchString(n.Commit) {
|
||||
return fmt.Errorf("model %q: custom node %s needs its full commit (40 hex), not %q: a branch moves",
|
||||
model, n.Repo, n.Commit)
|
||||
}
|
||||
for _, r := range n.Requirements {
|
||||
// A package and a version range: an option or a URL would install
|
||||
// from somewhere other than PyPI.
|
||||
if !packageSpec.MatchString(strings.TrimSpace(r)) {
|
||||
return fmt.Errorf("model %q: custom node %s requirement %q is not a package with a version range", model, n.Repo, r)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ComfyWorkflow is what a ComfyUI model's replicas are sent: the workflow
|
||||
// as the catalog names it, and the checkpoint it loads.
|
||||
func (m Model) ComfyWorkflow() (workflow, checkpoint string) {
|
||||
if m.External != nil {
|
||||
return m.External.Workflow, m.External.Model
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
workflow, checkpoint = m.ComfyUI.Workflow, m.ComfyUI.Checkpoint
|
||||
}
|
||||
if checkpoint == "" && !strings.HasSuffix(m.Source, "/") {
|
||||
checkpoint = path.Base(strings.TrimPrefix(m.Source, "hf:"))
|
||||
}
|
||||
return workflow, checkpoint
|
||||
}
|
||||
|
||||
// checkWorkflow refuses a workflow that is neither a built-in nor a file.
|
||||
func checkWorkflow(model, w string) error {
|
||||
// A file is read where the router runs; a name is checked here.
|
||||
if w != "" && !strings.ContainsAny(w, "/.") && !slices.Contains(comfy.Builtins(), w) {
|
||||
return fmt.Errorf("model %q: no built-in workflow %q (there are %s; or name a JSON file)",
|
||||
model, w, strings.Join(comfy.Builtins(), ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m Model) validateComfyUI() error {
|
||||
comfyProfiles := 0
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == EngineComfyUI {
|
||||
comfyProfiles++
|
||||
}
|
||||
}
|
||||
if comfyProfiles == 0 {
|
||||
if m.ComfyUI != nil {
|
||||
return fmt.Errorf("model %q: comfyui: is for a model whose profiles run engine comfyui", m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if comfyProfiles != len(m.Profiles) {
|
||||
return fmt.Errorf("model %q: every profile runs engine comfyui or none does: "+
|
||||
"a request for pictures cannot be answered by an engine that writes text", m.Name)
|
||||
}
|
||||
if len(m.Adapters) > 0 {
|
||||
return fmt.Errorf("model %q: ComfyUI serves no adapters", m.Name)
|
||||
}
|
||||
var w string
|
||||
if m.ComfyUI != nil {
|
||||
w = m.ComfyUI.Workflow
|
||||
}
|
||||
if err := checkWorkflow(m.Name, w); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
for _, n := range m.ComfyUI.Nodes {
|
||||
if err := n.validate(m.Name); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if out := m.comfyOutput(); out != OutputPictures && out != OutputMeshes {
|
||||
return fmt.Errorf("model %q: comfyui.output %q is not pictures or meshes", m.Name, out)
|
||||
}
|
||||
if m.comfyOutput() == OutputMeshes && w == "" {
|
||||
return fmt.Errorf("model %q: a model that makes meshes names its workflow: no built-in makes them", m.Name)
|
||||
}
|
||||
if _, ckpt := m.ComfyWorkflow(); ckpt == "" {
|
||||
return fmt.Errorf("model %q: its source is a set of files, so comfyui.checkpoint must name the one the workflow loads",
|
||||
m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Catalog is the set of models the cluster should serve.
|
||||
@@ -658,9 +878,15 @@ func (m Model) normalize() Model {
|
||||
// repository it names: the engine downloads a repository id itself,
|
||||
// where the page was fetched as one file of HTML and failed at start.
|
||||
if repo, ok := repositoryPage(m.Source); ok && len(m.Profiles) > 0 &&
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
!slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" }) {
|
||||
m.Source = repo
|
||||
}
|
||||
for i := range m.Profiles {
|
||||
p := &m.Profiles[i]
|
||||
if repo, ok := repositoryPage(p.Source); ok && p.Engine != "llamacpp" {
|
||||
p.Source = repo
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
@@ -671,15 +897,26 @@ func (m Model) validate() error {
|
||||
if m.Source == "" {
|
||||
return fmt.Errorf("model %q: source is required", m.Name)
|
||||
}
|
||||
if repo, ok := repositoryPage(m.Source); ok {
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a model
|
||||
// llama.cpp runs is left here, which needs a file.
|
||||
// A page, not weights: a node fetches the HTML as one file and the
|
||||
// engine fails at start, on a path that looks right. For vLLM it was
|
||||
// read as the repository id when the model was made; only a source
|
||||
// llama.cpp loads is left here, which needs a file.
|
||||
if repo, ok := repositoryPage(m.Source); ok && (len(m.Profiles) == 0 ||
|
||||
slices.ContainsFunc(m.Profiles, func(p Profile) bool { return p.Engine == "llamacpp" && p.Source == "" })) {
|
||||
return fmt.Errorf("model %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, m.Source, repo)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if repo, ok := repositoryPage(p.Source); ok {
|
||||
return fmt.Errorf("model %q profile %q: source %s is a Hugging Face repository page, not a file; "+
|
||||
"llama.cpp needs a .gguf: mirror one and use its source (repository %s)",
|
||||
m.Name, p.Name, p.Source, repo)
|
||||
}
|
||||
if p.Source != "" && m.External != nil {
|
||||
return fmt.Errorf("model %q profile %q: an external model has no profiles to give a source", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if err := validateAdapters(m); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -690,6 +927,9 @@ func (m Model) validate() error {
|
||||
} else if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if err := m.validateComfyUI(); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
@@ -817,7 +1057,7 @@ func (m Model) validate() error {
|
||||
return fmt.Errorf("model %q profile %q: trtllm runs a replica on one node; "+
|
||||
"use vllm or sglang for a %d-node replica", m.Name, p.Name, p.Nodes)
|
||||
}
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.Source, "/")), ".gguf") {
|
||||
if strings.HasSuffix(strings.ToLower(strings.TrimSuffix(m.SourceOf(p), "/")), ".gguf") {
|
||||
return fmt.Errorf("model %q profile %q: trtllm loads a Hugging Face checkpoint, not a .gguf; "+
|
||||
"serve a GGUF with llamacpp", m.Name, p.Name)
|
||||
}
|
||||
@@ -903,20 +1143,53 @@ func (m Model) PlacementProfiles() []placement.Profile {
|
||||
CPUCores: p.CPUCores,
|
||||
PowerWatts: p.PowerWatts,
|
||||
KeepWithin: facility.Level(strings.ToLower(strings.TrimSpace(p.KeepWithin))),
|
||||
// The model's source, so placement can prefer nodes that already
|
||||
// The profile's source, so placement can prefer nodes that already
|
||||
// hold the weights rather than one that must first move them.
|
||||
Source: m.Source,
|
||||
Source: m.SourceOf(p),
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// SourceOf is the weights a profile loads: its own, or the model's.
|
||||
func (m Model) SourceOf(p Profile) string {
|
||||
if p.Source != "" {
|
||||
return p.Source
|
||||
}
|
||||
return m.Source
|
||||
}
|
||||
|
||||
// WeightSources are every source a model's replicas may load: the model's
|
||||
// and each profile's of its own, once each. What a node must not have
|
||||
// collected from under it.
|
||||
func (m Model) WeightSources() []string {
|
||||
out := []string{}
|
||||
if m.Source != "" {
|
||||
out = append(out, m.Source)
|
||||
}
|
||||
for _, p := range m.Profiles {
|
||||
if p.Source != "" && !slices.Contains(out, p.Source) {
|
||||
out = append(out, p.Source)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// EngineProfile converts a catalog profile into the engine adapter's view.
|
||||
func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
for _, p := range m.Profiles {
|
||||
if p.Name != name {
|
||||
continue
|
||||
}
|
||||
env := p.Env
|
||||
if p.Engine == EngineComfyUI && m.ComfyUI != nil && len(m.ComfyUI.Nodes) > 0 {
|
||||
nodes, _ := json.Marshal(m.ComfyUI.Nodes)
|
||||
env = maps.Clone(p.Env)
|
||||
if env == nil {
|
||||
env = map[string]string{}
|
||||
}
|
||||
env[NodesEnv] = string(nodes)
|
||||
}
|
||||
return engines.Profile{
|
||||
Name: p.Name,
|
||||
Engine: p.Engine,
|
||||
@@ -926,8 +1199,8 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
GPUsPerNode: p.GPUsPerNode,
|
||||
Nodes: p.Nodes,
|
||||
Args: slices.Clone(p.Args),
|
||||
Env: p.Env,
|
||||
ModelPath: m.Source,
|
||||
Env: env,
|
||||
ModelPath: m.SourceOf(p),
|
||||
ServedModelName: m.Name,
|
||||
Adapters: m.engineAdapters(),
|
||||
KVEvents: p.KVEvents,
|
||||
@@ -1094,8 +1367,22 @@ type External struct {
|
||||
// than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
// goes with the peer's own list, and is sent only while the peer is up.
|
||||
Peer string `json:"peer,omitempty" koanf:"peer"`
|
||||
// API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
// a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
// sent the workflow below with the request's prompt in it. Model is then
|
||||
// the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
API string `json:"api,omitempty" koanf:"api"`
|
||||
// Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
// (flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
// plane.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Output is what it makes: pictures, or meshes (see ComfyUI.Output).
|
||||
Output string `json:"output,omitempty" koanf:"output"`
|
||||
}
|
||||
|
||||
// APIComfyUI is External.API for a ComfyUI server.
|
||||
const APIComfyUI = "comfyui"
|
||||
|
||||
// IsPeer reports whether a model is offered by a federated cluster.
|
||||
func (m Model) IsPeer() bool { return m.External != nil && m.External.Peer != "" }
|
||||
|
||||
@@ -1110,6 +1397,27 @@ func (e *External) validate(model string, profiles int) error {
|
||||
if strings.TrimSpace(e.Model) == "" {
|
||||
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
|
||||
}
|
||||
switch e.API {
|
||||
case "", "openai":
|
||||
if e.Workflow != "" || e.Output != "" {
|
||||
return fmt.Errorf("model %q: a workflow and an output are for api: comfyui", model)
|
||||
}
|
||||
case APIComfyUI:
|
||||
if e.Peer != "" {
|
||||
return fmt.Errorf("model %q: a federated model speaks OpenAI's API, not ComfyUI's", model)
|
||||
}
|
||||
if err := checkWorkflow(model, e.Workflow); err != nil {
|
||||
return err
|
||||
}
|
||||
if e.Output != "" && e.Output != OutputPictures && e.Output != OutputMeshes {
|
||||
return fmt.Errorf("model %q: external output %q is not pictures or meshes", model, e.Output)
|
||||
}
|
||||
if e.Output == OutputMeshes && e.Workflow == "" {
|
||||
return fmt.Errorf("model %q: a model that makes meshes names its workflow: no built-in makes them", model)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("model %q: external api %q is not openai or comfyui", model, e.API)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1127,6 +1435,27 @@ type Adapter struct {
|
||||
Source string `json:"source" koanf:"source"`
|
||||
// Scale is how strongly it is applied; zero takes 1.
|
||||
Scale float64 `json:"scale,omitempty" koanf:"scale"`
|
||||
// Owner is the part of the tenancy tree the adapter belongs to —
|
||||
// org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
|
||||
// it see it: to anyone else it is a model that does not exist. An
|
||||
// adapter trained on an organisation's documents carries some of what
|
||||
// they say in its weights. Empty is every caller the model allows.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
}
|
||||
|
||||
// ValidAdapterOwner checks an adapter's owner is scope:ID.
|
||||
func ValidAdapterOwner(owner string) error {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
scope, id, ok := strings.Cut(owner, ":")
|
||||
switch {
|
||||
case !ok || id == "":
|
||||
return fmt.Errorf("an adapter's owner is scope:ID, %q is not", owner)
|
||||
case scope != "org" && scope != "team" && scope != "project" && scope != "user" && scope != "key":
|
||||
return fmt.Errorf("an adapter's owner is an org, team, project, user or key, not %q", scope)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EffectiveScale is the scale an adapter is applied at.
|
||||
@@ -1175,6 +1504,9 @@ func validateAdapters(m Model) error {
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, a := range m.Adapters {
|
||||
if err := ValidAdapterOwner(a.Owner); err != nil {
|
||||
return fmt.Errorf("model %q adapter %q: %w", m.Name, a.Name, err)
|
||||
}
|
||||
switch {
|
||||
case a.Name == "" || a.Source == "":
|
||||
return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func comfyModel(source string, c *ComfyUI, profiles ...Profile) Model {
|
||||
if len(profiles) == 0 {
|
||||
profiles = []Profile{{Name: "gpu", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1}}
|
||||
}
|
||||
return Model{Name: "flux", Source: source, ComfyUI: c, Profiles: profiles}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelMakesPicturesFromItsSourcesFile(t *testing.T) {
|
||||
cat, err := New([]Model{comfyModel("hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors", nil)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
if !m.MakesImages() || !m.ServesComfyUI() {
|
||||
t.Error("a ComfyUI model does not say it makes pictures")
|
||||
}
|
||||
if w, ckpt := m.ComfyWorkflow(); w != "" || ckpt != "flux1-schnell-fp8.safetensors" {
|
||||
t.Errorf("workflow %q, checkpoint %q", w, ckpt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsRefusedWhatItCannotDo(t *testing.T) {
|
||||
for name, c := range map[string]struct {
|
||||
m Model
|
||||
want string
|
||||
}{
|
||||
"a set without a checkpoint": {comfyModel("https://mirror/flux-dev/v1/", nil), "comfyui.checkpoint"},
|
||||
"an unknown workflow": {comfyModel("/w/f.safetensors", &ComfyUI{Workflow: "sdxl"}), "no built-in workflow"},
|
||||
"mixed engines": {comfyModel("/w/f.safetensors", nil,
|
||||
Profile{Name: "a", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1},
|
||||
Profile{Name: "b", Engine: "vllm", GPUClass: "6000-24g", TP: 1}), "every profile"},
|
||||
"comfyui settings without the engine": {Model{Name: "x", Source: "/w/x", ComfyUI: &ComfyUI{},
|
||||
Profiles: []Profile{{Name: "a", Engine: "vllm", GPUClass: "6000-24g", TP: 1}}}, "engine comfyui"},
|
||||
"no card": {comfyModel("/w/f.safetensors", nil, Profile{Name: "a", Engine: EngineComfyUI, TP: 1}), "gpu_class"},
|
||||
} {
|
||||
if _, err := New([]Model{c.m}); err == nil || !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%s: %v", name, err)
|
||||
}
|
||||
}
|
||||
ok := comfyModel("https://mirror/flux-dev/v1/", &ComfyUI{Workflow: "flux-dev", Checkpoint: "flux1-dev-fp8.safetensors"})
|
||||
if _, err := New([]Model{ok}); err != nil {
|
||||
t.Errorf("a set with its checkpoint named was refused: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestACustomNodeIsPinnedAndItsPackagesArePackages(t *testing.T) {
|
||||
node := func(n CustomNode) Model {
|
||||
return comfyModel("hf:stabilityai/TripoSR/model.ckpt", &ComfyUI{Workflow: "w.json", Output: OutputMeshes,
|
||||
Nodes: []CustomNode{n}})
|
||||
}
|
||||
ok := CustomNode{Repo: "https://github.com/flowtyone/ComfyUI-Flowty-TripoSR",
|
||||
Commit: "0123456789abcdef0123456789abcdef01234567", Requirements: []string{"omegaconf", "trimesh>=4,<5", "scikit-image"}}
|
||||
cat, err := New([]Model{node(ok)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
p, _ := m.EngineProfile("gpu")
|
||||
if !strings.Contains(p.Env[NodesEnv], "ComfyUI-Flowty-TripoSR") || !m.MakesMeshes() || m.MakesImages() {
|
||||
t.Errorf("env %v meshes %v pictures %v", p.Env, m.MakesMeshes(), m.MakesImages())
|
||||
}
|
||||
for name, n := range map[string]CustomNode{
|
||||
"a branch": {Repo: ok.Repo, Commit: "main"},
|
||||
"http": {Repo: "http://example.com/n", Commit: ok.Commit},
|
||||
"an option": {Repo: ok.Repo, Commit: ok.Commit, Requirements: []string{"--index-url=https://evil"}},
|
||||
"a url": {Repo: ok.Repo, Commit: ok.Commit, Requirements: []string{"x @ https://evil/x.whl"}},
|
||||
} {
|
||||
if _, err := New([]Model{node(n)}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
noWorkflow := comfyModel("hf:a/b/c.ckpt", &ComfyUI{Output: OutputMeshes})
|
||||
if _, err := New([]Model{noWorkflow}); err == nil || !strings.Contains(err.Error(), "names its workflow") {
|
||||
t.Errorf("a mesh model without a workflow: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// embedModel is an embedding model the way it was wanted on the PoC: vLLM
|
||||
// on the card from the repository, llama.cpp on CPUs from a GGUF.
|
||||
func embedModel() Model {
|
||||
return Model{Name: "qwen3-embed-4b", Source: "Qwen/Qwen3-Embedding-4B",
|
||||
Profiles: []Profile{
|
||||
{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1},
|
||||
{Name: "cpu", Engine: "llamacpp", Source: "hf:Qwen/Qwen3-Embedding-4B-GGUF/Qwen3-Embedding-4B-Q8_0.gguf"},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestEachProfileLoadsItsOwnSource(t *testing.T) {
|
||||
// The catalog had one source a model, so serving an embedding model by
|
||||
// vLLM on a card and by llama.cpp on CPUs meant swapping the entry by
|
||||
// hand halfway through an import.
|
||||
cat, err := New([]Model{embedModel()})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
gpu, _ := m.EngineProfile("gpu")
|
||||
cpu, _ := m.EngineProfile("cpu")
|
||||
if gpu.ModelPath != "Qwen/Qwen3-Embedding-4B" || !strings.HasSuffix(cpu.ModelPath, "Q8_0.gguf") {
|
||||
t.Errorf("gpu loads %q, cpu %q", gpu.ModelPath, cpu.ModelPath)
|
||||
}
|
||||
for _, p := range m.PlacementProfiles() {
|
||||
if want := map[string]string{"gpu": gpu.ModelPath, "cpu": cpu.ModelPath}[p.Name]; p.Source != want {
|
||||
t.Errorf("placement for %s looks for %q, want %q", p.Name, p.Source, want)
|
||||
}
|
||||
}
|
||||
if got := m.WeightSources(); len(got) != 2 || !slices.Contains(got, cpu.ModelPath) {
|
||||
t.Errorf("weight sources %v: a profile's own must not be collected", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileWithoutASourceKeepsItsFingerprint(t *testing.T) {
|
||||
// Replicas are compared with their profile by fingerprint; a field added
|
||||
// to every profile must not make every running replica look stale.
|
||||
p := Profile{Name: "gpu", Engine: "vllm", GPUClass: "6000-24g", TP: 1}
|
||||
before := p.Fingerprint()
|
||||
if strings.Contains(mustJSON(t, p), "source") {
|
||||
t.Error("an empty source is written into the profile")
|
||||
}
|
||||
p.Source = "Qwen/Other"
|
||||
if p.Fingerprint() == before {
|
||||
t.Error("a profile whose source changed reads as unchanged")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProfileSourceIsCheckedAsTheModelsIs(t *testing.T) {
|
||||
page := embedModel()
|
||||
page.Profiles[1].Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B-GGUF"
|
||||
if _, err := New([]Model{page}); err == nil || !strings.Contains(err.Error(), `profile "cpu"`) {
|
||||
t.Errorf("a repository page for llama.cpp: %v", err)
|
||||
}
|
||||
// The model's own source may be a page vLLM reads as a repository, now
|
||||
// that the llama.cpp profile loads something else.
|
||||
repo := embedModel()
|
||||
repo.Source = "https://huggingface.co/Qwen/Qwen3-Embedding-4B"
|
||||
cat, err := New([]Model{repo})
|
||||
if err != nil {
|
||||
t.Fatalf("a repository page for the vLLM profile beside a llama.cpp one with its own GGUF: %v", err)
|
||||
}
|
||||
m, _ := cat.Resolve("qwen3-embed-4b")
|
||||
if m.Source != "Qwen/Qwen3-Embedding-4B" {
|
||||
t.Errorf("the page was not read as the repository: %q", m.Source)
|
||||
}
|
||||
trt := embedModel()
|
||||
trt.Profiles[0] = Profile{Name: "gpu", Engine: "trtllm", GPUClass: "6000-24g", TP: 1, Source: "hf:a/b/m.gguf"}
|
||||
if _, err := New([]Model{trt}); err == nil || !strings.Contains(err.Error(), "not a .gguf") {
|
||||
t.Errorf("trtllm given a GGUF by its profile: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func mustJSON(t *testing.T, v any) string {
|
||||
t.Helper()
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -360,7 +361,7 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
names := append([]string{m.Name}, m.Aliases...)
|
||||
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
out = append(out, endpoint.Info{
|
||||
info := endpoint.Info{
|
||||
ID: rep.ID,
|
||||
URL: strings.TrimRight(rep.Endpoint, "/"),
|
||||
Models: names,
|
||||
@@ -369,7 +370,18 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
Pool: rep.Pool,
|
||||
Weight: rep.Weight,
|
||||
Priority: prof.Priority,
|
||||
})
|
||||
}
|
||||
if prof.Engine == catalog.EngineComfyUI {
|
||||
workflow, ckpt := m.ComfyWorkflow()
|
||||
w, err := c.workflow(workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; its replica is not offered",
|
||||
"model", m.Name, "replica", rep.ID, "err", err)
|
||||
continue
|
||||
}
|
||||
info.Workflow, info.UpstreamModel = w, ckpt
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return append(out, c.externalEndpoints()...)
|
||||
}
|
||||
@@ -412,23 +424,45 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
|
||||
if m.IsPeer() {
|
||||
id = "peer:" + m.Name
|
||||
}
|
||||
engine, workflow := endpoint.EngineExternal, ""
|
||||
if m.External.API == catalog.APIComfyUI {
|
||||
w, err := c.workflow(m.External.Workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; not offered", "model", m.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
engine, workflow = endpoint.EngineComfyUI, w
|
||||
}
|
||||
out = append(out, endpoint.Info{
|
||||
ID: id, URL: strings.TrimRight(m.External.URL, "/"),
|
||||
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
|
||||
Engine: engine, Pool: "external", Weight: 1,
|
||||
UpstreamModel: m.External.Model,
|
||||
Peer: m.External.Peer, PeerCluster: cluster, PeerTags: tags,
|
||||
Workflow: workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// workflow is a ComfyUI workflow as the router is sent it: a built-in's
|
||||
// name as it is, a file read here, on the control plane the catalog says it
|
||||
// is on, and sent as the workflow itself.
|
||||
func (c *Controller) workflow(w string) (string, error) {
|
||||
if !strings.ContainsAny(w, "/.") {
|
||||
return w, nil
|
||||
}
|
||||
b, err := os.ReadFile(w)
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
// fingerprint summarizes a list so an unchanged one can be skipped.
|
||||
func fingerprint(infos []endpoint.Info) string {
|
||||
var b strings.Builder
|
||||
for _, i := range infos {
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)))
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)),
|
||||
sha256.Sum256([]byte(i.Workflow)))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -87,8 +87,8 @@ func (s *Service) Sources(ctx context.Context) (map[string][]catalog.Model, erro
|
||||
}
|
||||
out := map[string][]catalog.Model{}
|
||||
for _, m := range c.Models() {
|
||||
if m.Source != "" {
|
||||
out[m.Source] = append(out[m.Source], m)
|
||||
for _, src := range m.WeightSources() {
|
||||
out[src] = append(out[src], m)
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
out[ad.Source] = append(out[ad.Source], m)
|
||||
|
||||
@@ -111,7 +111,14 @@ func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []Ad
|
||||
}
|
||||
r.mergeAdopted(rep, group)
|
||||
r.replicas[replicaID] = rep
|
||||
|
||||
// An on-demand model that is up was asked for, by somebody this
|
||||
// control plane has not heard of: it started before a restart or a
|
||||
// change of leader. Without this it reads as wanted by nobody and is
|
||||
// drained on the next pass — every awake on-demand model, at every
|
||||
// failover. It gets a whole idle timeout from now instead.
|
||||
if _, asked := r.wanted[m.Name]; !asked && m.Scaling.OnDemand() {
|
||||
r.wanted[m.Name] = r.now()
|
||||
}
|
||||
}
|
||||
return unclaimed
|
||||
}
|
||||
|
||||
@@ -1409,6 +1409,29 @@ func TestUseKeepsAnOnDemandModelAlive(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAwakeOnDemandModelSurvivesARestartOfTheControlPlane(t *testing.T) {
|
||||
// A control plane that has just started, or just become leader, knows
|
||||
// nobody asked for anything. Taking that to mean nobody wants the
|
||||
// on-demand models a node reports running drained every one of them at
|
||||
// every failover, under whoever was using them.
|
||||
h := newHarness(t, onDemandModel(10*time.Minute), gpuNode("1", 8))
|
||||
h.r.Adopt("1", []AdoptedShard{adopted("llama-99", "llama-99-r0", "1", 0, []int{0, 1, 2, 3})})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(5 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
if reps := h.r.Replicas(); len(reps) != 1 || !reps[0].State.Counts() {
|
||||
t.Fatalf("the adopted replica of an on-demand model was taken down: %+v", reps)
|
||||
}
|
||||
// Still an on-demand model: unused for a whole idle timeout, it goes.
|
||||
h.advance(6 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State.Counts() {
|
||||
t.Errorf("an adopted replica nobody used outlived its idle timeout: %+v", rep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelWithMaxZeroStaysOff(t *testing.T) {
|
||||
// min 0 max 0 is configured and deliberately not running, which is a
|
||||
// different and equally useful thing from "load it when wanted".
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
// Package train runs LoRA training jobs from the control plane (BACKLOG F4):
|
||||
// queued in the store, sent to the node each names when its agent is there
|
||||
// and has nothing else training, followed as the agent reports, and what it
|
||||
// makes kept in the store.
|
||||
package train
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Store is where jobs are kept.
|
||||
type Store interface {
|
||||
QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
|
||||
ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
|
||||
TrainJob(ctx context.Context, id string) (store.TrainJob, error)
|
||||
UpdateTrainJob(ctx context.Context, j store.TrainJob) error
|
||||
PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error
|
||||
}
|
||||
|
||||
// Agents is how jobs reach the nodes.
|
||||
type Agents interface {
|
||||
Send(node string, msg *llmgwv1.ControlMessage) error
|
||||
Connected(node string) bool
|
||||
}
|
||||
|
||||
// Service sends queued jobs and records how they go.
|
||||
type Service struct {
|
||||
store Store
|
||||
agents Agents
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
// pieces are artifacts arriving, by job and kind.
|
||||
pieces map[[2]string][]byte
|
||||
}
|
||||
|
||||
// New is a service.
|
||||
func New(s Store, a Agents, log *slog.Logger) *Service {
|
||||
if log == nil {
|
||||
log = slog.New(slog.DiscardHandler)
|
||||
}
|
||||
return &Service{store: s, agents: a, log: log, pieces: map[[2]string][]byte{}}
|
||||
}
|
||||
|
||||
// Every is how often queued jobs are looked at.
|
||||
var Every = 5 * time.Second
|
||||
|
||||
// Run sends queued jobs until ctx ends.
|
||||
func (s *Service) Run(ctx context.Context) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
s.Dispatch(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Dispatch sends each queued job whose node is here and not already
|
||||
// training, one per node.
|
||||
func (s *Service) Dispatch(ctx context.Context) {
|
||||
queued, err := s.store.QueuedTrainJobs(ctx)
|
||||
if err != nil || len(queued) == 0 {
|
||||
return
|
||||
}
|
||||
active, err := s.store.ActiveTrainJobs(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
busy := map[string]bool{}
|
||||
for _, j := range active {
|
||||
busy[j.Node] = true
|
||||
}
|
||||
for _, q := range queued {
|
||||
if busy[q.Node] || !s.agents.Connected(q.Node) {
|
||||
continue
|
||||
}
|
||||
j, err := s.store.TrainJob(ctx, q.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
|
||||
JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
|
||||
GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
|
||||
if err != nil {
|
||||
s.log.Warn("could not send a training job to its node", "job", j.ID, "node", j.Node, "err", err)
|
||||
continue
|
||||
}
|
||||
j.State, j.StartedAt, j.Message = store.TrainStarting, time.Now(), "sent to "+j.Node
|
||||
_ = s.store.UpdateTrainJob(ctx, j)
|
||||
busy[j.Node] = true
|
||||
s.log.Info("training job sent", "job", j.ID, "node", j.Node, "adapter", j.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// OnTrainStatus records how a job goes, as its node says.
|
||||
func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
|
||||
ctx := context.Background()
|
||||
j, err := s.store.TrainJob(ctx, st.GetJobId())
|
||||
if err != nil || j.Node != node || store.TrainOver(j.State) {
|
||||
return
|
||||
}
|
||||
j.State = st.GetState()
|
||||
if m := st.GetMessage(); m != "" || j.State == store.TrainDone {
|
||||
j.Message = m
|
||||
}
|
||||
if st.GetSteps() > 0 {
|
||||
j.Step, j.Steps = int(st.GetStep()), int(st.GetSteps())
|
||||
}
|
||||
if st.GetLoss() > 0 {
|
||||
j.Loss = st.GetLoss()
|
||||
}
|
||||
if p := int(st.GetPeakMib()); p > j.PeakMiB {
|
||||
j.PeakMiB = p
|
||||
}
|
||||
if store.TrainOver(j.State) {
|
||||
j.FinishedAt = time.Now()
|
||||
s.mu.Lock()
|
||||
for k := range s.pieces {
|
||||
if k[0] == j.ID {
|
||||
delete(s.pieces, k)
|
||||
}
|
||||
}
|
||||
s.mu.Unlock()
|
||||
s.log.Info("training job ended", "job", j.ID, "state", j.State, "message", j.Message)
|
||||
}
|
||||
if err := s.store.UpdateTrainJob(ctx, j); err != nil {
|
||||
s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// OnTrainArtifact gathers what a job made, and keeps it when it is whole.
|
||||
func (s *Service) OnTrainArtifact(node string, a *llmgwv1.TrainArtifact) {
|
||||
key := [2]string{a.GetJobId(), a.GetKind()}
|
||||
s.mu.Lock()
|
||||
if a.GetSeq() == 0 {
|
||||
s.pieces[key] = nil
|
||||
}
|
||||
s.pieces[key] = append(s.pieces[key], a.GetData()...)
|
||||
whole := s.pieces[key]
|
||||
if a.GetLast() {
|
||||
delete(s.pieces, key)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
if !a.GetLast() {
|
||||
return
|
||||
}
|
||||
if err := s.store.PutTrainArtifact(context.Background(), key[0], key[1], whole); err != nil {
|
||||
s.log.Warn("could not keep what a training job made", "job", key[0], "kind", key[1], "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// OnNodeGone fails the jobs a node was running: its agent's stream ended,
|
||||
// and the job with it.
|
||||
func (s *Service) OnNodeGone(node string) {
|
||||
ctx := context.Background()
|
||||
active, err := s.store.ActiveTrainJobs(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, j := range active {
|
||||
if j.Node != node {
|
||||
continue
|
||||
}
|
||||
j.State, j.FinishedAt = store.TrainFailed, time.Now()
|
||||
j.Message = "the node's agent went away while it trained; start the job again"
|
||||
_ = s.store.UpdateTrainJob(ctx, j)
|
||||
}
|
||||
}
|
||||
|
||||
// Cancel stops a job: one still queued is not sent, one running is stopped
|
||||
// on its node.
|
||||
func (s *Service) Cancel(ctx context.Context, id string) error {
|
||||
j, err := s.store.TrainJob(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if store.TrainOver(j.State) {
|
||||
return nil
|
||||
}
|
||||
if j.State != store.TrainQueued && s.agents.Connected(j.Node) {
|
||||
_ = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainCancel{
|
||||
TrainCancel: &llmgwv1.TrainCancel{JobId: id}}})
|
||||
}
|
||||
j.State, j.FinishedAt, j.Message = store.TrainCancelled, time.Now(), "cancelled"
|
||||
return s.store.UpdateTrainJob(ctx, j)
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package train
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeAgents struct {
|
||||
mu sync.Mutex
|
||||
connected map[string]bool
|
||||
sent []*llmgwv1.ControlMessage
|
||||
}
|
||||
|
||||
func (f *fakeAgents) Send(_ string, m *llmgwv1.ControlMessage) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.sent = append(f.sent, m)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeAgents) Connected(n string) bool {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.connected[n]
|
||||
}
|
||||
|
||||
func setup(t *testing.T) (*Service, *store.DB, *fakeAgents) {
|
||||
t.Helper()
|
||||
s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = s.Close() })
|
||||
a := &fakeAgents{connected: map[string]bool{}}
|
||||
return New(s, a, nil), s, a
|
||||
}
|
||||
|
||||
func job(t *testing.T, s *store.DB, id, node string) {
|
||||
t.Helper()
|
||||
if err := s.CreateTrainJob(context.Background(), store.TrainJob{ID: id, Name: "a-" + id, Model: "m", BaseRepo: "o/n",
|
||||
Node: node, GPUMemoryMiB: 2000, Data: []byte(`{"q":"x","a":"y"}`), Pairs: 1, GGUF: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobGoesToItsNodeWhenItIsHereAndFree(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, s, a := setup(t)
|
||||
job(t, s, "j1", "cuda01")
|
||||
job(t, s, "j2", "cuda01")
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 0 {
|
||||
t.Fatal("sent to a node that is not here")
|
||||
}
|
||||
a.connected["cuda01"] = true
|
||||
svc.Dispatch(ctx)
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 1 {
|
||||
t.Fatalf("sent %d jobs to one node", len(a.sent))
|
||||
}
|
||||
start := a.sent[0].GetTrainStart()
|
||||
if start.GetJobId() != "j1" || start.GetGpuMemoryMib() != 2000 || !start.GetGguf() || string(start.GetData()) == "" {
|
||||
t.Errorf("sent %+v", start)
|
||||
}
|
||||
if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainStarting || j.StartedAt.IsZero() {
|
||||
t.Errorf("j1 %+v", j)
|
||||
}
|
||||
|
||||
// Progress, then what it made in pieces, then done: the next is sent.
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running", Step: 5, Steps: 10, Loss: 0.8, PeakMib: 1900})
|
||||
svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 0, Data: []byte("GG")})
|
||||
svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 1, Data: []byte("UF"), Last: true})
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
|
||||
j, _ := s.TrainJob(ctx, "j1")
|
||||
if j.State != store.TrainDone || j.Step != 5 || j.PeakMiB != 1900 || j.Loss != 0.8 || j.FinishedAt.IsZero() {
|
||||
t.Errorf("j1 after %+v", j)
|
||||
}
|
||||
if b, _ := s.TrainArtifact(ctx, "j1", "gguf"); string(b) != "GGUF" {
|
||||
t.Errorf("artifact %q", b)
|
||||
}
|
||||
// A late message does not bring an ended job back.
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running"})
|
||||
if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainDone {
|
||||
t.Errorf("revived: %s", j.State)
|
||||
}
|
||||
// Nor does another node speak for it.
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 2 || a.sent[1].GetTrainStart().GetJobId() != "j2" {
|
||||
t.Fatalf("j2 not sent: %d", len(a.sent))
|
||||
}
|
||||
svc.OnTrainStatus("node01", &llmgwv1.TrainStatus{JobId: "j2", State: "done"})
|
||||
if j, _ := s.TrainJob(ctx, "j2"); j.State == store.TrainDone {
|
||||
t.Error("another node ended the job")
|
||||
}
|
||||
// The node goes: its job fails, saying why.
|
||||
svc.OnNodeGone("cuda01")
|
||||
if j, _ := s.TrainJob(ctx, "j2"); j.State != store.TrainFailed || j.Message == "" {
|
||||
t.Errorf("after the node went: %+v", j)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCancellingAJob(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, s, a := setup(t)
|
||||
job(t, s, "j1", "cuda01")
|
||||
if err := svc.Cancel(ctx, "j1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a.connected["cuda01"] = true
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 0 {
|
||||
t.Error("a cancelled job was sent")
|
||||
}
|
||||
job(t, s, "j2", "cuda01")
|
||||
svc.Dispatch(ctx)
|
||||
_ = svc.Cancel(ctx, "j2")
|
||||
if len(a.sent) != 2 || a.sent[1].GetTrainCancel().GetJobId() != "j2" {
|
||||
t.Errorf("the node was not told: %+v", a.sent)
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
|
||||
{Key: "source", Type: "string", Doc: "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT\nadapter directory (a mirror version ending in \"/\", a path on the node,\nor a Hugging Face repository id). Fetched by each node as the model's\nweights are.", Nested: ""},
|
||||
{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree the adapter belongs to —\norg:ID, team:ID, project:ID, user:ID or key:ID — and only callers in\nit see it: to anyone else it is a model that does not exist. An\nadapter trained on an organisation's documents carries some of what\nthey say in its weights. Empty is every caller the model allows.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
|
||||
@@ -63,6 +64,18 @@ var Reference = []Block{
|
||||
{Key: "tools", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "images", Type: "bool", Doc: "Images is a model that makes pictures, asked for at\n/v1/images/generations. Implied by an external model whose API is\nComfyUI's.", Nested: ""},
|
||||
{Key: "meshes", Type: "bool", Doc: "Meshes is a model that makes 3D models from a picture, asked for as a\njob at /v1/3d/generations. Implied by a ComfyUI model whose output is\nmeshes.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ComfyUI", Title: "ComfyUI", Doc: "", Fields: []Field{
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is a built-in's name (flux-schnell, the default, or\nflux-dev) or a JSON file on the control plane, in ComfyUI's API\nformat.", Nested: ""},
|
||||
{Key: "checkpoint", Type: "string", Doc: "Checkpoint is the file the workflow's {{checkpoint}} names: the\nsource's own name unless set, which a source that is a set of files\nmust.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what the workflow makes: pictures (the default), asked for\nat /v1/images/generations, or meshes, asked for as a job at\n/v1/3d/generations from a picture.", Nested: ""},
|
||||
{Key: "nodes", Type: "[]CustomNode", Doc: "Nodes are custom nodes the workflow needs, put in each replica's own\ncustom_nodes at the commit named.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "ComfyUIConfig", Title: "ComfyUIConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is ComfyUI's directory, where main.py is. Readable by the user\nengines run as.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
|
||||
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
|
||||
@@ -77,12 +90,18 @@ var Reference = []Block{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled runs the control plane even when configuration declares no\nmodels.\n\nWithout it, a deployment whose catalog lives entirely in the database\ncannot start: control plane mode is inferred from the config file, so an\nempty models block means \"router only\", the process refuses to start for\nwant of anything to serve, and the models it would have loaded are never\nread. That is a cluster that cannot be managed at runtime because it is\nmanaged at runtime. AgentListen cannot carry this signal — it has a\ndefault, so every deployment would become a control plane.", Nested: ""},
|
||||
{Key: "lease_ttl", Type: "time.Duration", Doc: "LeaseTTL is how long this control plane's claim to be the one placing\nwork stands without being renewed.\n\nOnly meaningful with a shared store, where a second control plane can\nsee the claim. Long enough to survive a slow query or a brief network\nstall, short enough that a standby takes over before anyone has finished\nreading the alert. Zero takes the default.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "CustomNode", Title: "CustomNode", Doc: "", Fields: []Field{
|
||||
{Key: "repo", Type: "string", Doc: "Repo is the repository, https only.", Nested: ""},
|
||||
{Key: "commit", Type: "string", Doc: "Commit is the full commit the node is used at: a branch moves, and a\nnode that changed under a model is a model that changed unasked.", Nested: ""},
|
||||
{Key: "requirements", Type: "[]string", Doc: "Requirements are the PyPI packages the node needs, by name and version\nrange only. They are installed beside the replica, never into\nComfyUI's environment: a package named here at the range named, and\nevery other one at the version ComfyUI already runs. One named here\nis first on ComfyUI's path, so its range must suit ComfyUI too.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Endpoint", Title: "Endpoint", Doc: "", Fields: []Field{
|
||||
{Key: "id", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
|
||||
{Key: "models", Type: "[]string", Doc: "", Nested: ""},
|
||||
{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow for engine: comfyui, a built-in's\nname or its JSON; flux-schnell unless set.", Nested: ""},
|
||||
{Key: "pool", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
@@ -103,6 +122,9 @@ var Reference = []Block{
|
||||
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
|
||||
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
|
||||
{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
|
||||
{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
|
||||
{Key: "output", Type: "string", Doc: "Output is what it makes: pictures, or meshes (see ComfyUI.Output).", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
@@ -148,6 +170,11 @@ var Reference = []Block{
|
||||
{Key: "language", Type: "string", Doc: "Language is what the web portal speaks to someone who has not chosen\na language with the switcher at the foot of every page and whose\nbrowser asks for none it speaks: \"en\" or \"tr\". Empty is English.", Nested: ""},
|
||||
{Key: "docs_sign_in", Type: "bool", Doc: "DocsSignIn serves the docs portal at /docs only to someone signed in.\nOff, the portal is public like this page: its client guide is for the\nperson who has no account yet, and the operator documents describe\nllmgw in general — no address, model or key of this installation —\nwhich the project's repository publishes anyway. On, a visitor is sent\nto sign in, and without accounts the portal needs an API key.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "License", Title: "License", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "Name is the licence's own: MIT, Apache-2.0.", Nested: ""},
|
||||
{Key: "url", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "notice", Type: "string", Doc: "Notice is what must accompany what was made: attribution, terms on\nthe output, what the weights were trained on.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "LlamaCppConfig", Title: "LlamaCppConfig", Doc: "", Fields: []Field{
|
||||
{Key: "bin", Type: "string", Doc: "Bin is the llama-server binary. It is the server, not the CLI: the\ngateway talks OpenAI over HTTP and needs the one that serves it.\nEmpty means the agent installs llama.cpp itself, at the version the\ncontrol plane asks for. Setting it opts out of that.", Nested: ""},
|
||||
{Key: "parallel", Type: "int", Doc: "Parallel is how many requests a server handles at once, when a profile\ndoes not say. llama.cpp defaults to one, which would make every second\nrequest wait behind the first no matter what the router decided.", Nested: ""},
|
||||
@@ -170,6 +197,8 @@ var Reference = []Block{
|
||||
{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
|
||||
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
|
||||
{Key: "comfyui", Type: "*ComfyUI", Doc: "ComfyUI is how a model whose profiles run engine comfyui makes its\npictures: the workflow, and the file it loads.", Nested: "catalog.ComfyUI"},
|
||||
{Key: "license", Type: "*License", Doc: "License is the terms of what the model makes, for a model whose\noutput somebody ships: shown with every result and written into a\nmesh.", Nested: "catalog.License"},
|
||||
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it — unless\nMaxLoadedAdapters is set.", Nested: ""},
|
||||
{Key: "max_loaded_adapters", Type: "int", Doc: "MaxLoadedAdapters, above zero, has each replica hold at most this many\nadapters at once, loaded when a request first asks for one and unloaded\nleast recently used first to make room. The router sends a request for\nan adapter to a replica that already holds it while that one has room\nto take it. vLLM profiles only: llama-server loads adapters at start\nand cannot load one afterwards. Zero loads every adapter on every\nreplica at start.", Nested: ""},
|
||||
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
|
||||
@@ -180,6 +209,7 @@ var Reference = []Block{
|
||||
{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
|
||||
{Key: "max_output", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
|
||||
{Key: "query_prefix", Type: "string", Doc: "QueryPrefix is put before a question an embedding model embeds for a\nknowledge-base search; documents are embedded without it. Models\ntrained with an instruction on the query side want it, Qwen3-Embedding\n\"Instruct: Given a web search query, retrieve relevant passages that\nanswer the query\\nQuery: \" for one.", Nested: ""},
|
||||
{Key: "role", Type: "string", Doc: "Role \"guard\" is a model that judges the gateway's traffic for the\nguardrails rather than serving clients: hidden from /v1/models and the\nchat page, refused to any key that names it, critical by default, and\nalways running. Guard says how its answers are read.", Nested: ""},
|
||||
{Key: "guard", Type: "*GuardModel", Doc: "", Nested: "catalog.GuardModel"},
|
||||
{Key: "origin", Type: "*Origin", Doc: "Origin is the registry repository the weights came from, at which\ncommit, and each file's checksum there. Written when a model is served\nfrom the registry; for one that was not, worked out from the mirror's\nrecord of where it fetched the file. It is what lets the control plane\nsay that the repository has published different weights since, and\nmove the model to them.", Nested: "catalog.Origin"},
|
||||
@@ -269,6 +299,7 @@ var Reference = []Block{
|
||||
{Key: "engine", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "engine_version", Type: "string", Doc: "EngineVersion restricts the profile to nodes whose installed engine's\nversion starts with it: \"0.11\" takes vLLM 0.11.0 and 0.11.1, not\n0.10.2. Empty takes whatever version the node has.", Nested: ""},
|
||||
{Key: "gpu_class", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "source", Type: "string", Doc: "Source is the weights this profile loads, when not the model's: an\nembedding model served by vLLM from its repository on a card and by\nllama.cpp from a GGUF on CPUs is one model with two sources. Empty\nloads the model's. Updates and canaries move the model's source; a\nprofile with its own keeps it.", Nested: ""},
|
||||
{Key: "tp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "pp", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "gpus_per_node", Type: "int", Doc: "GPUsPerNode defaults to TP for a single-node profile.", Nested: ""},
|
||||
@@ -443,11 +474,14 @@ var Reference = []Block{
|
||||
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is what one 3D model costs: minutes of a card, which no\ntoken count measures.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "Priority orders bands; higher is served first.", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "SaturationCeiling is the pool saturation above which this band waits.\n1.0 means \"never hold this band back\".", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL, set, is how long this band's requests may wait instead of\nthe controller's: a band for bulk work — a knowledge-base import, a\nbackfill — whose requests had rather wait ten minutes behind people\nthan be turned away after thirty seconds.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Config", Title: "admission.Config", Doc: "", Fields: []Field{
|
||||
{Key: "bands", Type: "[]Band", Doc: "", Nested: ""},
|
||||
@@ -483,10 +517,13 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "saturation_ceiling", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "queue_ttl", Type: "time.Duration", Doc: "QueueTTL is how long this band's requests may wait for room; the\nadmission's queue_ttl unless set. Long for a band of bulk work, which\nhad rather wait behind people than be turned away.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
|
||||
{Key: "per_mesh", Type: "int64", Doc: "PerMesh is the cost of one 3D model (/v1/3d/generations).", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
|
||||
{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
|
||||
@@ -495,6 +532,7 @@ var Reference = []Block{
|
||||
{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
|
||||
{Key: "trtllm", Type: "TRTLLMConfig", Doc: "TRTLLM is TensorRT-LLM, which the agent does not install: a node\noffers it only when its interpreter is named here.", Nested: "engines.TRTLLMConfig"},
|
||||
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
|
||||
{Key: "comfyui", Type: "ComfyUIConfig", Doc: "ComfyUI makes pictures; offered by a node whose ComfyUI is named here.", Nested: "engines.ComfyUIConfig"},
|
||||
}},
|
||||
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
|
||||
|
||||
@@ -22,6 +22,7 @@ var Sources = []string{
|
||||
"internal/agent/engines/sglang.go",
|
||||
"internal/agent/engines/trtllm.go",
|
||||
"internal/agent/engines/sim.go",
|
||||
"internal/agent/engines/comfyui.go",
|
||||
"internal/agent/supervisor/supervisor.go",
|
||||
"internal/tlsconf/tlsconf.go",
|
||||
"internal/control/catalog/catalog.go",
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// A store member that is nothing else runs no llmgw process: Patroni, etcd
|
||||
// and PostgreSQL only. What the cluster's settings say of its firewall it
|
||||
// reads from its own copy of the store, where the leading control plane
|
||||
// keeps it (the cluster_state table, key firewall), once a minute, by a
|
||||
// root timer running FollowScript. It turns llmgw's table off and on, and
|
||||
// lets in the networks the settings add; what `server init` let in stays.
|
||||
|
||||
// FollowScript is the script, as python3 runs it. LLMGW_FW_STATE and
|
||||
// LLMGW_FW_RULES replace the store and the rules file, for its tests.
|
||||
const FollowScript = `#!/usr/bin/python3
|
||||
# Written by llmgw: llmgw's firewall on this store member, as the cluster's
|
||||
# settings say. Run by llmgw-firewall-follow.timer.
|
||||
import ipaddress, json, os, subprocess, sys, tempfile
|
||||
|
||||
RULES = os.environ.get("LLMGW_FW_RULES", "` + RulesFile + `")
|
||||
|
||||
def run(*a):
|
||||
return subprocess.run(a, capture_output=True, text=True)
|
||||
|
||||
raw = os.environ.get("LLMGW_FW_STATE")
|
||||
if raw is None:
|
||||
r = run("runuser", "-u", "postgres", "--", "psql", "-d", "llmgw", "-tAqc",
|
||||
"SELECT value FROM cluster_state WHERE key = 'firewall'")
|
||||
# No table yet, or nothing told: a control plane older than this, or
|
||||
# none leading. Left as it is.
|
||||
if r.returncode != 0:
|
||||
sys.exit(0)
|
||||
raw = r.stdout
|
||||
if not raw.strip():
|
||||
sys.exit(0)
|
||||
st = json.loads(raw)
|
||||
present = run("nft", "list", "table", "inet", "` + Table + `").returncode == 0
|
||||
|
||||
if not st.get("enabled", True):
|
||||
if present:
|
||||
run("systemctl", "disable", "--now", "` + Unit + `")
|
||||
run("nft", "delete", "table", "inet", "` + Table + `")
|
||||
print("llmgw's firewall is off here")
|
||||
sys.exit(0)
|
||||
|
||||
if not os.path.exists(RULES):
|
||||
sys.exit(0)
|
||||
lines = open(RULES).read().split("\n")
|
||||
want = []
|
||||
for n in st.get("networks") or []:
|
||||
try:
|
||||
want.append(ipaddress.ip_network(n, strict=False))
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def saddr(l):
|
||||
s = l.strip()
|
||||
return s.startswith("ip saddr {") or s.startswith("ip6 saddr {")
|
||||
|
||||
changed = False
|
||||
for fam, kw in ((4, "ip saddr {"), (6, "ip6 saddr {")):
|
||||
add = [str(n) for n in want if n.version == fam]
|
||||
if not add:
|
||||
continue
|
||||
same = [i for i, l in enumerate(lines) if l.strip().startswith(kw)]
|
||||
if same:
|
||||
i = same[0]
|
||||
l = lines[i]
|
||||
a, b = l.index("{") + 1, l.index("}")
|
||||
have = [x.strip() for x in l[a:b].split(",") if x.strip()]
|
||||
new = [x for x in add if x not in have]
|
||||
if new:
|
||||
lines[i] = l[:a] + " " + ", ".join(have + new) + " " + l[b:]
|
||||
changed = True
|
||||
continue
|
||||
other = [i for i, l in enumerate(lines) if saddr(l)]
|
||||
if not other:
|
||||
continue
|
||||
l = lines[other[0]]
|
||||
indent = l[:len(l) - len(l.lstrip())]
|
||||
lines.insert(other[0] + 1, indent + kw + " " + ", ".join(add) + " }" + l[l.index("}") + 1:])
|
||||
changed = True
|
||||
|
||||
if changed:
|
||||
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(RULES))
|
||||
with os.fdopen(fd, "w") as f:
|
||||
f.write("\n".join(lines))
|
||||
if run("nft", "-c", "-f", tmp).returncode != 0:
|
||||
os.unlink(tmp)
|
||||
print("the settings' networks make a ruleset nft refuses; left as it was", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, RULES)
|
||||
if changed or not present:
|
||||
run("systemctl", "enable", "` + Unit + `")
|
||||
if run("nft", "-f", RULES).returncode != 0:
|
||||
sys.exit(1)
|
||||
print("llmgw's firewall is on here")
|
||||
`
|
||||
|
||||
// FollowerPath is where FollowScript is installed.
|
||||
const FollowerPath = "/usr/local/lib/llmgw/firewall-follow"
|
||||
|
||||
// FollowerTimer is the timer that runs it.
|
||||
const FollowerTimer = "llmgw-firewall-follow.timer"
|
||||
|
||||
const followService = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this store member
|
||||
After=postgresql.service patroni.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/python3 ` + FollowerPath + `
|
||||
`
|
||||
|
||||
const followTimer = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this store member
|
||||
|
||||
[Timer]
|
||||
OnBootSec=1min
|
||||
OnUnitActiveSec=1min
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
`
|
||||
|
||||
// InstallFollower puts FollowScript and its timer on a store member.
|
||||
func InstallFollower(ctx context.Context, r Runner) error {
|
||||
body := FollowScript + "\x00" + followService + "\x00" + followTimer
|
||||
out, err := r.Run(ctx, `set -e
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
python3 - "$t" <<'PY'
|
||||
import sys
|
||||
d = sys.argv[1]
|
||||
s, svc, tim = open(d + "/all", "rb").read().split(b"\x00")
|
||||
open(d + "/script", "wb").write(s)
|
||||
open(d + "/service", "wb").write(svc)
|
||||
open(d + "/timer", "wb").write(tim)
|
||||
PY
|
||||
sudo install -d -m 755 /usr/local/lib/llmgw
|
||||
sudo install -m 755 "$t/script" `+FollowerPath+`
|
||||
sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-follow.service
|
||||
sudo install -m 644 "$t/timer" /etc/systemd/system/`+FollowerTimer+`
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now `+FollowerTimer+` >/dev/null 2>&1`, []byte(body))
|
||||
if err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", FollowerTimer, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"net/netip"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// follow runs FollowScript against a rules file and a state, with nft and
|
||||
// systemctl replaced by scripts that log what they were asked and answer as
|
||||
// a machine whose table is loaded or not.
|
||||
func follow(t *testing.T, rules, state string, loaded bool) (string, string, string) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("python3"); err != nil {
|
||||
t.Skip("no python3")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
bin := filepath.Join(dir, "bin")
|
||||
_ = os.Mkdir(bin, 0o755)
|
||||
log := filepath.Join(dir, "log")
|
||||
list := "exit 1"
|
||||
if loaded {
|
||||
list = "exit 0"
|
||||
}
|
||||
for name, body := range map[string]string{
|
||||
"nft": `echo "nft $*" >> ` + log + "\n" + `[ "$1" = list ] && ` + list + "\nexit 0\n",
|
||||
"systemctl": `echo "systemctl $*" >> ` + log + "\nexit 0\n",
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(bin, name), []byte("#!/bin/sh\n"+body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
rf := filepath.Join(dir, "firewall.nft")
|
||||
if rules != "" {
|
||||
_ = os.WriteFile(rf, []byte(rules), 0o644)
|
||||
}
|
||||
script := filepath.Join(dir, "follow")
|
||||
_ = os.WriteFile(script, []byte(FollowScript), 0o755)
|
||||
cmd := exec.Command("python3", script)
|
||||
cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), "LLMGW_FW_STATE="+state, "LLMGW_FW_RULES="+rf)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("%v: %s", err, out)
|
||||
}
|
||||
got, _ := os.ReadFile(rf)
|
||||
calls, _ := os.ReadFile(log)
|
||||
return string(out), string(got), string(calls)
|
||||
}
|
||||
|
||||
var memberRules = Script(Spec{Ports: []Port{Postgres, EtcdClient}, Allow: mustPrefixes("10.0.0.0/24")})
|
||||
|
||||
func TestAStoreMemberFollowsTheSettings(t *testing.T) {
|
||||
// Off: the table goes, the rules stay for later.
|
||||
out, rules, calls := follow(t, memberRules, `{"enabled":false}`, true)
|
||||
if !strings.Contains(out, "off") || !strings.Contains(calls, "nft delete table inet llmgw") ||
|
||||
!strings.Contains(calls, "systemctl disable --now "+Unit) || rules != memberRules {
|
||||
t.Errorf("off: %q %q", out, calls)
|
||||
}
|
||||
// On again: loaded from the rules kept.
|
||||
out, _, calls = follow(t, memberRules, `{"enabled":true}`, false)
|
||||
if !strings.Contains(out, "on") || !strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("on: %q %q", out, calls)
|
||||
}
|
||||
// On and loaded, nothing new: nothing done.
|
||||
out, _, calls = follow(t, memberRules, `{"enabled":true,"networks":["10.0.0.0/24"]}`, true)
|
||||
if out != "" || strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("unchanged: %q %q", out, calls)
|
||||
}
|
||||
// Networks the settings add, of both families, are let in beside what
|
||||
// init let in; the result reads back as the same ports.
|
||||
_, rules, calls = follow(t, memberRules, `{"enabled":true,"networks":["192.168.5.0/24","fd00::/64","nonsense"]}`, true)
|
||||
if !strings.Contains(calls, "nft -c -f") || !strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("not checked and loaded: %q", calls)
|
||||
}
|
||||
back := parse(rules)
|
||||
var nets []string
|
||||
for _, p := range back.Allow {
|
||||
nets = append(nets, p.String())
|
||||
}
|
||||
if strings.Join(nets, ",") != "10.0.0.0/24,192.168.5.0/24,fd00::/64" || len(back.Ports) != 2 {
|
||||
t.Errorf("rules now:\n%s", rules)
|
||||
}
|
||||
// Nothing told yet, or no rules: left alone.
|
||||
if out, _, calls := follow(t, memberRules, "", true); out != "" || calls != "" {
|
||||
t.Errorf("nothing told: %q %q", out, calls)
|
||||
}
|
||||
if out, _, calls := follow(t, "", `{"enabled":true}`, false); out != "" || strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("no rules: %q %q", out, calls)
|
||||
}
|
||||
}
|
||||
|
||||
func mustPrefixes(ss ...string) []netip.Prefix {
|
||||
var out []netip.Prefix
|
||||
for _, s := range ss {
|
||||
out = append(out, netip.MustParsePrefix(s))
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
@@ -37,7 +38,9 @@ func (a *API) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /v1/models", a.fetch)
|
||||
mux.HandleFunc("GET /v1/models/{model}/{version}", a.status)
|
||||
mux.HandleFunc("DELETE /v1/models/{model}/{version}", a.remove)
|
||||
mux.HandleFunc("PUT /v1/models/{model}/{version}/{file}", a.upload)
|
||||
mux.HandleFunc("GET /v1/health", a.health)
|
||||
mux.HandleFunc("PUT /v1/firewall", a.putFirewall)
|
||||
return mux
|
||||
}
|
||||
|
||||
@@ -199,3 +202,59 @@ func LastPathElement(u string) string {
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
// maxUpload bounds a file sent to the mirror rather than fetched by it.
|
||||
const maxUpload = 8 << 30
|
||||
|
||||
// upload keeps a file the caller sends — an adapter a node trained, which
|
||||
// is on no registry to fetch from — with the same discipline as an import:
|
||||
// its checksum and source beside it, and an existing version never
|
||||
// replaced. X-Llmgw-Source says where it came from.
|
||||
func (a *API) upload(w http.ResponseWriter, r *http.Request) {
|
||||
e := Entry{Model: r.PathValue("model"), Version: r.PathValue("version"), File: r.PathValue("file")}
|
||||
for _, n := range []string{e.Model, e.Version, e.File} {
|
||||
if err := ValidName(n); err != nil {
|
||||
fail(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(e.Path(a.m.root)); err == nil {
|
||||
fail(w, http.StatusConflict, fmt.Errorf("%s/%s/%s: %w", e.Model, e.Version, e.File, ErrExists))
|
||||
return
|
||||
}
|
||||
tmp, err := os.CreateTemp(a.m.root, ".upload-*")
|
||||
if err != nil {
|
||||
fail(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
defer os.Remove(tmp.Name())
|
||||
n, err := io.Copy(tmp, http.MaxBytesReader(w, r.Body, maxUpload))
|
||||
if cerr := tmp.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
fail(w, http.StatusBadRequest, fmt.Errorf("receiving the file: %w", err))
|
||||
return
|
||||
}
|
||||
if n == 0 {
|
||||
fail(w, http.StatusBadRequest, errors.New("the file is empty"))
|
||||
return
|
||||
}
|
||||
source := strings.TrimSpace(r.Header.Get("X-Llmgw-Source"))
|
||||
if source == "" {
|
||||
source = "uploaded"
|
||||
}
|
||||
added, err := Import(r.Context(), a.m.root, tmp.Name(), e, "", source, nil)
|
||||
switch {
|
||||
case errors.Is(err, ErrExists):
|
||||
fail(w, http.StatusConflict, err)
|
||||
return
|
||||
case err != nil:
|
||||
fail(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, FetchResponse{
|
||||
Status: Status{Model: e.Model, Version: e.Version, File: e.File, State: Present, BytesDone: added.Entry.Bytes, BytesTotal: added.Entry.Bytes},
|
||||
URL: e.URL(a.base),
|
||||
})
|
||||
}
|
||||
@@ -100,6 +100,36 @@ func (c *Client) Fetch(ctx context.Context, req FetchRequest) (FetchResponse, er
|
||||
return out, err
|
||||
}
|
||||
|
||||
// Upload sends a file for the mirror to keep at model/version/file, with
|
||||
// where it came from; it answers the URL nodes fetch it from.
|
||||
func (c *Client) Upload(ctx context.Context, model, version, file string, body []byte, source string) (FetchResponse, error) {
|
||||
var out FetchResponse
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPut,
|
||||
c.base+"/v1/models/"+url.PathEscape(model)+"/"+url.PathEscape(version)+"/"+url.PathEscape(file), bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
req.Header.Set("X-Llmgw-Source", source)
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("reaching the mirror at %s: %w", c.base, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if resp.StatusCode >= 300 {
|
||||
var e struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.Unmarshal(raw, &e)
|
||||
return out, &APIError{Status: resp.StatusCode, Msg: e.Error}
|
||||
}
|
||||
return out, json.Unmarshal(raw, &out)
|
||||
}
|
||||
|
||||
// Status is where one model version stands.
|
||||
func (c *Client) Status(ctx context.Context, model, version string) (FetchResponse, error) {
|
||||
var out FetchResponse
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// The mirror's machine keeps llmgw's firewall as the cluster's settings
|
||||
// say, as every other machine does. The mirror runs unprivileged and cannot
|
||||
// change the ruleset: the control plane tells it what the settings say, it
|
||||
// writes that beside its models, and a root path unit on the machine
|
||||
// applies it with `llmgw firewall sync`.
|
||||
|
||||
// FirewallState is what the cluster's settings say of a machine's firewall.
|
||||
type FirewallState struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
// Networks are let in besides those the machine reaches others on.
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
// Toward are the cluster's machines, by address: each one's network is
|
||||
// let in, as the machine's route to it says.
|
||||
Toward []string `json:"toward,omitempty"`
|
||||
}
|
||||
|
||||
// FirewallFile is where, in the mirror's directory, the state is written.
|
||||
// Hidden, so it is never taken for a model.
|
||||
const FirewallFile = ".firewall.json"
|
||||
|
||||
func (a *API) putFirewall(w http.ResponseWriter, r *http.Request) {
|
||||
var st FirewallState
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&st); err != nil {
|
||||
fail(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
b, _ := json.Marshal(st)
|
||||
p := filepath.Join(a.m.root, FirewallFile)
|
||||
changed := false
|
||||
if old, err := os.ReadFile(p); err != nil || !bytes.Equal(old, b) {
|
||||
// Renamed into place: the path unit sees one change, of a whole file.
|
||||
tmp := p + ".new"
|
||||
if err := os.WriteFile(tmp, b, 0o644); err != nil {
|
||||
fail(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
if err := os.Rename(tmp, p); err != nil {
|
||||
fail(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]bool{"changed": changed})
|
||||
}
|
||||
|
||||
// SetFirewall tells the mirror what the cluster's settings say of its
|
||||
// machine's firewall; it answers whether that changed anything.
|
||||
func (c *Client) SetFirewall(ctx context.Context, st FirewallState) (bool, error) {
|
||||
var out struct {
|
||||
Changed bool `json:"changed"`
|
||||
}
|
||||
err := c.do(ctx, http.MethodPut, "/v1/firewall", st, &out)
|
||||
return out.Changed, err
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestTheMirrorKeepsWhatTheSettingsSayOfItsFirewall(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
srv := httptest.NewServer(NewAPI(NewManager(root, discardLog()), "http://m:8081").Handler())
|
||||
defer srv.Close()
|
||||
c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
st := FirewallState{Enabled: true, Networks: []string{"192.168.5.0/24"}, Toward: []string{"10.0.0.147"}}
|
||||
if changed, err := c.SetFirewall(ctx, st); err != nil || !changed {
|
||||
t.Fatalf("first: %v %v", changed, err)
|
||||
}
|
||||
if changed, err := c.SetFirewall(ctx, st); err != nil || changed {
|
||||
t.Errorf("the same again: %v %v", changed, err)
|
||||
}
|
||||
var back FirewallState
|
||||
b, _ := os.ReadFile(filepath.Join(root, FirewallFile))
|
||||
if err := json.Unmarshal(b, &back); err != nil || !back.Enabled || back.Toward[0] != "10.0.0.147" {
|
||||
t.Errorf("kept %s", b)
|
||||
}
|
||||
if all, err := c.List(ctx); err != nil || len(all.Models) != 0 {
|
||||
t.Errorf("the file is taken for a model: %+v %v", all, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestAFileSentToTheMirrorIsKeptWithItsChecksumAndNeverReplaced(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
srv := httptest.NewServer(NewAPI(NewManager(root, nil), "http://mirror:8081").Handler())
|
||||
defer srv.Close()
|
||||
c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
got, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("GGUF adapter"), "trained by job train-1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.URL != "http://mirror:8081/politika-tonu/train-1/politika-tonu.gguf" || got.Status.BytesTotal != 12 {
|
||||
t.Errorf("answered %+v", got)
|
||||
}
|
||||
e := Entry{Model: "politika-tonu", Version: "train-1", File: "politika-tonu.gguf"}
|
||||
if b, _ := os.ReadFile(e.Path(root)); string(b) != "GGUF adapter" {
|
||||
t.Errorf("kept %q", b)
|
||||
}
|
||||
if b, _ := os.ReadFile(sourcePath(e.Path(root))); !strings.Contains(string(b), "train-1") {
|
||||
t.Errorf("source %q", b)
|
||||
}
|
||||
if _, err := os.Stat(checksumPath(e.Path(root))); err != nil {
|
||||
t.Errorf("no checksum: %v", err)
|
||||
}
|
||||
if _, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("other"), "x"); err == nil {
|
||||
t.Error("an existing version was replaced")
|
||||
}
|
||||
if _, err := c.Upload(ctx, "../etc", "v", "f", []byte("x"), "x"); err == nil {
|
||||
t.Error("a path outside the mirror was taken")
|
||||
}
|
||||
}
|
||||
@@ -110,6 +110,23 @@ type Notifier struct {
|
||||
mu sync.Mutex
|
||||
sent map[string]time.Time
|
||||
wg sync.WaitGroup
|
||||
|
||||
claim Claimer
|
||||
}
|
||||
|
||||
// Claimer is where control planes agree which of them sends an event: each
|
||||
// raises the same ones — a budget is spent on every control plane serving
|
||||
// its requests — and only the one that claims an event sends it.
|
||||
type Claimer interface {
|
||||
ClaimNotice(ctx context.Context, key string, at time.Time, window time.Duration) (bool, error)
|
||||
}
|
||||
|
||||
// SetClaimer has events claimed before they are sent. Set before the first
|
||||
// event; nil leaves each process to send its own.
|
||||
func (n *Notifier) SetClaimer(c Claimer) {
|
||||
if n != nil {
|
||||
n.claim = c
|
||||
}
|
||||
}
|
||||
|
||||
type hook struct {
|
||||
@@ -199,6 +216,9 @@ func (n *Notifier) run(ctx context.Context) {
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !n.claimed(ctx, e) {
|
||||
continue
|
||||
}
|
||||
for _, h := range n.hooks {
|
||||
if len(h.Events) > 0 && !slices.Contains(h.Events, e.Kind) {
|
||||
continue
|
||||
@@ -295,3 +315,19 @@ func (n *Notifier) send(ctx context.Context, h hook, e Event, attempts int) erro
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// claimed reports whether this process is the one to send an event. A store
|
||||
// that cannot be asked sends it anyway: told twice beats not told.
|
||||
func (n *Notifier) claimed(ctx context.Context, e Event) bool {
|
||||
if n.claim == nil {
|
||||
return true
|
||||
}
|
||||
cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
ok, err := n.claim.ClaimNotice(cctx, e.Kind+"\x00"+e.Subject, e.At, dedupe)
|
||||
if err != nil {
|
||||
n.log.Warn("could not ask the store whether a notification was sent; sending it", "kind", e.Kind, "err", err)
|
||||
return true
|
||||
}
|
||||
return ok
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -167,3 +168,57 @@ func TestTestSaysHowEachAnsweredWithoutTheSecretPath(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// sharedClaims is a store two control planes share.
|
||||
type sharedClaims struct {
|
||||
mu sync.Mutex
|
||||
last map[string]time.Time
|
||||
err error
|
||||
}
|
||||
|
||||
func (s *sharedClaims) ClaimNotice(_ context.Context, key string, at time.Time, window time.Duration) (bool, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.err != nil {
|
||||
return false, s.err
|
||||
}
|
||||
if prev, ok := s.last[key]; ok && at.Sub(prev) < window {
|
||||
return false, nil
|
||||
}
|
||||
s.last[key] = at
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func TestTwoControlPlanesSendAnEventOnce(t *testing.T) {
|
||||
// Each control plane serving a team's requests sees its budget spent,
|
||||
// and each kept its own hour of quiet: two behind a balancer sent every
|
||||
// budget event twice.
|
||||
var g got
|
||||
srv := httptest.NewServer(g.handler(0))
|
||||
defer srv.Close()
|
||||
claims := &sharedClaims{last: map[string]time.Time{}}
|
||||
a := newNotifier(t, Config{Host: "ctl1", Webhooks: []Webhook{{URL: srv.URL}}})
|
||||
b := newNotifier(t, Config{Host: "ctl2", Webhooks: []Webhook{{URL: srv.URL}}})
|
||||
a.SetClaimer(claims)
|
||||
b.SetClaimer(claims)
|
||||
a.Notify(BudgetSpent, "team:t1", "spent", nil)
|
||||
b.Notify(BudgetSpent, "team:t1", "spent", nil)
|
||||
b.Notify(BudgetSpent, "team:t2", "spent", nil)
|
||||
a.Wait()
|
||||
b.Wait()
|
||||
if len(g.calls) != 2 {
|
||||
t.Errorf("%d sent, want one for each team", len(g.calls))
|
||||
}
|
||||
|
||||
// A store that cannot be asked: sent, rather than lost.
|
||||
var g2 got
|
||||
srv2 := httptest.NewServer(g2.handler(0))
|
||||
defer srv2.Close()
|
||||
c := newNotifier(t, Config{Webhooks: []Webhook{{URL: srv2.URL}}})
|
||||
c.SetClaimer(&sharedClaims{err: errors.New("store down")})
|
||||
c.Notify(NodeLost, "node3", "lost", nil)
|
||||
c.Wait()
|
||||
if len(g2.calls) != 1 {
|
||||
t.Errorf("with the store down, %d sent, want 1", len(g2.calls))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PostgreSQL's own defaults — 128MB of cache, 4MB to sort in — are for a
|
||||
// machine it barely uses. A knowledge base's vector index is searched well
|
||||
// only from memory: one that does not fit is read from disk on every search.
|
||||
// So each member is sized to the memory it has, read when setup runs, and
|
||||
// again on every later run, so a machine given more memory gets the use of it.
|
||||
|
||||
// MemorySettings are PostgreSQL's memory parameters for a machine with total
|
||||
// bytes of memory. shared is a machine that also runs a control plane or an
|
||||
// agent: the database is given less of it, since a gateway swapped out to
|
||||
// give the database cache is worse than a smaller cache.
|
||||
func MemorySettings(total int64, shared bool) map[string]string {
|
||||
const mb = 1 << 20
|
||||
ram := total / mb
|
||||
// Shares of memory, in MB: the cache PostgreSQL keeps itself, what the
|
||||
// planner may count on the kernel caching besides, what building or
|
||||
// vacuuming an index may take, and what one sort may.
|
||||
buffers, cache, maint, sort := ram/4, ram*3/4, ram/16, ram/256
|
||||
if shared {
|
||||
buffers, cache, maint, sort = ram*3/20, ram/2, ram/32, ram/512
|
||||
}
|
||||
clamp := func(v, lo, hi int64) int64 { return min(max(v, lo), hi) }
|
||||
return map[string]string{
|
||||
"shared_buffers": strconv.FormatInt(clamp(buffers, 96, 32<<10), 10) + "MB",
|
||||
"effective_cache_size": strconv.FormatInt(clamp(cache, 256, 1<<20), 10) + "MB",
|
||||
"maintenance_work_mem": strconv.FormatInt(clamp(maint, 64, 2<<10), 10) + "MB",
|
||||
"work_mem": strconv.FormatInt(clamp(sort, 4, 64), 10) + "MB",
|
||||
}
|
||||
}
|
||||
|
||||
// MemTotal reads the machine's memory, in bytes, from /proc/meminfo.
|
||||
func MemTotal(meminfo string) (int64, bool) {
|
||||
for _, line := range strings.Split(meminfo, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 2 && f[0] == "MemTotal:" {
|
||||
kb, err := strconv.ParseInt(f[1], 10, 64)
|
||||
return kb << 10, err == nil && kb > 0
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// memory sizes each member to its machine, in its own configuration — the
|
||||
// machines need not be alike. What takes effect on reload is applied so;
|
||||
// shared_buffers needs a restart, given one member at a time, the leader
|
||||
// last, each waited on before the next.
|
||||
func (s *Setup) memory(ctx context.Context, sec Secrets) error {
|
||||
var restart []Node
|
||||
for _, d := range s.DB {
|
||||
out, err := d.Remote.Run(ctx, `cat /proc/meminfo
|
||||
if [ -f /etc/llmgw/server.yaml ] || [ -f /etc/llmgw/agent.yaml ]; then echo "llmgw-shared"; fi`, nil)
|
||||
total, ok := MemTotal(string(out))
|
||||
if err != nil || !ok {
|
||||
s.emit("memory", false, "%s: could not read its memory; PostgreSQL left as it is", d.Name)
|
||||
continue
|
||||
}
|
||||
shared := strings.Contains(string(out), "llmgw-shared")
|
||||
params := MemorySettings(total, shared)
|
||||
body, _ := json.Marshal(params)
|
||||
out, err = d.Remote.Run(ctx, `set -e
|
||||
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
|
||||
want=$(cat)
|
||||
r=$(sudo env WANT="$want" python3 - <<'PY'
|
||||
import json, os, yaml
|
||||
p = "/etc/patroni/config.yml"
|
||||
want = json.loads(os.environ["WANT"])
|
||||
c = yaml.safe_load(open(p))
|
||||
pg = c.setdefault("postgresql", {})
|
||||
params = pg.get("parameters") or {}
|
||||
pg["parameters"] = params
|
||||
if all(str(params.get(k)) == v for k, v in want.items()):
|
||||
print("same")
|
||||
else:
|
||||
restart = str(params.get("shared_buffers")) != want["shared_buffers"]
|
||||
params.update(want)
|
||||
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
|
||||
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
|
||||
print("restart" if restart else "changed")
|
||||
PY
|
||||
)
|
||||
if [ "$r" != same ]; then
|
||||
sudo chown root:postgres /etc/patroni/config.yml.new
|
||||
sudo chmod 640 /etc/patroni/config.yml.new
|
||||
sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
|
||||
sudo systemctl reload patroni
|
||||
fi
|
||||
echo "$r"`, body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
what := fmt.Sprintf("%s: %d MB of memory%s; shared_buffers %s, effective_cache_size %s, maintenance_work_mem %s, work_mem %s",
|
||||
d.Name, total>>20, map[bool]string{true: ", shared with llmgw"}[shared],
|
||||
params["shared_buffers"], params["effective_cache_size"], params["maintenance_work_mem"], params["work_mem"])
|
||||
switch r := strings.TrimSpace(lastLine(out)); r {
|
||||
case "same", "missing":
|
||||
s.emit("memory", false, "%s", what)
|
||||
case "restart":
|
||||
s.emit("memory", true, "%s", what)
|
||||
restart = append(restart, d)
|
||||
default:
|
||||
s.emit("memory", true, "%s", what)
|
||||
}
|
||||
}
|
||||
// Patroni takes a reload in its own time: restarted before it has written
|
||||
// the new configuration, PostgreSQL comes back on the old one. So each is
|
||||
// waited on until Patroni says a restart is pending — and a member found
|
||||
// pending, by a run that stopped before restarting it, is restarted too.
|
||||
pending := func(c Cluster, name string) bool {
|
||||
for _, m := range c.Members {
|
||||
if m.Name == name {
|
||||
return m.PendingRestart
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
var c Cluster
|
||||
err := s.waitFor(ctx, s.dbHosts(), func(got Cluster) bool {
|
||||
c = got
|
||||
for _, d := range restart {
|
||||
if !pending(got, d.Name) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("waiting for Patroni to take the new configuration: %w", err)
|
||||
}
|
||||
for _, d := range s.DB {
|
||||
if pending(c, d.Name) && !slices.ContainsFunc(restart, func(n Node) bool { return n.Name == d.Name }) {
|
||||
restart = append(restart, d)
|
||||
}
|
||||
}
|
||||
if len(restart) == 0 {
|
||||
return nil
|
||||
}
|
||||
// The leader last: its restart is the only one the control planes see,
|
||||
// a few seconds without writes, and by then the rest are back.
|
||||
slices.SortStableFunc(restart, func(a, b Node) int {
|
||||
al, bl := a.Name == c.Leader(), b.Name == c.Leader()
|
||||
switch {
|
||||
case al && !bl:
|
||||
return 1
|
||||
case bl && !al:
|
||||
return -1
|
||||
}
|
||||
return 0
|
||||
})
|
||||
for _, d := range restart {
|
||||
if _, err := restJSON(ctx, d.IP, http.MethodPost, "/restart", []byte(`{"restart_pending": true}`), sec.RestAPI, 2*time.Minute); err != nil {
|
||||
return fmt.Errorf("restarting PostgreSQL on %s for its new shared_buffers: %w", d.Name, err)
|
||||
}
|
||||
if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
|
||||
return fmt.Errorf("after restarting %s: %w", d.Name, err)
|
||||
}
|
||||
s.emit("memory", true, "%s: PostgreSQL restarted with the new shared_buffers", d.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -465,3 +465,97 @@ func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *tes
|
||||
t.Errorf("TLS asked of a cluster without it: %s", dsn)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgreSQLIsGivenAShareOfTheMachinesMemory(t *testing.T) {
|
||||
total, ok := MemTotal("MemTotal: 4009848 kB\nMemFree: 976000 kB\n")
|
||||
if !ok || total != 4009848<<10 {
|
||||
t.Fatalf("MemTotal %d %v", total, ok)
|
||||
}
|
||||
own := MemorySettings(total, false)
|
||||
if own["shared_buffers"] != "978MB" || own["effective_cache_size"] != "2936MB" ||
|
||||
own["maintenance_work_mem"] != "244MB" || own["work_mem"] != "15MB" {
|
||||
t.Errorf("a machine of its own: %v", own)
|
||||
}
|
||||
// Beside a control plane, less of it.
|
||||
if sh := MemorySettings(total, true); sh["shared_buffers"] != "587MB" || sh["work_mem"] != "7MB" {
|
||||
t.Errorf("a shared machine: %v", sh)
|
||||
}
|
||||
// Never below what PostgreSQL needs, never past what it can use.
|
||||
if tiny := MemorySettings(512<<20, true); tiny["shared_buffers"] != "96MB" || tiny["work_mem"] != "4MB" {
|
||||
t.Errorf("a small machine: %v", tiny)
|
||||
}
|
||||
if big := MemorySettings(1<<40, false); big["shared_buffers"] != "32768MB" || big["maintenance_work_mem"] != "2048MB" {
|
||||
t.Errorf("a large machine: %v", big)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewSharedBuffersRestartTheReplicasBeforeTheLeader(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var restarted []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodPost && r.URL.Path == "/restart" {
|
||||
if _, pass, _ := r.BasicAuth(); pass != "pw" {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
host, _, _ := strings.Cut(r.Host, ":")
|
||||
mu.Lock()
|
||||
restarted = append(restarted, host)
|
||||
mu.Unlock()
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
old := restPort
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "restart\n"}
|
||||
leader := &machine{name: "a", answers: answers}
|
||||
replica := &machine{name: "b", answers: answers}
|
||||
up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running", PendingRestart: true},
|
||||
{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
|
||||
s := Setup{
|
||||
DB: []Node{{Member{"a", "127.0.0.1"}, leader}, {Member{"b", "localhost"}, replica}},
|
||||
Status: func(context.Context, []string) (Cluster, error) { return up, nil },
|
||||
Wait: time.Second,
|
||||
}
|
||||
if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Join(restarted, " ") != "localhost 127.0.0.1" {
|
||||
t.Errorf("restarted %v; the replica first, then the leader", restarted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMemberLeftPendingARestartIsRestartedOnTheNextRun(t *testing.T) {
|
||||
// A run that stopped after writing the configuration, or restarted before
|
||||
// Patroni had taken it: nothing to change, but PostgreSQL still on the old.
|
||||
var restarted []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/restart" {
|
||||
host, _, _ := strings.Cut(r.Host, ":")
|
||||
restarted = append(restarted, host)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
old := restPort
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "same\n"}
|
||||
up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running"},
|
||||
{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
|
||||
s := Setup{
|
||||
DB: []Node{{Member{"a", "127.0.0.1"}, &machine{answers: answers}}, {Member{"b", "localhost"}, &machine{answers: answers}}},
|
||||
Status: func(context.Context, []string) (Cluster, error) { return up, nil },
|
||||
Wait: time.Second,
|
||||
}
|
||||
if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Join(restarted, " ") != "localhost" {
|
||||
t.Errorf("restarted %v; only the member pending", restarted)
|
||||
}
|
||||
}
|
||||
@@ -128,6 +128,9 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
|
||||
if err := s.ensurePatroniTLS(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("patroni api: %w", err)
|
||||
}
|
||||
if err := s.memory(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("memory: %w", err)
|
||||
}
|
||||
if err := s.database(ctx); err != nil {
|
||||
return "", fmt.Errorf("database: %w", err)
|
||||
}
|
||||
|
||||
@@ -19,6 +19,9 @@ type MemberState struct {
|
||||
State string `json:"state"`
|
||||
Host string `json:"host"`
|
||||
Timeline int `json:"timeline"`
|
||||
// PendingRestart is a parameter changed that PostgreSQL takes only on
|
||||
// restart.
|
||||
PendingRestart bool `json:"pending_restart"`
|
||||
// Lag is how far behind the leader a replica is, in bytes; -1 when
|
||||
// Patroni does not know, which is itself worth an alert.
|
||||
Lag int64 `json:"-"`
|
||||
|
||||
Loaded 100 of 186 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user