Compare commits
33
Commits
settings-e
...
v0.14.2
No files matched your search
+92
-7
@@ -569,9 +569,12 @@ portal is not done — the page shows; the console and ctl change.
|
||||
client tests against a fake admin API and the OpenAPI document;
|
||||
acceptance tests with Terraform and OpenTofu against the fake; one
|
||||
apply, empty plan, in-place change and destroy against a real `llmgw
|
||||
server` run by hand. Left: organizations, teams, projects, people and
|
||||
keys are not in the admin API (they are `llmgw org`/`users`/`keys`
|
||||
against the store), so not in the provider; a model's definition beyond
|
||||
server` run by hand. Organizations, teams, projects, keys and
|
||||
memberships are in the admin API since 2026-10-08 (`/api/v1/orgs`,
|
||||
`teams`, `projects`, `keys`, `users/{id}/memberships`) and in the
|
||||
provider (`llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key`,
|
||||
`llmgw_membership`); accounts are still made by sign-up, invitation or
|
||||
OIDC/SCIM. Left: a model's definition beyond
|
||||
scaling, class, owner and adapters is changed by replacing it (the API
|
||||
has no edit, and `resources` is not used yet); `canary`, `slo`,
|
||||
`scaling.schedule` and `prewarm`, `guard` role and `origin` are not in
|
||||
@@ -793,10 +796,19 @@ leader 30s later with /health answering throughout; rollback did the same.
|
||||
facility state from the store (it kept what it read at start).
|
||||
- Retires `appendGuardIfMissing` (server init): the import does it.
|
||||
|
||||
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06
|
||||
### E2. Generated files from settings (L) — firewall on/off done 2026-10-06, every machine 2026-10-09
|
||||
`firewall.enabled` in the cluster's settings reaches nodes over the agent
|
||||
stream and control planes through llmgw-firewall-sync.path. Left: the mirror
|
||||
machine and store members, the allowed set from membership, the other files.
|
||||
stream and control planes through llmgw-firewall-sync.path. Since
|
||||
2026-10-09 the mirror machine too (the leader PUTs /v1/firewall, the mirror
|
||||
writes `.firewall.json`, the same path unit applies it) and store members
|
||||
that run no llmgw (the leader keeps it in `cluster_state`, a root timer
|
||||
reads the member's own copy with psql); `firewall_networks` reach all of
|
||||
them, and the mirror lets in the network of every node and control plane
|
||||
the leader knows. Tried on the PoC: off and on, and a network added, on
|
||||
control planes, mirror, store member and nodes. Left: the allowed set
|
||||
replaced by membership rather than added to (a network taken out of the
|
||||
settings stays let in), the other files (the mirror's unit flags,
|
||||
keepalived, etcd and Patroni configuration).
|
||||
|
||||
Every machine runs the agent (root) with roles — node, control plane, mirror,
|
||||
store member — and the agent renders and applies what was written once by
|
||||
@@ -863,6 +875,12 @@ Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
|
||||
partial HNSW index per collection on its organization's partition
|
||||
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
|
||||
20k rows: partition pruned, that index used, policy applied.
|
||||
Later (scale, 2026-10-08): a partial GIN per collection too
|
||||
(`kb_tsv_<id>`, backfilled at open); `vec_type` per collection, `halfvec`
|
||||
on pgvector ≥ 0.7 (up to 4000 dims); `SET LOCAL hnsw.ef_search` to
|
||||
candidates(k) clamped 40..1000 (EXPLAIN: 40 rows without it, all with
|
||||
it); the text arm takes candidates by `ts_rank` in a subquery and scores
|
||||
only those.
|
||||
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
|
||||
(set with set_config per transaction), one role — the table owner — not
|
||||
a separate one. Chunks are deleted by the foreign-key cascade from their
|
||||
@@ -1005,7 +1023,7 @@ a request without the header, `none` turns it off, an explicit header with
|
||||
k=2 gives two sources and no more, llmgw-kb lists owners by path. Left: returned passages
|
||||
are not audited per request (the header and the kept request record them).
|
||||
|
||||
### F4. Adapter training (L, after F1–F3)
|
||||
### F4. Adapter training (L, after F1–F3) — done 2026-10-08 (branch `adapters-f4`)
|
||||
A run-to-completion job placed by the control plane like a replica and
|
||||
supervised by the agent like an engine, in a Python environment the agent
|
||||
installs (TRL + PEFT, QLoRA). Data: Q/A pairs generated from a collection by
|
||||
@@ -1013,9 +1031,76 @@ a cluster model through `/v1/batches`, reviewed by people. Output to the
|
||||
mirror, into the model's `adapters`, through canary and a quality suite.
|
||||
Runs when the scaling schedule frees a GPU; 8–14B dense bases on one card.
|
||||
Never train on a collection some users of the adapter may not read.
|
||||
Tried 2026-10-08 on the PoC, by hand: Qwen2.5-0.5B-Instruct (fp16) with a
|
||||
LoRA (r=16, every linear layer, 8.8M trainable), trained beside the 35B in
|
||||
cuda01's last 2.3 GB (peak 2158 MiB on the card, 64 s for 138 steps, PEFT in a
|
||||
venv over the vLLM engine's torch). Data: 90 Turkish Q/A pairs a cluster
|
||||
model wrote from the nine Demo/politikalar documents, through the gateway.
|
||||
On nine questions it had not seen, phrased otherwise: base 0/9, 6 epochs
|
||||
4/9, 4 epochs at a lower rate 2/9. Wrong answers mixed facts across
|
||||
documents (the meal card's amount, the office day). The same nine through
|
||||
retrieval: 9/9 with sources. So F4 is for format, vocabulary and tone, and
|
||||
facts come from F1–F3; a bigger base on a free card would do better but not
|
||||
change that. The training itself fits a small slot on a shared card, which
|
||||
the job design can rely on for small bases.
|
||||
|
||||
Built: `ctl kb dataset` (POST /api/v1/kb/collections/{id}/dataset, NDJSON
|
||||
with each pair's collection); train_jobs/train_artifacts (migrations 0037,
|
||||
0038 owner); internal/control/train (queue, one job per node, artifacts in
|
||||
1 MiB pieces, a node that goes fails its job); internal/agent/train (the
|
||||
newest vLLM's python3 + `uv pip install --no-deps --target` peft 0.21.2 and
|
||||
accelerate 1.15.0; llama.cpp's conversion code for the installed release,
|
||||
untarred from its source; the card with the most free memory, if enough;
|
||||
runs as llmgw-engine); `ctl train start|list|show --follow|cancel|rm|
|
||||
download|publish`; the mirror's PUT /v1/models/{model}/{version}/{file};
|
||||
publish uploads the GGUF (llama.cpp) or PEFT files (vLLM) and sets the
|
||||
adapter after checkAdapters. Owners: catalog.Adapter.Owner, enforced in the
|
||||
router (404, hidden from /v1/models); a job's owner defaults to its single
|
||||
collection's owner, must be able to read every collection the pairs came
|
||||
from, and pairs from a knowledge base are never public. Tried on the PoC:
|
||||
dataset of 90 pairs in 89 s; a job trained beside the 35B (peak 2214 MiB,
|
||||
138 steps in about a minute) and sent back PEFT and GGUF.
|
||||
Left: a job dies with its agent's stream (no resume); no console screen for
|
||||
training yet; quality gating of a published adapter is by hand (ctl eval).
|
||||
|
||||
---
|
||||
|
||||
### F5. Large imports (M) — measured 2026-10-09, open
|
||||
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
|
||||
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
|
||||
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
|
||||
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||||
at a third of its CPUs.
|
||||
- An embedding model's profile with many slots by default (`--parallel`
|
||||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||||
longest chunk), or `ctl model serve` choosing that for a model with
|
||||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||||
the text and vector go up again (~11 KB).
|
||||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||||
0.7) and a larger store machine.
|
||||
- Done: a batch of embeddings checked text by text against the context
|
||||
(branch `embeddings-batch`); it was refused as one over-long prompt.
|
||||
- Done: `query_prefix` on an embedding model, before questions only.
|
||||
- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575
|
||||
tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache,
|
||||
FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner
|
||||
pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import,
|
||||
and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks
|
||||
took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU
|
||||
profile on llama.cpp, which needs a source per profile (the catalog has
|
||||
one per model; the import swapped the entry by hand), and an admission
|
||||
queue limit that suits bulk imports (30 s turned some away).
|
||||
- Retrieval on 40 questions written from the corpus, document in the top
|
||||
5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40
|
||||
(18/21), with its query prefix 36/40 and first place 31/40 instead of
|
||||
28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the
|
||||
chat model, and a reranker.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.
|
||||
|
||||
+150
-1
@@ -18,7 +18,7 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
@@ -26,6 +26,100 @@ APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
## v0.14.2 — 2026-10-10
|
||||
|
||||
Pictures: OpenAI's `/v1/images/generations`, made by ComfyUI on the
|
||||
cluster's own cards. The agents install ComfyUI themselves, the gateway
|
||||
prices pictures by the picture, and the portal has a page to make them.
|
||||
Also: on-demand models that survive a failover, and knowledge-base
|
||||
embeddings fixes.
|
||||
|
||||
**Upgrading.**
|
||||
- The store gains a column (`usage.images`); nothing to do.
|
||||
- **ComfyUI is installed only where asked**: `llmgw ctl config set
|
||||
engines.comfyui 0.39.2` has every node with an NVIDIA GPU build it (about
|
||||
10 GB under `/opt/llmgw/engines`). Nodes need this release to install it
|
||||
or run engine `comfyui`.
|
||||
|
||||
### Added
|
||||
|
||||
- **Pictures at `/v1/images/generations`**, OpenAI's shape, through the same
|
||||
key, allowlist, limit, admission and guardrail checks as any request. A
|
||||
model with profiles on engine `comfyui` is placed on a GPU node like any
|
||||
other. A ComfyUI server outside the cluster can be an external model with
|
||||
`api: comfyui` instead. The workflow is a built-in (`flux-schnell`, the
|
||||
default, or `flux-dev`) or a JSON file in ComfyUI's API format, with
|
||||
`{{prompt}}`, `{{width}}`, `{{seed}}` and the like where the request's
|
||||
values go. Requests may also send `seed`, `steps`, `negative_prompt` and
|
||||
`guidance`. The seed used comes back in `x-llmgw-seed`. Pictures come back
|
||||
as `b64_json`; ComfyUI's workflow is stripped from each PNG, and a client
|
||||
that leaves takes its workflow off the queue. `/v1/models` says `images`
|
||||
for such a model.
|
||||
- **Engine `comfyui`.** ComfyUI asks for no key, so the agent starts it
|
||||
behind `llmgw comfyui-gate`. ComfyUI listens on the loopback only. The
|
||||
gate listens on the replica's port, demands the replica's key, and lets
|
||||
through only what the router asks. The model's `comfyui:` block names
|
||||
the workflow and, for a source that is a set of files, the checkpoint.
|
||||
- **The agents install ComfyUI** (`engines.comfyui`). The release's source is
|
||||
checked against a digest llmgw holds, since GitHub publishes none, and a
|
||||
version without one is refused. Its requirements are resolved as PyPI
|
||||
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
|
||||
CUDA 12.9 one for an older driver, and the result is checked on the card.
|
||||
A ComfyUI installed by hand is named by `engines.comfyui.dir` in
|
||||
agent.yaml.
|
||||
- **A price per picture**: `per_image` in `auth.pricing`, on top of the
|
||||
prompt's tokens. A model priced only per picture is not free, so a spent
|
||||
budget still stops it. Each usage row keeps how many pictures were made,
|
||||
shown by `llmgw ctl request`.
|
||||
- **`/images` on the portal**, beside the chat, for the models that make
|
||||
pictures. The chat no longer offers them. The page offers size, how many,
|
||||
seed, steps and a negative prompt, and shows each picture with its seed
|
||||
and a download link. Nothing is kept but the usage.
|
||||
- **An embedding model's `query_prefix`**, put before the question a
|
||||
knowledge-base search embeds and never before documents, for Qwen3-Embedding
|
||||
and models trained like it.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **An awake on-demand model survived no restart or change of leader of the
|
||||
control plane.** A control plane that had heard no requests read every
|
||||
on-demand replica its nodes reported as wanted by nobody, and drained them
|
||||
all. An adopted replica now gets a whole idle timeout.
|
||||
- `ctl models` said `not placed` for an on-demand model that had been woken
|
||||
and was serving; it now says `ready (on demand)`.
|
||||
- A batch of embeddings was refused as one over-long prompt when the
|
||||
passages together exceeded the context, though each one fit. The longest
|
||||
input is what has to fit.
|
||||
- The chat's kept conversation sat in the browser under one key for
|
||||
everybody, so the next person to sign in on the same browser opened the
|
||||
last one's conversation. It is kept per account, and signing out clears
|
||||
what the pages stored.
|
||||
|
||||
## v0.14.1 — 2026-10-09
|
||||
|
||||
Adapters trained on the nodes from a knowledge base and kept private to
|
||||
their owners; the cluster's and the agents' settings edited in one place;
|
||||
the tenancy tree and keys over the admin API and in Terraform; knowledge
|
||||
bases that hold up with many documents; the firewall setting on every
|
||||
machine.
|
||||
|
||||
**Upgrading.**
|
||||
- **Nodes** need this release to follow `agents.*` in the cluster's
|
||||
settings; until a node is upgraded its agent.yaml is what it runs on.
|
||||
- **A separate mirror machine and store members that run nothing else**
|
||||
follow the firewall setting once what reads it is installed: run `sudo
|
||||
llmgw mirror init` on the mirror again with the flags it was set up with,
|
||||
and `llmgw server init --patroni` again for the store's members. A control
|
||||
plane set up before v0.12.0 needs `sudo llmgw firewall sync --install`.
|
||||
- **The store's PostgreSQL memory** applies after each member is restarted,
|
||||
one at a time, the leader last; until then it runs as before.
|
||||
- The store gains tables and columns; nothing to do. Existing knowledge-base
|
||||
collections get their full-text index when a control plane starts, and
|
||||
keep `vector` storage; only new collections use `halfvec`, and only on
|
||||
pgvector 0.7 or later.
|
||||
- Cached knowledge-base passages in vLLM are made again once, under the new
|
||||
`cache_salt`.
|
||||
|
||||
### Added
|
||||
|
||||
- **The cluster's settings on the portal** (BACKLOG E4): `/admin/settings`
|
||||
@@ -41,6 +135,61 @@ APIs change between minors without a deprecation period.
|
||||
that would not load are refused on save, and set aside on a node rather
|
||||
than keep its agent from starting. Nodes need this release to follow
|
||||
them.
|
||||
- **LoRA adapters trained on the nodes** (BACKLOG F4): `llmgw ctl kb dataset`
|
||||
writes question/answer pairs from a collection with a model of the
|
||||
cluster. `llmgw ctl train start` trains an adapter on a node with its
|
||||
installed vLLM's torch and a small overlay, on a card with enough memory
|
||||
free (beside an engine, if it leaves that much), converting to GGUF for
|
||||
llama.cpp. `llmgw ctl train publish` keeps it on the mirror (which now
|
||||
accepts uploads) and makes it the model's adapter.
|
||||
- **Adapters with owners.** An adapter's `owner` (org, team, project, user
|
||||
or key) is the only part of the tree that sees it; to anyone else it is a
|
||||
model that does not exist. An adapter trained on a knowledge base belongs
|
||||
to a part of the tree that may read it, and is never made public.
|
||||
- The knowledge bases' vLLM `cache_salt` follows the collections searched,
|
||||
not the organization, so two teams of one organization no longer share
|
||||
cached passages.
|
||||
- **Organizations, teams, projects, API keys and memberships in the admin
|
||||
API**: `/api/v1/orgs` (the tree), `POST /api/v1/teams` and `projects`,
|
||||
`PATCH` to rename or move, `DELETE` for what is empty; `/api/v1/keys` to
|
||||
issue (the key is in the answer, once), list and revoke; `/api/v1/users`
|
||||
and `/api/v1/users/{id}/memberships`. They were only `llmgw org`, `keys`
|
||||
and `users` on a control plane before. The Terraform provider manages them
|
||||
as `llmgw_org`, `llmgw_team`, `llmgw_project`, `llmgw_key` (the key in a
|
||||
sensitive `secret`) and `llmgw_membership`.
|
||||
- **The firewall setting reaches every machine.** A separate mirror and the
|
||||
store's members follow `firewall.enabled` and `firewall_networks` as nodes
|
||||
and control planes did: the leading control plane tells the mirror over
|
||||
its API, and keeps the state in the store for members that run no llmgw,
|
||||
which a timer on each reads. The mirror also lets in the network of every
|
||||
node the leader knows. `mirror init` and `server init --patroni` install
|
||||
what follows it; run them again on a cluster set up before.
|
||||
|
||||
### Changed
|
||||
|
||||
- **The store's PostgreSQL is sized to its machine's memory.** Before, it ran
|
||||
on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
|
||||
machine. Now each member gets a share of its own RAM, a smaller one where
|
||||
a control plane or agent shares the machine, and the share is read again
|
||||
every time `server init --patroni` runs. The new cache is applied by
|
||||
restarting one member at a time, the leader last.
|
||||
- **Knowledge-base searches hold up with many documents.** New collections
|
||||
store their index as `halfvec` where pgvector is 0.7 or later (half the
|
||||
memory, and embeddings up to 4000 dimensions instead of 2000); older
|
||||
collections, and clusters on pgvector 0.6, keep `vector`. Each collection
|
||||
has its own full-text index, as it has its own vector index, made for
|
||||
existing collections when the control plane starts. The text arm ranks by
|
||||
the cheap measure first and scores only those candidates.
|
||||
|
||||
### Fixed
|
||||
|
||||
- A knowledge-base search asking for more than 10 passages
|
||||
(`X-Llmgw-Knowledge-K`) got at most 40 candidates per collection from the
|
||||
vector index, whatever it asked for: pgvector's `hnsw.ef_search` is now
|
||||
raised to the number wanted, up to 1000.
|
||||
- `llmgw server backup` and `restore` against a replicated store over TLS
|
||||
failed looking for `~/.postgresql/root.crt`: pg_dump was given
|
||||
`sslmode=verify-full` without the cluster's authority.
|
||||
|
||||
## v0.14.0 — 2026-10-08
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
|
||||
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
|
||||
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
|
||||
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI) |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
@@ -50,6 +50,13 @@ budgets and rate limits decide who may use any of it.
|
||||
|
||||
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
|
||||
|
||||
- **Adapters, and the tenancy tree over the API** (v0.14.1). `llmgw ctl
|
||||
kb dataset` and `llmgw ctl train` make a LoRA adapter from a collection
|
||||
on a node's spare GPU memory, kept on the mirror and seen only by its
|
||||
owner. Organizations, teams, projects, keys and memberships are in the
|
||||
admin API and the Terraform provider. The agents' settings are edited
|
||||
with the cluster's, on the portal too, and the firewall setting reaches
|
||||
every machine.
|
||||
- **Knowledge bases** (v0.14.0). People put a company's documents in from
|
||||
their own machines with `llmgw-kb`, a separate download for Windows,
|
||||
macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
|
||||
|
||||
+502
-71
@@ -106,6 +106,8 @@ type AgentMessage struct {
|
||||
// *AgentMessage_FabricTest
|
||||
// *AgentMessage_Upgrade
|
||||
// *AgentMessage_KvEvents
|
||||
// *AgentMessage_TrainStatus
|
||||
// *AgentMessage_TrainArtifact
|
||||
Msg isAgentMessage_Msg `protobuf_oneof:"msg"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
@@ -220,6 +222,24 @@ func (x *AgentMessage) GetKvEvents() *KVEvents {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetTrainStatus() *TrainStatus {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_TrainStatus); ok {
|
||||
return x.TrainStatus
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *AgentMessage) GetTrainArtifact() *TrainArtifact {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*AgentMessage_TrainArtifact); ok {
|
||||
return x.TrainArtifact
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type isAgentMessage_Msg interface {
|
||||
isAgentMessage_Msg()
|
||||
}
|
||||
@@ -256,6 +276,14 @@ type AgentMessage_KvEvents struct {
|
||||
KvEvents *KVEvents `protobuf:"bytes,8,opt,name=kv_events,json=kvEvents,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_TrainStatus struct {
|
||||
TrainStatus *TrainStatus `protobuf:"bytes,9,opt,name=train_status,json=trainStatus,proto3,oneof"`
|
||||
}
|
||||
|
||||
type AgentMessage_TrainArtifact struct {
|
||||
TrainArtifact *TrainArtifact `protobuf:"bytes,10,opt,name=train_artifact,json=trainArtifact,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*AgentMessage_Register) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_Heartbeat) isAgentMessage_Msg() {}
|
||||
@@ -272,6 +300,10 @@ func (*AgentMessage_Upgrade) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_KvEvents) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainStatus) isAgentMessage_Msg() {}
|
||||
|
||||
func (*AgentMessage_TrainArtifact) isAgentMessage_Msg() {}
|
||||
|
||||
// KVEvents is a change to what one vLLM replica's prefix cache holds, read
|
||||
// by the agent from the engine's KV cache events and sent on for the router's
|
||||
// precise index (plan §7.5 tier 2).
|
||||
@@ -392,6 +424,8 @@ type ControlMessage struct {
|
||||
// *ControlMessage_Firewall
|
||||
// *ControlMessage_RestartAgent
|
||||
// *ControlMessage_AgentSettings
|
||||
// *ControlMessage_TrainStart
|
||||
// *ControlMessage_TrainCancel
|
||||
Msg isControlMessage_Msg `protobuf_oneof:"msg"`
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -566,6 +600,24 @@ func (x *ControlMessage) GetAgentSettings() *AgentSettings {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrainStart() *TrainStart {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_TrainStart); ok {
|
||||
return x.TrainStart
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrainCancel() *TrainCancel {
|
||||
if x != nil {
|
||||
if x, ok := x.Msg.(*ControlMessage_TrainCancel); ok {
|
||||
return x.TrainCancel
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *ControlMessage) GetTrace() map[string]string {
|
||||
if x != nil {
|
||||
return x.Trace
|
||||
@@ -633,6 +685,14 @@ type ControlMessage_AgentSettings struct {
|
||||
AgentSettings *AgentSettings `protobuf:"bytes,15,opt,name=agent_settings,json=agentSettings,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_TrainStart struct {
|
||||
TrainStart *TrainStart `protobuf:"bytes,16,opt,name=train_start,json=trainStart,proto3,oneof"`
|
||||
}
|
||||
|
||||
type ControlMessage_TrainCancel struct {
|
||||
TrainCancel *TrainCancel `protobuf:"bytes,17,opt,name=train_cancel,json=trainCancel,proto3,oneof"`
|
||||
}
|
||||
|
||||
func (*ControlMessage_Start) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_Stop) isControlMessage_Msg() {}
|
||||
@@ -661,6 +721,10 @@ func (*ControlMessage_RestartAgent) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_AgentSettings) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainStart) isControlMessage_Msg() {}
|
||||
|
||||
func (*ControlMessage_TrainCancel) isControlMessage_Msg() {}
|
||||
|
||||
// RestartAgent has the agent start again, as llmgw restart asks of every
|
||||
// node: its engines keep serving and are taken over again.
|
||||
type RestartAgent struct {
|
||||
@@ -4699,11 +4763,335 @@ func (x *LoraAdapter) GetSource() string {
|
||||
return ""
|
||||
}
|
||||
|
||||
// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
|
||||
// weights from its registry, on question/answer pairs, with a Python trainer
|
||||
// the agent carries, in an environment it makes over an installed engine's.
|
||||
// One at a time per node.
|
||||
type TrainStart struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
// base_repo and base_revision name the base model's weights on its
|
||||
// registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
|
||||
BaseRepo string `protobuf:"bytes,2,opt,name=base_repo,json=baseRepo,proto3" json:"base_repo,omitempty"`
|
||||
BaseRevision string `protobuf:"bytes,3,opt,name=base_revision,json=baseRevision,proto3" json:"base_revision,omitempty"`
|
||||
// data is the training pairs, NDJSON of {"q", "a"}.
|
||||
Data []byte `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`
|
||||
// params is the trainer's settings as JSON: rank, alpha, epochs, lr,
|
||||
// batch, max_len.
|
||||
Params string `protobuf:"bytes,5,opt,name=params,proto3" json:"params,omitempty"`
|
||||
// gpu_memory_mib is what the job needs of a card; the agent starts it
|
||||
// only when one has that much free, which lets it run beside an engine.
|
||||
GpuMemoryMib int32 `protobuf:"varint,6,opt,name=gpu_memory_mib,json=gpuMemoryMib,proto3" json:"gpu_memory_mib,omitempty"`
|
||||
// gguf asks for the adapter in llama.cpp's format too.
|
||||
Gguf bool `protobuf:"varint,7,opt,name=gguf,proto3" json:"gguf,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainStart) Reset() {
|
||||
*x = TrainStart{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[44]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainStart) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainStart) ProtoMessage() {}
|
||||
|
||||
func (x *TrainStart) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[44]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainStart.ProtoReflect.Descriptor instead.
|
||||
func (*TrainStart) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{44}
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetBaseRepo() string {
|
||||
if x != nil {
|
||||
return x.BaseRepo
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetBaseRevision() string {
|
||||
if x != nil {
|
||||
return x.BaseRevision
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetData() []byte {
|
||||
if x != nil {
|
||||
return x.Data
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetParams() string {
|
||||
if x != nil {
|
||||
return x.Params
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetGpuMemoryMib() int32 {
|
||||
if x != nil {
|
||||
return x.GpuMemoryMib
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStart) GetGguf() bool {
|
||||
if x != nil {
|
||||
return x.Gguf
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// TrainCancel stops a training job.
|
||||
type TrainCancel struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainCancel) Reset() {
|
||||
*x = TrainCancel{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[45]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainCancel) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainCancel) ProtoMessage() {}
|
||||
|
||||
func (x *TrainCancel) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[45]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainCancel.ProtoReflect.Descriptor instead.
|
||||
func (*TrainCancel) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{45}
|
||||
}
|
||||
|
||||
func (x *TrainCancel) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// TrainStatus is how a training job is going, sent as it changes.
|
||||
type TrainStatus struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
// state is preparing, running, done or failed.
|
||||
State string `protobuf:"bytes,2,opt,name=state,proto3" json:"state,omitempty"`
|
||||
Step int32 `protobuf:"varint,3,opt,name=step,proto3" json:"step,omitempty"`
|
||||
Steps int32 `protobuf:"varint,4,opt,name=steps,proto3" json:"steps,omitempty"`
|
||||
Loss float64 `protobuf:"fixed64,5,opt,name=loss,proto3" json:"loss,omitempty"`
|
||||
// message is what is being done, or why it failed.
|
||||
Message string `protobuf:"bytes,6,opt,name=message,proto3" json:"message,omitempty"`
|
||||
// peak_mib is the most the job held of the card.
|
||||
PeakMib int32 `protobuf:"varint,7,opt,name=peak_mib,json=peakMib,proto3" json:"peak_mib,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainStatus) Reset() {
|
||||
*x = TrainStatus{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[46]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainStatus) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainStatus) ProtoMessage() {}
|
||||
|
||||
func (x *TrainStatus) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[46]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainStatus.ProtoReflect.Descriptor instead.
|
||||
func (*TrainStatus) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{46}
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetState() string {
|
||||
if x != nil {
|
||||
return x.State
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetStep() int32 {
|
||||
if x != nil {
|
||||
return x.Step
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetSteps() int32 {
|
||||
if x != nil {
|
||||
return x.Steps
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetLoss() float64 {
|
||||
if x != nil {
|
||||
return x.Loss
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetMessage() string {
|
||||
if x != nil {
|
||||
return x.Message
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainStatus) GetPeakMib() int32 {
|
||||
if x != nil {
|
||||
return x.PeakMib
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// TrainArtifact is a piece of what a training job made, sent before its
|
||||
// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
|
||||
// adapter (kind gguf).
|
||||
type TrainArtifact struct {
|
||||
state protoimpl.MessageState `protogen:"open.v1"`
|
||||
JobId string `protobuf:"bytes,1,opt,name=job_id,json=jobId,proto3" json:"job_id,omitempty"`
|
||||
Kind string `protobuf:"bytes,2,opt,name=kind,proto3" json:"kind,omitempty"`
|
||||
Seq int32 `protobuf:"varint,3,opt,name=seq,proto3" json:"seq,omitempty"`
|
||||
Data []byte `protobuf:"bytes,4,opt,name=data,proto3" json:"data,omitempty"`
|
||||
Last bool `protobuf:"varint,5,opt,name=last,proto3" json:"last,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) Reset() {
|
||||
*x = TrainArtifact{}
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[47]
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) String() string {
|
||||
return protoimpl.X.MessageStringOf(x)
|
||||
}
|
||||
|
||||
func (*TrainArtifact) ProtoMessage() {}
|
||||
|
||||
func (x *TrainArtifact) ProtoReflect() protoreflect.Message {
|
||||
mi := &file_llmgw_v1_agent_proto_msgTypes[47]
|
||||
if x != nil {
|
||||
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
|
||||
if ms.LoadMessageInfo() == nil {
|
||||
ms.StoreMessageInfo(mi)
|
||||
}
|
||||
return ms
|
||||
}
|
||||
return mi.MessageOf(x)
|
||||
}
|
||||
|
||||
// Deprecated: Use TrainArtifact.ProtoReflect.Descriptor instead.
|
||||
func (*TrainArtifact) Descriptor() ([]byte, []int) {
|
||||
return file_llmgw_v1_agent_proto_rawDescGZIP(), []int{47}
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetJobId() string {
|
||||
if x != nil {
|
||||
return x.JobId
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetKind() string {
|
||||
if x != nil {
|
||||
return x.Kind
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetSeq() int32 {
|
||||
if x != nil {
|
||||
return x.Seq
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetData() []byte {
|
||||
if x != nil {
|
||||
return x.Data
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (x *TrainArtifact) GetLast() bool {
|
||||
if x != nil {
|
||||
return x.Last
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
var File_llmgw_v1_agent_proto protoreflect.FileDescriptor
|
||||
|
||||
const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb9\x03\n" +
|
||||
"\x14llmgw/v1/agent.proto\x12\bllmgw.v1\"\xb7\x04\n" +
|
||||
"\fAgentMessage\x120\n" +
|
||||
"\bregister\x18\x01 \x01(\v2\x12.llmgw.v1.RegisterH\x00R\bregister\x123\n" +
|
||||
"\theartbeat\x18\x02 \x01(\v2\x13.llmgw.v1.HeartbeatH\x00R\theartbeat\x12-\n" +
|
||||
@@ -4714,7 +5102,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\vfabric_test\x18\x06 \x01(\v2\x19.llmgw.v1.FabricTestReplyH\x00R\n" +
|
||||
"fabricTest\x122\n" +
|
||||
"\aupgrade\x18\a \x01(\v2\x16.llmgw.v1.UpgradeReplyH\x00R\aupgrade\x121\n" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEventsB\x05\n" +
|
||||
"\tkv_events\x18\b \x01(\v2\x12.llmgw.v1.KVEventsH\x00R\bkvEvents\x12:\n" +
|
||||
"\ftrain_status\x18\t \x01(\v2\x15.llmgw.v1.TrainStatusH\x00R\vtrainStatus\x12@\n" +
|
||||
"\x0etrain_artifact\x18\n" +
|
||||
" \x01(\v2\x17.llmgw.v1.TrainArtifactH\x00R\rtrainArtifactB\x05\n" +
|
||||
"\x03msg\"\xb5\x01\n" +
|
||||
"\bKVEvents\x12\x1d\n" +
|
||||
"\n" +
|
||||
@@ -4725,7 +5116,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"clearFirst\x12\x16\n" +
|
||||
"\x06stored\x18\x04 \x03(\x06R\x06stored\x12\x18\n" +
|
||||
"\aremoved\x18\x05 \x03(\x06R\aremoved\x12\x18\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\x83\a\n" +
|
||||
"\aproblem\x18\x06 \x01(\tR\aproblem\"\xf8\a\n" +
|
||||
"\x0eControlMessage\x12,\n" +
|
||||
"\x05start\x18\x01 \x01(\v2\x14.llmgw.v1.StartShardH\x00R\x05start\x12)\n" +
|
||||
"\x04stop\x18\x02 \x01(\v2\x13.llmgw.v1.StopShardH\x00R\x04stop\x12,\n" +
|
||||
@@ -4743,7 +5134,10 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x0econtrol_planes\x18\v \x01(\v2\x17.llmgw.v1.ControlPlanesH\x00R\rcontrolPlanes\x120\n" +
|
||||
"\bfirewall\x18\r \x01(\v2\x12.llmgw.v1.FirewallH\x00R\bfirewall\x12=\n" +
|
||||
"\rrestart_agent\x18\x0e \x01(\v2\x16.llmgw.v1.RestartAgentH\x00R\frestartAgent\x12@\n" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x129\n" +
|
||||
"\x0eagent_settings\x18\x0f \x01(\v2\x17.llmgw.v1.AgentSettingsH\x00R\ragentSettings\x127\n" +
|
||||
"\vtrain_start\x18\x10 \x01(\v2\x14.llmgw.v1.TrainStartH\x00R\n" +
|
||||
"trainStart\x12:\n" +
|
||||
"\ftrain_cancel\x18\x11 \x01(\v2\x15.llmgw.v1.TrainCancelH\x00R\vtrainCancel\x129\n" +
|
||||
"\x05trace\x18\f \x03(\v2#.llmgw.v1.ControlMessage.TraceEntryR\x05trace\x1a8\n" +
|
||||
"\n" +
|
||||
"TraceEntry\x12\x10\n" +
|
||||
@@ -5150,7 +5544,32 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x05error\x18\x03 \x01(\tR\x05error\"9\n" +
|
||||
"\vLoraAdapter\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06source\x18\x02 \x01(\tR\x06source2O\n" +
|
||||
"\x06source\x18\x02 \x01(\tR\x06source\"\xcb\x01\n" +
|
||||
"\n" +
|
||||
"TrainStart\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x1b\n" +
|
||||
"\tbase_repo\x18\x02 \x01(\tR\bbaseRepo\x12#\n" +
|
||||
"\rbase_revision\x18\x03 \x01(\tR\fbaseRevision\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x16\n" +
|
||||
"\x06params\x18\x05 \x01(\tR\x06params\x12$\n" +
|
||||
"\x0egpu_memory_mib\x18\x06 \x01(\x05R\fgpuMemoryMib\x12\x12\n" +
|
||||
"\x04gguf\x18\a \x01(\bR\x04gguf\"$\n" +
|
||||
"\vTrainCancel\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\"\xad\x01\n" +
|
||||
"\vTrainStatus\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x14\n" +
|
||||
"\x05state\x18\x02 \x01(\tR\x05state\x12\x12\n" +
|
||||
"\x04step\x18\x03 \x01(\x05R\x04step\x12\x14\n" +
|
||||
"\x05steps\x18\x04 \x01(\x05R\x05steps\x12\x12\n" +
|
||||
"\x04loss\x18\x05 \x01(\x01R\x04loss\x12\x18\n" +
|
||||
"\amessage\x18\x06 \x01(\tR\amessage\x12\x19\n" +
|
||||
"\bpeak_mib\x18\a \x01(\x05R\apeakMib\"t\n" +
|
||||
"\rTrainArtifact\x12\x15\n" +
|
||||
"\x06job_id\x18\x01 \x01(\tR\x05jobId\x12\x12\n" +
|
||||
"\x04kind\x18\x02 \x01(\tR\x04kind\x12\x10\n" +
|
||||
"\x03seq\x18\x03 \x01(\x05R\x03seq\x12\x12\n" +
|
||||
"\x04data\x18\x04 \x01(\fR\x04data\x12\x12\n" +
|
||||
"\x04last\x18\x05 \x01(\bR\x04last2O\n" +
|
||||
"\fAgentService\x12?\n" +
|
||||
"\aConnect\x12\x16.llmgw.v1.AgentMessage\x1a\x18.llmgw.v1.ControlMessage(\x010\x01B\x93\x01\n" +
|
||||
"\fcom.llmgw.v1B\n" +
|
||||
@@ -5169,7 +5588,7 @@ func file_llmgw_v1_agent_proto_rawDescGZIP() []byte {
|
||||
}
|
||||
|
||||
var file_llmgw_v1_agent_proto_enumTypes = make([]protoimpl.EnumInfo, 1)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 52)
|
||||
var file_llmgw_v1_agent_proto_msgTypes = make([]protoimpl.MessageInfo, 56)
|
||||
var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(ShardStatus_State)(0), // 0: llmgw.v1.ShardStatus.State
|
||||
(*AgentMessage)(nil), // 1: llmgw.v1.AgentMessage
|
||||
@@ -5216,14 +5635,18 @@ var file_llmgw_v1_agent_proto_goTypes = []any{
|
||||
(*UpgradeAgent)(nil), // 42: llmgw.v1.UpgradeAgent
|
||||
(*UpgradeReply)(nil), // 43: llmgw.v1.UpgradeReply
|
||||
(*LoraAdapter)(nil), // 44: llmgw.v1.LoraAdapter
|
||||
nil, // 45: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 46: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 47: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 48: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 49: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 50: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 51: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 52: llmgw.v1.EngineProfile.EnvEntry
|
||||
(*TrainStart)(nil), // 45: llmgw.v1.TrainStart
|
||||
(*TrainCancel)(nil), // 46: llmgw.v1.TrainCancel
|
||||
(*TrainStatus)(nil), // 47: llmgw.v1.TrainStatus
|
||||
(*TrainArtifact)(nil), // 48: llmgw.v1.TrainArtifact
|
||||
nil, // 49: llmgw.v1.ControlMessage.TraceEntry
|
||||
nil, // 50: llmgw.v1.Register.LabelsEntry
|
||||
nil, // 51: llmgw.v1.Register.EngineVersionsEntry
|
||||
nil, // 52: llmgw.v1.GPU.NicPathsEntry
|
||||
nil, // 53: llmgw.v1.IBPort.CountersEntry
|
||||
nil, // 54: llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
nil, // 55: llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
nil, // 56: llmgw.v1.EngineProfile.EnvEntry
|
||||
}
|
||||
var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
8, // 0: llmgw.v1.AgentMessage.register:type_name -> llmgw.v1.Register
|
||||
@@ -5234,62 +5657,66 @@ var file_llmgw_v1_agent_proto_depIdxs = []int32{
|
||||
41, // 5: llmgw.v1.AgentMessage.fabric_test:type_name -> llmgw.v1.FabricTestReply
|
||||
43, // 6: llmgw.v1.AgentMessage.upgrade:type_name -> llmgw.v1.UpgradeReply
|
||||
2, // 7: llmgw.v1.AgentMessage.kv_events:type_name -> llmgw.v1.KVEvents
|
||||
30, // 8: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 9: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 10: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 11: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 12: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 13: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 14: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 15: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 16: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 17: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 18: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 19: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 20: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 21: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 22: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
46, // 23: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 24: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 25: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
47, // 26: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 27: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 28: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 29: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 30: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 31: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 32: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 33: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
48, // 34: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 35: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 36: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 37: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
49, // 38: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 39: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 40: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 41: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
50, // 42: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 43: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 44: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 45: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 46: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 47: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 48: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 49: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
51, // 50: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 51: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 52: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 53: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 54: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
52, // 55: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 56: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 57: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 58: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
58, // [58:59] is the sub-list for method output_type
|
||||
57, // [57:58] is the sub-list for method input_type
|
||||
57, // [57:57] is the sub-list for extension type_name
|
||||
57, // [57:57] is the sub-list for extension extendee
|
||||
0, // [0:57] is the sub-list for field type_name
|
||||
47, // 8: llmgw.v1.AgentMessage.train_status:type_name -> llmgw.v1.TrainStatus
|
||||
48, // 9: llmgw.v1.AgentMessage.train_artifact:type_name -> llmgw.v1.TrainArtifact
|
||||
30, // 10: llmgw.v1.ControlMessage.start:type_name -> llmgw.v1.StartShard
|
||||
33, // 11: llmgw.v1.ControlMessage.stop:type_name -> llmgw.v1.StopShard
|
||||
34, // 12: llmgw.v1.ControlMessage.drain:type_name -> llmgw.v1.DrainShard
|
||||
35, // 13: llmgw.v1.ControlMessage.ack:type_name -> llmgw.v1.Ack
|
||||
32, // 14: llmgw.v1.ControlMessage.remove_weights:type_name -> llmgw.v1.RemoveWeights
|
||||
24, // 15: llmgw.v1.ControlMessage.provision:type_name -> llmgw.v1.Provision
|
||||
36, // 16: llmgw.v1.ControlMessage.logs:type_name -> llmgw.v1.LogsRequest
|
||||
38, // 17: llmgw.v1.ControlMessage.fabric_scan:type_name -> llmgw.v1.FabricScanRequest
|
||||
40, // 18: llmgw.v1.ControlMessage.fabric_test:type_name -> llmgw.v1.FabricTestRequest
|
||||
42, // 19: llmgw.v1.ControlMessage.upgrade:type_name -> llmgw.v1.UpgradeAgent
|
||||
7, // 20: llmgw.v1.ControlMessage.control_planes:type_name -> llmgw.v1.ControlPlanes
|
||||
5, // 21: llmgw.v1.ControlMessage.firewall:type_name -> llmgw.v1.Firewall
|
||||
4, // 22: llmgw.v1.ControlMessage.restart_agent:type_name -> llmgw.v1.RestartAgent
|
||||
6, // 23: llmgw.v1.ControlMessage.agent_settings:type_name -> llmgw.v1.AgentSettings
|
||||
45, // 24: llmgw.v1.ControlMessage.train_start:type_name -> llmgw.v1.TrainStart
|
||||
46, // 25: llmgw.v1.ControlMessage.train_cancel:type_name -> llmgw.v1.TrainCancel
|
||||
49, // 26: llmgw.v1.ControlMessage.trace:type_name -> llmgw.v1.ControlMessage.TraceEntry
|
||||
50, // 27: llmgw.v1.Register.labels:type_name -> llmgw.v1.Register.LabelsEntry
|
||||
12, // 28: llmgw.v1.Register.gpus:type_name -> llmgw.v1.GPU
|
||||
14, // 29: llmgw.v1.Register.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
51, // 30: llmgw.v1.Register.engine_versions:type_name -> llmgw.v1.Register.EngineVersionsEntry
|
||||
11, // 31: llmgw.v1.Register.running_shards:type_name -> llmgw.v1.RunningShard
|
||||
15, // 32: llmgw.v1.Register.weights:type_name -> llmgw.v1.LocalWeights
|
||||
10, // 33: llmgw.v1.Register.storage:type_name -> llmgw.v1.Storage
|
||||
9, // 34: llmgw.v1.Register.host:type_name -> llmgw.v1.Host
|
||||
21, // 35: llmgw.v1.Register.imex:type_name -> llmgw.v1.IMEX
|
||||
0, // 36: llmgw.v1.RunningShard.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
27, // 37: llmgw.v1.RunningShard.compute:type_name -> llmgw.v1.Compute
|
||||
52, // 38: llmgw.v1.GPU.nic_paths:type_name -> llmgw.v1.GPU.NicPathsEntry
|
||||
13, // 39: llmgw.v1.GPU.mig:type_name -> llmgw.v1.MIGDevice
|
||||
18, // 40: llmgw.v1.NodeLoad.shards:type_name -> llmgw.v1.ShardLoad
|
||||
17, // 41: llmgw.v1.NodeLoad.gpus:type_name -> llmgw.v1.GPUSensor
|
||||
53, // 42: llmgw.v1.IBPort.counters:type_name -> llmgw.v1.IBPort.CountersEntry
|
||||
20, // 43: llmgw.v1.IBPort.subnet_managers:type_name -> llmgw.v1.SubnetManager
|
||||
10, // 44: llmgw.v1.Heartbeat.storage:type_name -> llmgw.v1.Storage
|
||||
15, // 45: llmgw.v1.Heartbeat.weights:type_name -> llmgw.v1.LocalWeights
|
||||
54, // 46: llmgw.v1.Heartbeat.engine_versions:type_name -> llmgw.v1.Heartbeat.EngineVersionsEntry
|
||||
26, // 47: llmgw.v1.Heartbeat.provision:type_name -> llmgw.v1.ProvisionStatus
|
||||
16, // 48: llmgw.v1.Heartbeat.load:type_name -> llmgw.v1.NodeLoad
|
||||
19, // 49: llmgw.v1.Heartbeat.ib_ports:type_name -> llmgw.v1.IBPort
|
||||
14, // 50: llmgw.v1.Heartbeat.fabrics:type_name -> llmgw.v1.FabricAttachment
|
||||
22, // 51: llmgw.v1.Heartbeat.gpu_fabric:type_name -> llmgw.v1.GPUFabricState
|
||||
21, // 52: llmgw.v1.Heartbeat.imex:type_name -> llmgw.v1.IMEX
|
||||
25, // 53: llmgw.v1.Provision.engines:type_name -> llmgw.v1.EngineWant
|
||||
55, // 54: llmgw.v1.EngineWant.offline_sha256:type_name -> llmgw.v1.EngineWant.OfflineSha256Entry
|
||||
0, // 55: llmgw.v1.ShardStatus.state:type_name -> llmgw.v1.ShardStatus.State
|
||||
29, // 56: llmgw.v1.ShardStatus.fetch:type_name -> llmgw.v1.FetchProgress
|
||||
27, // 57: llmgw.v1.ShardStatus.compute:type_name -> llmgw.v1.Compute
|
||||
31, // 58: llmgw.v1.StartShard.profile:type_name -> llmgw.v1.EngineProfile
|
||||
56, // 59: llmgw.v1.EngineProfile.env:type_name -> llmgw.v1.EngineProfile.EnvEntry
|
||||
44, // 60: llmgw.v1.EngineProfile.adapters:type_name -> llmgw.v1.LoraAdapter
|
||||
1, // 61: llmgw.v1.AgentService.Connect:input_type -> llmgw.v1.AgentMessage
|
||||
3, // 62: llmgw.v1.AgentService.Connect:output_type -> llmgw.v1.ControlMessage
|
||||
62, // [62:63] is the sub-list for method output_type
|
||||
61, // [61:62] is the sub-list for method input_type
|
||||
61, // [61:61] is the sub-list for extension type_name
|
||||
61, // [61:61] is the sub-list for extension extendee
|
||||
0, // [0:61] is the sub-list for field type_name
|
||||
}
|
||||
|
||||
func init() { file_llmgw_v1_agent_proto_init() }
|
||||
@@ -5306,6 +5733,8 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*AgentMessage_FabricTest)(nil),
|
||||
(*AgentMessage_Upgrade)(nil),
|
||||
(*AgentMessage_KvEvents)(nil),
|
||||
(*AgentMessage_TrainStatus)(nil),
|
||||
(*AgentMessage_TrainArtifact)(nil),
|
||||
}
|
||||
file_llmgw_v1_agent_proto_msgTypes[2].OneofWrappers = []any{
|
||||
(*ControlMessage_Start)(nil),
|
||||
@@ -5322,6 +5751,8 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
(*ControlMessage_Firewall)(nil),
|
||||
(*ControlMessage_RestartAgent)(nil),
|
||||
(*ControlMessage_AgentSettings)(nil),
|
||||
(*ControlMessage_TrainStart)(nil),
|
||||
(*ControlMessage_TrainCancel)(nil),
|
||||
}
|
||||
type x struct{}
|
||||
out := protoimpl.TypeBuilder{
|
||||
@@ -5329,7 +5760,7 @@ func file_llmgw_v1_agent_proto_init() {
|
||||
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
|
||||
RawDescriptor: unsafe.Slice(unsafe.StringData(file_llmgw_v1_agent_proto_rawDesc), len(file_llmgw_v1_agent_proto_rawDesc)),
|
||||
NumEnums: 1,
|
||||
NumMessages: 52,
|
||||
NumMessages: 56,
|
||||
NumExtensions: 0,
|
||||
NumServices: 1,
|
||||
},
|
||||
|
||||
@@ -28,6 +28,8 @@ message AgentMessage {
|
||||
FabricTestReply fabric_test = 6;
|
||||
UpgradeReply upgrade = 7;
|
||||
KVEvents kv_events = 8;
|
||||
TrainStatus train_status = 9;
|
||||
TrainArtifact train_artifact = 10;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -74,6 +76,8 @@ message ControlMessage {
|
||||
Firewall firewall = 13;
|
||||
RestartAgent restart_agent = 14;
|
||||
AgentSettings agent_settings = 15;
|
||||
TrainStart train_start = 16;
|
||||
TrainCancel train_cancel = 17;
|
||||
}
|
||||
// trace is the W3C trace context (traceparent, and tracestate when there
|
||||
// is one) of the control plane's span that sent this command, so what the
|
||||
@@ -884,3 +888,55 @@ message LoraAdapter {
|
||||
string name = 1;
|
||||
string source = 2;
|
||||
}
|
||||
|
||||
// TrainStart has a node train a LoRA adapter (BACKLOG F4): on a base model's
|
||||
// weights from its registry, on question/answer pairs, with a Python trainer
|
||||
// the agent carries, in an environment it makes over an installed engine's.
|
||||
// One at a time per node.
|
||||
message TrainStart {
|
||||
string job_id = 1;
|
||||
// base_repo and base_revision name the base model's weights on its
|
||||
// registry (Hugging Face): safetensors, not the GGUF a catalog may serve.
|
||||
string base_repo = 2;
|
||||
string base_revision = 3;
|
||||
// data is the training pairs, NDJSON of {"q", "a"}.
|
||||
bytes data = 4;
|
||||
// params is the trainer's settings as JSON: rank, alpha, epochs, lr,
|
||||
// batch, max_len.
|
||||
string params = 5;
|
||||
// gpu_memory_mib is what the job needs of a card; the agent starts it
|
||||
// only when one has that much free, which lets it run beside an engine.
|
||||
int32 gpu_memory_mib = 6;
|
||||
// gguf asks for the adapter in llama.cpp's format too.
|
||||
bool gguf = 7;
|
||||
}
|
||||
|
||||
// TrainCancel stops a training job.
|
||||
message TrainCancel {
|
||||
string job_id = 1;
|
||||
}
|
||||
|
||||
// TrainStatus is how a training job is going, sent as it changes.
|
||||
message TrainStatus {
|
||||
string job_id = 1;
|
||||
// state is preparing, running, done or failed.
|
||||
string state = 2;
|
||||
int32 step = 3;
|
||||
int32 steps = 4;
|
||||
double loss = 5;
|
||||
// message is what is being done, or why it failed.
|
||||
string message = 6;
|
||||
// peak_mib is the most the job held of the card.
|
||||
int32 peak_mib = 7;
|
||||
}
|
||||
|
||||
// TrainArtifact is a piece of what a training job made, sent before its
|
||||
// final status: a gzipped tar of the PEFT adapter (kind peft), or the GGUF
|
||||
// adapter (kind gguf).
|
||||
message TrainArtifact {
|
||||
string job_id = 1;
|
||||
string kind = 2;
|
||||
int32 seq = 3;
|
||||
bytes data = 4;
|
||||
bool last = 5;
|
||||
}
|
||||
+16
-7
@@ -8,8 +8,9 @@ description is.
|
||||
anything they do can be done with `curl`, a generated client or the
|
||||
[Terraform provider](../terraform-provider-llmgw/README.md). It adds and
|
||||
removes models, scales them, cordons and upgrades nodes, sets capacity
|
||||
quotas, attaches guardrail policies, and reads the fleet, usage and the audit
|
||||
log.
|
||||
quotas, attaches guardrail policies, adds organizations, teams and projects,
|
||||
issues and revokes API keys, puts people in the tree, and reads the fleet,
|
||||
usage and the audit log.
|
||||
|
||||
## Reaching it
|
||||
|
||||
@@ -99,6 +100,14 @@ curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2,
|
||||
# A team's capacity quota.
|
||||
curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'
|
||||
|
||||
# An organization, a team in it and a project in that; then a key for the
|
||||
# project. The answer holds the key once: only its hash is kept.
|
||||
curl -s "${H[@]}" -X POST $API/orgs -d '{"name": "Acme"}'
|
||||
curl -s "${H[@]}" -X POST $API/teams -d '{"name": "ml", "in": "Acme"}'
|
||||
curl -s "${H[@]}" -X POST $API/projects -d '{"name": "search", "in": "Acme/ml"}'
|
||||
curl -s "${H[@]}" -X POST $API/keys -d '{"name": "ci", "project": "Acme/ml/search", "expires_in": "720h"}'
|
||||
curl -s "${H[@]}" -X POST $API/users/ayse@example.com/memberships -d '{"target": "team:Acme/ml"}'
|
||||
|
||||
# Remove the model; a DELETE carries the content type too.
|
||||
curl -s "${H[@]}" -X DELETE $API/models/qwen-small
|
||||
```
|
||||
@@ -149,11 +158,11 @@ go generate ./internal/docgen
|
||||
|
||||
## What it does not cover
|
||||
|
||||
- **Organizations, teams, projects, people and API keys.** These live in the
|
||||
store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
|
||||
control plane, or in the web console's `/admin` pages on the client port
|
||||
behind a session. The admin API reads the tree only where a quota or a
|
||||
policy names a part of it.
|
||||
- **Making people's accounts.** People sign up, are invited, or come from
|
||||
the identity provider (OIDC, SCIM); the API lists them and puts them in
|
||||
the tree, and `llmgw users` approves, disables and deletes them. Key
|
||||
lifetime limits and spending limits are `llmgw keys ttl` and `llmgw keys
|
||||
limits`.
|
||||
- **The client-facing API** (`/v1/chat/completions` and the rest), which is
|
||||
OpenAI's and Anthropic's, on `listen`, with API keys.
|
||||
- **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
+160
-10
@@ -201,6 +201,16 @@ this by running the same command again, on the control plane that holds the
|
||||
authority: etcd is rebuilt on TLS with Patroni in maintenance mode, so
|
||||
PostgreSQL serves throughout and no failover happens.
|
||||
|
||||
Each member's PostgreSQL is sized to the machine's memory, read each time
|
||||
the command runs: a quarter of it as cache (`shared_buffers`), three quarters
|
||||
counted on for the planner, and room to sort and build indexes, all capped. A
|
||||
member that also runs a control plane or an agent gets less: about 15% as
|
||||
cache, so the gateway is not pushed into swap. A knowledge base's vector
|
||||
index is searched well only from memory, so give a store machine more RAM and
|
||||
run the command again. The new cache takes a PostgreSQL restart, which Patroni
|
||||
does one member at a time, the leader last. The leader's restart costs a few
|
||||
seconds without writes, and there is no failover.
|
||||
|
||||
Every step looks before it acts. Running it again finishes an install that
|
||||
stopped halfway and changes nothing on one that did not; it never overwrites
|
||||
`server.yaml`, the authority, or an existing account's password.
|
||||
@@ -1260,6 +1270,128 @@ prices and budgets, rate limits, usage records, and key allowlists.
|
||||
whose allowlist leaves the external model out cannot reach it, as a fallback or
|
||||
by name. The cluster page and `ctl models` mark it `external`.
|
||||
|
||||
### Pictures from ComfyUI
|
||||
|
||||
Clients ask for pictures at OpenAI's `/v1/images/generations`. llmgw has them
|
||||
made by ComfyUI: it fills a ComfyUI workflow with the request, queues it,
|
||||
waits, and sends the pictures back as `b64_json`.
|
||||
|
||||
**On the cluster's own cards.** Use engine `comfyui`. The model is placed on a
|
||||
GPU node like any other, started when asked for, and stopped when unused.
|
||||
|
||||
The node must offer the engine. Ask for it in the cluster's settings, and every
|
||||
node with an NVIDIA GPU installs it:
|
||||
|
||||
```
|
||||
llmgw ctl config set engines.comfyui 0.39.2
|
||||
```
|
||||
|
||||
The agent fetches ComfyUI's source from GitHub and checks it against the
|
||||
digest llmgw holds for that version. GitHub publishes none for a tag's archive,
|
||||
so a version llmgw does not know is refused. The agent then builds a Python
|
||||
environment beside the source, under `/opt/llmgw/engines/comfyui/`. Its
|
||||
dependencies are resolved as PyPI stood the day after the release. torch comes
|
||||
from PyPI (CUDA 13), or from PyTorch's CUDA 12.9 build where the driver is older
|
||||
than 580. The install is checked on the node's GPU, and takes about 10 GB.
|
||||
`engine_overrides` can keep it off nodes that should not have it.
|
||||
|
||||
Alternatively, install ComfyUI by hand. You need a directory with `main.py` and
|
||||
its environment in `.venv`, readable by the user engines run as
|
||||
(`llmgw-engine`). Name it in that node's `agent.yaml`; a node with one named
|
||||
there is not given the installed one:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
comfyui:
|
||||
dir: /opt/comfyui # python: defaults to /opt/comfyui/.venv/bin/python
|
||||
```
|
||||
|
||||
Then add the model:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors
|
||||
comfyui:
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
scaling: {min: 0, max: 1, idle_timeout: 15m}
|
||||
profiles:
|
||||
- name: gpu
|
||||
engine: comfyui
|
||||
gpu_class: 6000-24g
|
||||
tp: 1
|
||||
```
|
||||
|
||||
The source is fetched as for any model. It can be one file, such as the
|
||||
single-file checkpoints the built-in workflows load. It can also be a mirror
|
||||
set (a URL ending in `/`) for a model in several files, such as a diffusion
|
||||
model, its text encoders and its VAE. ComfyUI sees every file of a set by its
|
||||
own name, so a workflow of your own can load each one. A set must name the
|
||||
file that `{{checkpoint}}` stands for in `comfyui.checkpoint`. A single file
|
||||
is named by its own file name.
|
||||
|
||||
ComfyUI asks nobody for a key. The agent therefore starts it behind a gate
|
||||
(`llmgw comfyui-gate`). ComfyUI listens on the node's loopback only. The gate
|
||||
listens on the replica's port, demands the replica's key, and lets through only
|
||||
what the router asks: queueing, waiting, fetching a picture and cancelling.
|
||||
Uploads and node installs are refused. Each replica has its own directory,
|
||||
`~llmgw-engine/comfyui/REPLICA`, and ComfyUI's output can be read with `llmgw
|
||||
ctl logs NODE --source SHARD`. A profile's `args` go to `main.py`.
|
||||
`--disable-smart-memory` frees the card between pictures, at the cost of loading
|
||||
the model again for each one.
|
||||
|
||||
**On a ComfyUI server outside the cluster.** Make it an external model with
|
||||
`api: comfyui`:
|
||||
|
||||
```yaml
|
||||
- name: flux-schnell
|
||||
source: external
|
||||
external:
|
||||
url: http://10.0.0.147:8188 # ComfyUI, started with --listen
|
||||
model: flux1-schnell-fp8.safetensors # the checkpoint, as ComfyUI lists it
|
||||
api: comfyui
|
||||
workflow: flux-schnell # or flux-dev, or a JSON file
|
||||
```
|
||||
|
||||
Like any external model, its requests count as leaving the cluster.
|
||||
|
||||
`flux-schnell` (the default) and `flux-dev` are built in. They suit the
|
||||
single-file FLUX checkpoints that ComfyUI loads with `CheckpointLoaderSimple`.
|
||||
For any other model, export the workflow from ComfyUI with **Save (API)** and
|
||||
name the file. In the file, put placeholders where the request's values go:
|
||||
|
||||
- `{{prompt}}` and `{{negative_prompt}}` are the prompts.
|
||||
- `{{width}}` and `{{height}}` are the size.
|
||||
- `{{n}}` is the batch size.
|
||||
- `{{seed}}` is the seed.
|
||||
- `{{steps|4}}` and `{{guidance|3.5}}` take the value after the `|` when the
|
||||
request leaves them out.
|
||||
- `{{checkpoint}}` is the `model` above.
|
||||
|
||||
A request is OpenAI's: `prompt`, `n` (at most 4), `size` (each side a multiple
|
||||
of 16, from 256 to 2048) and `response_format: b64_json`. Beyond those, it may
|
||||
send `seed`, `steps`, `negative_prompt` and `guidance`. The seed used comes
|
||||
back in `x-llmgw-seed`. `url` is not offered, because pictures are kept
|
||||
nowhere. ComfyUI also writes the workflow into each PNG; llmgw strips it
|
||||
before the picture is sent.
|
||||
|
||||
Requests for pictures pass the same checks as any other request: keys,
|
||||
allowlists, limits, admission, and the guardrails on the prompt. A client that
|
||||
leaves takes its workflow off the ComfyUI queue.
|
||||
|
||||
Each request's usage row records how many pictures it made (`llmgw ctl request
|
||||
ID` shows them). A picture's price is set with `per_image`, in micro-units like
|
||||
the other prices, and is charged on top of the prompt's tokens:
|
||||
|
||||
```yaml
|
||||
auth:
|
||||
pricing:
|
||||
flux-schnell: {prompt_per_1k: 0, per_image: 20000}
|
||||
```
|
||||
|
||||
An external ComfyUI on a card that llmgw also places models on should be
|
||||
started with `--disable-smart-memory`. Without it, ComfyUI keeps the model on
|
||||
the card after each picture, and llmgw sees that memory as somebody else's.
|
||||
|
||||
### Federating clusters
|
||||
|
||||
Another llmgw cluster — another site, another team's GPUs — can be a provider
|
||||
@@ -1736,9 +1868,11 @@ the file again.
|
||||
|
||||
On a machine that also runs the control plane, PostgreSQL's defaults assume it
|
||||
owns the box. A gateway swapped out to give the database cache is a worse
|
||||
outcome than a smaller cache, so cap it: `shared_buffers = 96MB` and
|
||||
`max_connections = 40` are enough for a store whose working set is measured in
|
||||
megabytes.
|
||||
outcome than a smaller cache, so size it to the machine. `server init` does this
|
||||
for a PostgreSQL it installs: `shared_buffers` about 15% of memory (at least
|
||||
96MB), `effective_cache_size` half, `work_mem` and `maintenance_work_mem`
|
||||
scaled to it, and `max_connections = 40`. A store with a large knowledge base
|
||||
wants a machine of its own, where its vector index fits in memory.
|
||||
|
||||
## Tagging requests
|
||||
|
||||
@@ -2184,13 +2318,29 @@ llmgw ctl config set firewall.enabled false # every machine's table taken out
|
||||
llmgw ctl config set firewall.enabled true # and put back as it was
|
||||
```
|
||||
|
||||
It is in force within seconds: each node's agent takes its table out or puts
|
||||
it back, and each control plane's `llmgw-firewall-sync.path` (installed by
|
||||
`server init` and `server add`; `sudo llmgw firewall sync --install` on one
|
||||
set up before) does the same there. The rules files stay, so turning it back
|
||||
on restores them. Nodes added while it is off get none. A separate mirror
|
||||
machine and the store's members keep theirs for now. `firewall_networks`
|
||||
changes reach the nodes the same way.
|
||||
It is in force on every machine within a minute, each in its own way:
|
||||
|
||||
- **nodes**: the agent takes its table out or puts it back;
|
||||
- **control planes**: `llmgw-firewall-sync.path`, installed by `server
|
||||
init` and `server add` (`sudo llmgw firewall sync --install` on one set up
|
||||
before);
|
||||
- **a separate mirror machine**: the leading control plane tells the mirror,
|
||||
which writes it to `.firewall.json` in its directory, and the same path
|
||||
unit applies it. `mirror init` installs it; on a mirror set up before, run
|
||||
`sudo llmgw mirror init` again with the same flags;
|
||||
- **store members that are nothing else**: they run no llmgw, so
|
||||
`llmgw-firewall-follow.timer` reads what the leading control plane keeps in
|
||||
the store, from the member's own copy, once a minute. `server init
|
||||
--patroni` installs it; run it again on a cluster set up before.
|
||||
|
||||
The rules files stay, so turning it back on restores them. Nodes added while
|
||||
it is off get none. Networks added to `firewall_networks` reach every machine
|
||||
the same way. One taken out of it stays let in until it is taken out of
|
||||
`/etc/llmgw/firewall.nft` by hand: the rules are only ever added to.
|
||||
|
||||
The leading control plane also tells the mirror machine where every node and
|
||||
control plane is, so a node on a network the mirror had not let in can fetch
|
||||
its weights without a `firewall_networks` entry.
|
||||
|
||||
Only those ports are touched. The table's policy is to accept, so whatever
|
||||
else the machine serves — and whatever firewall it already has — is left as
|
||||
|
||||
+62
-3
@@ -45,6 +45,20 @@ is available to everyone who may use them.
|
||||
A collection stays with the model it was made with: vectors from two
|
||||
models cannot be compared.
|
||||
|
||||
Some embedding models are trained with an instruction before the
|
||||
question and none before the documents. Give it as the entry's
|
||||
`query_prefix`; llmgw puts it before questions only. For
|
||||
Qwen3-Embedding:
|
||||
|
||||
```yaml
|
||||
query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
|
||||
```
|
||||
|
||||
With pgvector 0.7 or later a collection's index is kept at half
|
||||
precision (`halfvec`), which halves its memory and takes models of up to
|
||||
4000 dimensions; with an older pgvector the limit is 2000. A collection
|
||||
keeps the kind it was made with.
|
||||
|
||||
## For administrators
|
||||
|
||||
### Make a collection
|
||||
@@ -317,9 +331,11 @@ Behaviour to know about:
|
||||
(Codex, Claude Code) work the same way, since they are answered as chat
|
||||
completions.
|
||||
- **Separate prefix caches.** On vLLM replicas, a request given passages
|
||||
carries a `cache_salt` derived from its organization. One organization's
|
||||
passages then never share cached blocks with another's, so how fast an
|
||||
answer starts says nothing about what another tenant asked.
|
||||
carries a `cache_salt` derived from the collections they came from.
|
||||
Requests that could be given the same passages still share cached blocks,
|
||||
and no one else's do: not another organization's, nor another team's of
|
||||
the same one. How fast an answer starts therefore says nothing about what
|
||||
someone else asked.
|
||||
- **No answer cache.** Answers from knowledge bases are never served from
|
||||
the answer cache, because two keys of one project can read different
|
||||
collections.
|
||||
@@ -340,6 +356,49 @@ a product code or a name.
|
||||
Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
|
||||
`llmgw_kb_retrieval_seconds`.
|
||||
|
||||
## Training an adapter on a collection
|
||||
|
||||
An adapter (LoRA) changes how a model *writes*: the organisation's words,
|
||||
format and tone. It does not reliably teach facts. On the PoC, a small
|
||||
model trained on nine policy documents answered 4 of 9 rephrased questions
|
||||
right and mixed facts up on the rest, while the same questions through the
|
||||
knowledge base got 9 of 9. Use both: the knowledge base for what answers
|
||||
say, an adapter for how they sound.
|
||||
|
||||
```sh
|
||||
# 1. Questions and answers from the collection, written by a model of the
|
||||
# cluster. Read the file through and correct it.
|
||||
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl
|
||||
|
||||
# 2. Train on a node. --gpu-memory is what the job needs of a card; it
|
||||
# runs beside an engine when the engine leaves that much free.
|
||||
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
|
||||
--base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
|
||||
llmgw ctl train show JOB --follow
|
||||
|
||||
# 3. Make it the model's adapter, kept on the mirror.
|
||||
llmgw ctl train publish JOB --restart
|
||||
```
|
||||
|
||||
The node trains with its installed vLLM's torch and a small overlay (peft
|
||||
and accelerate), as the engines' user. For a model llama.cpp serves, it also
|
||||
converts the adapter to GGUF with the installed llama.cpp's own code. What
|
||||
it makes is kept by the control plane (`llmgw ctl train download`).
|
||||
|
||||
An adapter is private like its collection:
|
||||
|
||||
- **It has an owner.** An adapter trained on a collection carries some of
|
||||
what the collection says in its weights, so it belongs to a part of the
|
||||
tree.
|
||||
- By default the owner is the collection's owner.
|
||||
- `--owner` names another, which must be able to read every collection
|
||||
the pairs came from.
|
||||
- Only callers in the owner's part of the tree see the adapter. To anyone
|
||||
else it is a model that does not exist, absent from `/v1/models` too.
|
||||
- **It is public only by choice.** `--public` makes an adapter every
|
||||
caller's. It is refused for pairs from a knowledge base, so it is only for
|
||||
pairs written by hand.
|
||||
|
||||
## How it is kept apart
|
||||
|
||||
Passages live in PostgreSQL, partitioned by organization, with an index per
|
||||
|
||||
@@ -18,6 +18,7 @@ rather than repeated.
|
||||
| `name` | `string` | [Adapter](#adapter) |
|
||||
| `source` | `string` | [Adapter](#adapter) |
|
||||
| `scale` | `float64` | [Adapter](#adapter) |
|
||||
| `owner` | `string` | [Adapter](#adapter) |
|
||||
| `enabled` | `bool` | [Admission](#admission) |
|
||||
| `bands` | `[]Band` | [Admission](#admission) |
|
||||
| `default_band` | `string` | [Admission](#admission) |
|
||||
@@ -59,6 +60,11 @@ rather than repeated.
|
||||
| `tools` | `bool` | [Capabilities](#capabilities) |
|
||||
| `vision` | `bool` | [Capabilities](#capabilities) |
|
||||
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
||||
| `images` | `bool` | [Capabilities](#capabilities) |
|
||||
| `workflow` | `string` | [ComfyUI](#comfyui) |
|
||||
| `checkpoint` | `string` | [ComfyUI](#comfyui) |
|
||||
| `dir` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `python` | `string` | [ComfyUIConfig](#comfyuiconfig) |
|
||||
| `interval` | `time.Duration` | [Control](#control) |
|
||||
| `base_backoff` | `time.Duration` | [Control](#control) |
|
||||
| `max_backoff` | `time.Duration` | [Control](#control) |
|
||||
@@ -75,6 +81,7 @@ rather than repeated.
|
||||
| `api_key` | `string` | [Endpoint](#endpoint) |
|
||||
| `models` | `[]string` | [Endpoint](#endpoint) |
|
||||
| `engine` | `string` | [Endpoint](#endpoint) |
|
||||
| `workflow` | `string` | [Endpoint](#endpoint) |
|
||||
| `pool` | `string` | [Endpoint](#endpoint) |
|
||||
| `weight` | `float64` | [Endpoint](#endpoint) |
|
||||
| `priority` | `int` | [Endpoint](#endpoint) |
|
||||
@@ -89,6 +96,8 @@ rather than repeated.
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
| `peer` | `string` | [External](#external) |
|
||||
| `api` | `string` | [External](#external) |
|
||||
| `workflow` | `string` | [External](#external) |
|
||||
| `budgets` | `[]FacilityBudget` | [Facility](#facility) |
|
||||
| `margin` | `float64` | [Facility](#facility) |
|
||||
| `gpu_watts` | `float64` | [Facility](#facility) |
|
||||
@@ -134,6 +143,7 @@ rather than repeated.
|
||||
| `class` | `string` | [Model](#model) |
|
||||
| `owner` | `string` | [Model](#model) |
|
||||
| `external` | `*External` | [Model](#model) |
|
||||
| `comfyui` | `*ComfyUI` | [Model](#model) |
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `max_loaded_adapters` | `int` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
@@ -144,6 +154,7 @@ rather than repeated.
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
| `max_output` | `int` | [Model](#model) |
|
||||
| `capabilities` | `Capabilities` | [Model](#model) |
|
||||
| `query_prefix` | `string` | [Model](#model) |
|
||||
| `role` | `string` | [Model](#model) |
|
||||
| `guard` | `*GuardModel` | [Model](#model) |
|
||||
| `origin` | `*Origin` | [Model](#model) |
|
||||
@@ -341,6 +352,7 @@ rather than repeated.
|
||||
| `secret_file` | `string` | [Webhook](#webhook) |
|
||||
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `per_image` | `int64` | [accounting.Price](#accountingprice) |
|
||||
| `name` | `string` | [admission.Band](#admissionband) |
|
||||
| `priority` | `int` | [admission.Band](#admissionband) |
|
||||
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
||||
@@ -375,12 +387,14 @@ rather than repeated.
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
||||
| `per_image` | `int64` | [config.Price](#configprice) |
|
||||
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
|
||||
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -426,6 +440,16 @@ weights are.
|
||||
|
||||
Scale is how strongly it is applied; zero takes 1.
|
||||
|
||||
### `owner`
|
||||
|
||||
`string`
|
||||
|
||||
Owner is the part of the tenancy tree the adapter belongs to —
|
||||
org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
|
||||
it see it: to anyone else it is a model that does not exist. An
|
||||
adapter trained on an organisation's documents carries some of what
|
||||
they say in its weights. Empty is every caller the model allows.
|
||||
|
||||
## Admission
|
||||
|
||||
### `enabled`
|
||||
@@ -704,6 +728,47 @@ token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
||||
|
||||
`bool`
|
||||
|
||||
### `images`
|
||||
|
||||
`bool`
|
||||
|
||||
Images is a model that makes pictures, asked for at
|
||||
/v1/images/generations. Implied by an external model whose API is
|
||||
ComfyUI's.
|
||||
|
||||
## ComfyUI
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is a built-in's name (flux-schnell, the default, or
|
||||
flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
format.
|
||||
|
||||
### `checkpoint`
|
||||
|
||||
`string`
|
||||
|
||||
Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
source's own name unless set, which a source that is a set of files
|
||||
must.
|
||||
|
||||
## ComfyUIConfig
|
||||
|
||||
### `dir`
|
||||
|
||||
`string`
|
||||
|
||||
Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
engines run as.
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
|
||||
## Control
|
||||
|
||||
### `interval`
|
||||
@@ -817,6 +882,13 @@ with one.
|
||||
Engine names the inference engine, which decides what its metrics are
|
||||
called. Empty means vLLM's names.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
name or its JSON; flux-schnell unless set.
|
||||
|
||||
### `pool`
|
||||
|
||||
`string`
|
||||
@@ -908,6 +980,23 @@ a model that federation added (`llmgw ctl federation add`) rather
|
||||
than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
goes with the peer's own list, and is sent only while the peer is up.
|
||||
|
||||
### `api`
|
||||
|
||||
`string`
|
||||
|
||||
API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
sent the workflow below with the request's prompt in it. Model is then
|
||||
the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
|
||||
### `workflow`
|
||||
|
||||
`string`
|
||||
|
||||
Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
(flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
plane.
|
||||
|
||||
## Facility
|
||||
|
||||
### `budgets`
|
||||
@@ -1294,6 +1383,15 @@ allowlist.
|
||||
|
||||
Takes a [External](#external) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`*ComfyUI`
|
||||
|
||||
ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
pictures: the workflow, and the file it loads.
|
||||
|
||||
Takes a [ComfyUI](#comfyui) block.
|
||||
|
||||
### `adapters`
|
||||
|
||||
`[]Adapter`
|
||||
@@ -1378,6 +1476,16 @@ Capabilities is what this model can do beyond plain completion.
|
||||
|
||||
Takes a [Capabilities](#capabilities) block.
|
||||
|
||||
### `query_prefix`
|
||||
|
||||
`string`
|
||||
|
||||
QueryPrefix is put before a question an embedding model embeds for a
|
||||
knowledge-base search; documents are embedded without it. Models
|
||||
trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
"Instruct: Given a web search query, retrieve relevant passages that
|
||||
answer the query\nQuery: " for one.
|
||||
|
||||
### `role`
|
||||
|
||||
`string`
|
||||
@@ -2878,6 +2986,14 @@ is sha256=<hex HMAC of the body>.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is what one picture costs, for a model that makes them: a
|
||||
picture's cost is the card's time, which its prompt's few tokens do
|
||||
not measure.
|
||||
|
||||
## admission.Band
|
||||
|
||||
### `name`
|
||||
@@ -3120,6 +3236,13 @@ Takes a [Tracing](#tracing) block.
|
||||
|
||||
`int64`
|
||||
|
||||
### `per_image`
|
||||
|
||||
`int64`
|
||||
|
||||
PerImage is the cost of one picture, for a model that makes them
|
||||
(/v1/images/generations), on top of its prompt's tokens.
|
||||
|
||||
## engines.Config
|
||||
|
||||
### `disabled`
|
||||
@@ -3163,6 +3286,14 @@ Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
Takes a [SimConfig](#simconfig) block.
|
||||
|
||||
### `comfyui`
|
||||
|
||||
`ComfyUIConfig`
|
||||
|
||||
ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
|
||||
Takes a [ComfyUIConfig](#comfyuiconfig) block.
|
||||
|
||||
## guard.Config
|
||||
|
||||
### `enabled`
|
||||
|
||||
@@ -323,7 +323,7 @@ func (a *Accountant) Record(res Reservation, u store.Usage) {
|
||||
if u.ServedModel != "" && u.ServedModel != u.Model && a.pricing != nil {
|
||||
price = a.pricing.For(u.ServedModel)
|
||||
}
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens)
|
||||
u.CostMicros = price.Cost(u.PromptTokens, u.CompletionTokens, u.Images)
|
||||
for _, b := range res.Budgets {
|
||||
a.budgets.Add(b.Counter, b.Period, u.CostMicros)
|
||||
}
|
||||
|
||||
@@ -39,17 +39,27 @@ func TestCostRoundsUpSoSmallRequestsAreNotFree(t *testing.T) {
|
||||
// Rounding down would make a stream of tiny requests cost nothing, which is
|
||||
// exactly the traffic shape a quota exists to bound.
|
||||
p := Price{PromptPer1k: 1000, CompletionPer1k: 3000}
|
||||
if got := p.Cost(1, 0); got != 1 {
|
||||
if got := p.Cost(1, 0, 0); got != 1 {
|
||||
t.Errorf("one prompt token cost %d", got)
|
||||
}
|
||||
if got := p.Cost(1000, 1000); got != 4000 {
|
||||
if got := p.Cost(1000, 1000, 0); got != 4000 {
|
||||
t.Errorf("a thousand of each cost %d, want 4000", got)
|
||||
}
|
||||
if got := p.Cost(0, 0); got != 0 {
|
||||
if got := p.Cost(0, 0, 0); got != 0 {
|
||||
t.Errorf("an empty request cost %d", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureCostsItsPriceOnTopOfItsPrompt(t *testing.T) {
|
||||
p := Price{PromptPer1k: 1000, PerImage: 50000}
|
||||
if got := p.Cost(20, 0, 3); got != 20+150000 {
|
||||
t.Errorf("three pictures from a 20-token prompt cost %d", got)
|
||||
}
|
||||
if (Price{PerImage: 1}).Zero() {
|
||||
t.Error("a model priced per picture reads as free, and would pass a spent budget")
|
||||
}
|
||||
}
|
||||
|
||||
func TestZeroPricedModelsStayAvailableWhenTheBudgetIsSpent(t *testing.T) {
|
||||
// This is what makes budget fallback work: a key with nothing left can
|
||||
// still be served something that costs nothing.
|
||||
|
||||
@@ -18,6 +18,10 @@ import (
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is what one picture costs, for a model that makes them: a
|
||||
// picture's cost is the card's time, which its prompt's few tokens do
|
||||
// not measure.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
}
|
||||
|
||||
// Zero reports whether this model spends nothing.
|
||||
@@ -25,14 +29,15 @@ type Price struct {
|
||||
// It matters beyond arithmetic: a zero-cost model stays available to a key
|
||||
// whose budget is spent, which is what makes the plan's budget fallback work
|
||||
// (§7.7).
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 }
|
||||
func (p Price) Zero() bool { return p.PromptPer1k == 0 && p.CompletionPer1k == 0 && p.PerImage == 0 }
|
||||
|
||||
// Cost computes what a request spent, rounding up.
|
||||
//
|
||||
// Up rather than to nearest: rounding down would make a stream of tiny requests
|
||||
// free, which is exactly the shape of traffic a quota exists to bound.
|
||||
func (p Price) Cost(promptTokens, completionTokens int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000)
|
||||
func (p Price) Cost(promptTokens, completionTokens, images int64) int64 {
|
||||
return divUp(promptTokens*p.PromptPer1k, 1000) + divUp(completionTokens*p.CompletionPer1k, 1000) +
|
||||
max(images, 0)*p.PerImage
|
||||
}
|
||||
|
||||
func divUp(n, d int64) int64 {
|
||||
|
||||
@@ -34,6 +34,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
@@ -140,6 +141,9 @@ type Agent struct {
|
||||
// settingsCheck says whether the cluster's settings for this agent
|
||||
// would load beside its agent.yaml; see SetSettingsCheck.
|
||||
settingsCheck func([]byte) error
|
||||
|
||||
// trainer runs LoRA training jobs, one at a time (BACKLOG F4).
|
||||
trainer *train.Runner
|
||||
}
|
||||
|
||||
// connectHTTPClient is the HTTP client connect uses; an interface so tests can
|
||||
@@ -186,6 +190,18 @@ func New(cfg Config, httpClient connectHTTPClient, log *slog.Logger) (*Agent, er
|
||||
kvSubs: map[string]*kvevents.Subscriber{},
|
||||
}
|
||||
a.prov = &provision.Installer{Root: cfg.EnginesDir, Log: log, OnStatus: a.provisionStatus}
|
||||
runAs := cfg.Superv.RunAs
|
||||
if runAs == "" {
|
||||
runAs = supervisor.DefaultRunAs
|
||||
}
|
||||
// Where the engines are, as the installer has it when the file says
|
||||
// nothing.
|
||||
enginesDir := cfg.EnginesDir
|
||||
if enginesDir == "" {
|
||||
enginesDir = provision.DefaultRoot
|
||||
}
|
||||
a.trainer = train.New(train.Env{EnginesDir: enginesDir, WorkDir: filepath.Join(cfg.Superv.WorkDir, "trainer"),
|
||||
RunAs: runAs, Log: log})
|
||||
return a, nil
|
||||
}
|
||||
|
||||
@@ -886,6 +902,10 @@ func (a *Agent) handle(ctx context.Context, msg *llmgwv1.ControlMessage, out cha
|
||||
go a.applyFirewall(m.Firewall)
|
||||
case *llmgwv1.ControlMessage_AgentSettings:
|
||||
go a.applySettings(m.AgentSettings)
|
||||
case *llmgwv1.ControlMessage_TrainStart:
|
||||
go a.trainJob(ctx, m.TrainStart, out)
|
||||
case *llmgwv1.ControlMessage_TrainCancel:
|
||||
a.trainer.Cancel(m.TrainCancel.GetJobId())
|
||||
case *llmgwv1.ControlMessage_RestartAgent:
|
||||
go func() {
|
||||
a.log.Info("restarting, as the control plane asks; the engines keep serving",
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
// ComfyUIConfig locates a ComfyUI installation.
|
||||
type ComfyUIConfig struct {
|
||||
// Dir is ComfyUI's directory, where main.py is. Readable by the user
|
||||
// engines run as.
|
||||
Dir string `koanf:"dir"`
|
||||
// Python is the interpreter of its environment; Dir's .venv when unset.
|
||||
Python string `koanf:"python"`
|
||||
}
|
||||
|
||||
// ComfyUI runs ComfyUI as an engine, behind llmgw's gate (llmgw
|
||||
// comfyui-gate): ComfyUI asks for no key, so it listens on the loopback and
|
||||
// the gate, which does, on the replica's port. Its replicas make pictures
|
||||
// for /v1/images/generations.
|
||||
type ComfyUI struct {
|
||||
cfg ComfyUIConfig
|
||||
installed atomic.Pointer[string]
|
||||
}
|
||||
|
||||
// Name identifies the engine in the catalog.
|
||||
func (*ComfyUI) Name() string { return "comfyui" }
|
||||
|
||||
// HealthPath is the gate's, which answers once ComfyUI does.
|
||||
func (*ComfyUI) HealthPath() string { return "/health" }
|
||||
|
||||
// MetricsPath is empty: ComfyUI publishes no metrics, and the router reads
|
||||
// none from it.
|
||||
func (*ComfyUI) MetricsPath() string { return "" }
|
||||
|
||||
// ServesHTTP is true: a ComfyUI replica is one process on one node.
|
||||
func (*ComfyUI) ServesHTTP(ShardSpec) bool { return true }
|
||||
|
||||
// UseInstalled points the engine at the environment the agent installed:
|
||||
// its interpreter, with ComfyUI's code beside its bin, in comfyui/.
|
||||
func (c *ComfyUI) UseInstalled(python string) { c.installed.Store(&python) }
|
||||
|
||||
// Configured reports whether the configuration names a ComfyUI.
|
||||
func (c *ComfyUI) Configured() bool { return c.cfg.Dir != "" }
|
||||
|
||||
func (c *ComfyUI) dir() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return filepath.Join(filepath.Dir(filepath.Dir(*p)), "comfyui")
|
||||
}
|
||||
return c.cfg.Dir
|
||||
}
|
||||
|
||||
func (c *ComfyUI) python() string {
|
||||
if p := c.installed.Load(); p != nil {
|
||||
return *p
|
||||
}
|
||||
if c.cfg.Python != "" {
|
||||
return c.cfg.Python
|
||||
}
|
||||
return filepath.Join(c.cfg.Dir, ".venv", "bin", "python")
|
||||
}
|
||||
|
||||
var comfyVersion = regexp.MustCompile(`__version__\s*=\s*"([^"]+)"`)
|
||||
|
||||
// Probe reports ComfyUI's version, from the file its build writes it to:
|
||||
// importing ComfyUI to ask would load torch and look for cards.
|
||||
func (c *ComfyUI) Probe(context.Context) (string, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return "", fmt.Errorf("no ComfyUI directory is configured")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "main.py")); err != nil {
|
||||
return "", fmt.Errorf("%s is not ComfyUI: %w", dir, err)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(dir, "comfyui_version.py"))
|
||||
if err != nil {
|
||||
return Unknown, nil //nolint:nilerr // installed, of a version it does not say
|
||||
}
|
||||
if m := comfyVersion.FindSubmatch(b); m != nil {
|
||||
return string(m[1]), nil
|
||||
}
|
||||
return Unknown, nil
|
||||
}
|
||||
|
||||
// Validate checks the profile: one card, on one node. ComfyUI runs a
|
||||
// workflow on one device and has no tensor parallelism to spread it by.
|
||||
func (*ComfyUI) Validate(p Profile, gpus []GPUInfo) error {
|
||||
if p.Nodes > 1 || p.TP > 1 || p.PP > 1 || p.GPUsPerNode > 1 {
|
||||
return fmt.Errorf("profile %q: ComfyUI runs on one GPU of one node", p.Name)
|
||||
}
|
||||
return validateCommon(p, gpus)
|
||||
}
|
||||
|
||||
// Commands is the gate, which starts ComfyUI itself.
|
||||
func (c *ComfyUI) Commands(s ShardSpec) ([]Command, error) {
|
||||
dir := c.dir()
|
||||
if dir == "" {
|
||||
return nil, fmt.Errorf("comfyui: no ComfyUI directory is configured (engines.comfyui.dir)")
|
||||
}
|
||||
bin, err := os.Executable()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := s.Profile
|
||||
args := []string{"comfyui-gate",
|
||||
"--listen", net.JoinHostPort(s.Host(), strconv.Itoa(s.HTTPPort)),
|
||||
"--model", p.ServedModelName,
|
||||
"--weights", p.ModelPath,
|
||||
// Relative to where engines run — their user's home — so it is
|
||||
// theirs to write.
|
||||
"--state", filepath.Join("comfyui", s.ReplicaID),
|
||||
"--python", c.python(),
|
||||
"--comfyui", dir,
|
||||
"--",
|
||||
}
|
||||
args = append(args, p.Args...)
|
||||
env := baseEnv(s)
|
||||
if s.APIKey != "" {
|
||||
env["LLMGW_ENGINE_KEY"] = s.APIKey
|
||||
}
|
||||
return []Command{{Name: "comfyui", Path: bin, Args: args, Env: materializeEnv(env)}}, nil
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package engines
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestComfyUIRunsBehindTheGateWithItsKeyOutOfSight(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/comfyui"}}
|
||||
cmds, err := c.Commands(ShardSpec{
|
||||
ReplicaID: "r-flux-1", HTTPPort: 8100, BindAddr: "10.0.0.147", APIKey: "sekret", GPUIndices: []int{0},
|
||||
Profile: Profile{Name: "p", Engine: "comfyui", ModelPath: "/w/flux.safetensors", ServedModelName: "flux",
|
||||
Args: []string{"--disable-smart-memory"}},
|
||||
})
|
||||
if err != nil || len(cmds) != 1 {
|
||||
t.Fatalf("%v %v", cmds, err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
for _, want := range []string{"comfyui-gate", "--listen 10.0.0.147:8100", "--model flux", "--weights /w/flux.safetensors",
|
||||
"--state comfyui/r-flux-1", "--python /opt/comfyui/.venv/bin/python", "--comfyui /opt/comfyui", "-- --disable-smart-memory"} {
|
||||
if !strings.Contains(args, want) {
|
||||
t.Errorf("%q not in %q", want, args)
|
||||
}
|
||||
}
|
||||
if strings.Contains(args, "sekret") {
|
||||
t.Error("the key is on the command line, where every user can read it")
|
||||
}
|
||||
if !slices.Contains(cmds[0].Env, "LLMGW_ENGINE_KEY=sekret") || !slices.Contains(cmds[0].Env, "CUDA_VISIBLE_DEVICES=0") {
|
||||
t.Error("the key or the card is not in the environment")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIRunsOnOneCard(t *testing.T) {
|
||||
c := &ComfyUI{}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 2}, h100(2)); err == nil {
|
||||
t.Error("a two-card ComfyUI profile was accepted")
|
||||
}
|
||||
if err := c.Validate(Profile{Name: "p", ModelPath: "/w", TP: 1}, h100(1)); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if _, err := c.Commands(ShardSpec{Profile: Profile{ModelPath: "/w"}}); err == nil {
|
||||
t.Error("commands without a ComfyUI directory")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsOfferedByTheVersionItsBuildWrote(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: dir}}
|
||||
if _, err := c.Probe(t.Context()); err == nil {
|
||||
t.Error("a directory without main.py passed for ComfyUI")
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(dir, "main.py"), nil, 0o644)
|
||||
_ = os.WriteFile(filepath.Join(dir, "comfyui_version.py"), []byte("__version__ = \"0.39.2\"\n"), 0o644)
|
||||
if v, err := c.Probe(t.Context()); err != nil || v != "0.39.2" {
|
||||
t.Errorf("%q %v", v, err)
|
||||
}
|
||||
if got := NewRegistry(Config{ComfyUI: ComfyUIConfig{Dir: dir}}).ProbeConfigured(t.Context()); got["comfyui"] != "0.39.2" {
|
||||
t.Errorf("a configured ComfyUI is reported as %v", got)
|
||||
}
|
||||
if err := UnsafeArgs("comfyui", []string{"--enable-manager"}); err == nil {
|
||||
t.Error("a profile may install ComfyUI nodes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnInstalledComfyUIIsFoundBesideItsInterpreter(t *testing.T) {
|
||||
c := &ComfyUI{cfg: ComfyUIConfig{Dir: "/opt/by-hand"}}
|
||||
c.UseInstalled("/opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python")
|
||||
cmds, err := c.Commands(ShardSpec{ReplicaID: "r", HTTPPort: 1, Profile: Profile{ModelPath: "/w", ServedModelName: "m"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
args := strings.Join(cmds[0].Args, " ")
|
||||
if !strings.Contains(args, "--python /opt/llmgw/engines/comfyui/0.39.2-cu13/bin/python") ||
|
||||
!strings.Contains(args, "--comfyui /opt/llmgw/engines/comfyui/0.39.2-cu13/comfyui") {
|
||||
t.Errorf("the installed ComfyUI is not what runs: %s", args)
|
||||
}
|
||||
}
|
||||
@@ -214,6 +214,7 @@ func NewRegistry(cfg Config) *Registry {
|
||||
r.Add(&LlamaCpp{cfg: cfg.LlamaCpp})
|
||||
r.Add(&TRTLLM{cfg: cfg.TRTLLM})
|
||||
r.Add(&Sim{cfg: cfg.Sim})
|
||||
r.Add(&ComfyUI{cfg: cfg.ComfyUI})
|
||||
return r
|
||||
}
|
||||
|
||||
@@ -262,6 +263,8 @@ type Config struct {
|
||||
// offers it only when its interpreter is named here.
|
||||
TRTLLM TRTLLMConfig `koanf:"trtllm"`
|
||||
Sim SimConfig `koanf:"sim"`
|
||||
// ComfyUI makes pictures; offered by a node whose ComfyUI is named here.
|
||||
ComfyUI ComfyUIConfig `koanf:"comfyui"`
|
||||
}
|
||||
|
||||
// --- shared helpers ---
|
||||
|
||||
@@ -477,7 +477,7 @@ func TestValidateRequiresModelPath(t *testing.T) {
|
||||
|
||||
func TestRegistry(t *testing.T) {
|
||||
r := NewRegistry(Config{})
|
||||
if got := r.Names(); !slices.Equal(got, []string{"llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
if got := r.Names(); !slices.Equal(got, []string{"comfyui", "llamacpp", "sglang", "sim", "trtllm", "vllm"}) {
|
||||
t.Errorf("registry contents: %v", got)
|
||||
}
|
||||
if _, err := r.Get("vllm"); err != nil {
|
||||
|
||||
@@ -119,6 +119,19 @@ var flagsNever = map[string]map[string]string{
|
||||
"--model-path": "is the model's source, set by the agent", "--served-model-name": "is set by the agent",
|
||||
"--lora-paths": "loads files", "--download-dir": "writes files",
|
||||
},
|
||||
// ComfyUI's main.py, behind the gate that sets where it listens and
|
||||
// where its files are.
|
||||
"comfyui": {
|
||||
"--listen": "is set by the agent", "--base-directory": "is set by the agent",
|
||||
"--extra-model-paths-config": "is set by the agent",
|
||||
"--output-directory": "writes files", "--temp-directory": "writes files",
|
||||
"--input-directory": "reads files", "--user-directory": "writes files",
|
||||
"--database-url": "writes a database elsewhere", "--front-end-root": "serves a directory",
|
||||
"--enable-cors-header": "lets other sites' pages call it",
|
||||
"--enable-manager": "installs nodes, which run Python",
|
||||
"--tls-keyfile": "reads a file", "--tls-certfile": "reads a file",
|
||||
"--comfy-api-base": "sends work to another service",
|
||||
},
|
||||
// trtllm-serve spells its flags with underscores, so the shared list's
|
||||
// hyphenated names do not reach them.
|
||||
"trtllm": {
|
||||
|
||||
@@ -0,0 +1,132 @@
|
||||
package provision
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
|
||||
// comfyuiSource is a ComfyUI release archive as GitHub makes one, and the
|
||||
// recipe pointed at it.
|
||||
func comfyuiSource(t *testing.T, in *Installer, rel *release, digest string) {
|
||||
t.Helper()
|
||||
archive := tarGz(t, map[string]string{
|
||||
"ComfyUI-0.39.2/main.py": "print('comfy')\n",
|
||||
"ComfyUI-0.39.2/comfyui_version.py": "__version__ = \"0.39.2\"\n",
|
||||
"ComfyUI-0.39.2/requirements.txt": "comfyui-frontend-package==1.53.10\ntorch\ntorchvision\n" +
|
||||
"numpy>=1.25.0\n\n#non essential dependencies:\nkornia>=0.7.1\n",
|
||||
}, nil)
|
||||
rel.assets["v0.39.2.tar.gz"] = archive
|
||||
sum := sha256.Sum256(archive)
|
||||
if digest == "" {
|
||||
digest = hex.EncodeToString(sum[:])
|
||||
}
|
||||
old := comfyuiReleases
|
||||
comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{"0.39.2": {digest, "2026-10-07T21:58:07Z"}}
|
||||
t.Cleanup(func() { comfyuiReleases = old })
|
||||
}
|
||||
|
||||
var turingNode = discovery.Host{OSID: "ubuntu", OSVersion: "24.04", Arch: "amd64",
|
||||
NVIDIADriver: "580.82.07", MinComputeCapability: 7.5}
|
||||
|
||||
func TestComfyUIIsInstalledFromItsSourceCheckedAgainstTheRecipe(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready {
|
||||
t.Fatalf("not ready: %+v", got[0])
|
||||
}
|
||||
dir := filepath.Join(in.Root, "comfyui", "0.39.2-cu13")
|
||||
if got[0].Binary != filepath.Join(dir, "bin", "python") {
|
||||
t.Errorf("binary %s, want the environment's interpreter", got[0].Binary)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "comfyui", "main.py")); err != nil {
|
||||
t.Errorf("ComfyUI's code is not beside the interpreter: %v", err)
|
||||
}
|
||||
all := strings.Join(sys.commands, "\n")
|
||||
for _, want := range []string{
|
||||
// Not PyPI's date, which ComfyUI has none of: the recipe's, a day on.
|
||||
"--exclude-newer 2026-10-08T21:58:07Z",
|
||||
"comfyui-frontend-package==1.53.10", "numpy>=1.25.0", "kornia>=0.7.1", "torch",
|
||||
"--require-hashes",
|
||||
} {
|
||||
if !strings.Contains(all, want) {
|
||||
t.Errorf("no %q in:\n%s", want, all)
|
||||
}
|
||||
}
|
||||
if strings.Contains(all, "/pypi/") {
|
||||
t.Error("PyPI was asked when ComfyUI was published")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIOnAnOlderDriverTakesPyTorchsCUDA12Torch(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
host := turingNode
|
||||
host.NVIDIADriver = "550.163.01"
|
||||
in, rel := pythonInstaller(t, host, sys)
|
||||
comfyuiSource(t, in, rel, "")
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Ready || got[0].Variant != "cu129" {
|
||||
t.Fatalf("got %+v", got[0])
|
||||
}
|
||||
var project string
|
||||
for _, c := range sys.commands {
|
||||
if strings.HasPrefix(c, "pyproject: ") {
|
||||
project = c
|
||||
}
|
||||
}
|
||||
// torch from PyTorch's index alone, named once, as the build names it.
|
||||
if !strings.Contains(project, `torch = { index = "extra" }`) || strings.Count(project, `"torch"`) != 1 {
|
||||
t.Errorf("torch is not from PyTorch's index alone:\n%s", project)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIWhoseSourceIsNotTheOneTheRecipeNamesIsNotInstalled(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, rel := pythonInstaller(t, turingNode, sys)
|
||||
comfyuiSource(t, in, rel, strings.Repeat("ab", 32))
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}})
|
||||
if got[0].State != Failed || !strings.Contains(got[0].Detail, "does not match") {
|
||||
t.Fatalf("got %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
for _, c := range sys.commands {
|
||||
if strings.Contains(c, "pip install") {
|
||||
t.Errorf("installed from a source that did not match: %s", c)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIIsInstalledOnlyWhereItCanRun(t *testing.T) {
|
||||
sys := &fakePython{fakeSystem: fakeSystem{installed: map[string]bool{}}}
|
||||
in, _ := pythonInstaller(t, ubuntu, sys)
|
||||
if got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.39.2"}}); got[0].State != Skipped {
|
||||
t.Errorf("a node with no GPU: %+v", got[0])
|
||||
}
|
||||
in, _ = pythonInstaller(t, turingNode, sys)
|
||||
got := in.Converge(t.Context(), []Want{{Engine: "comfyui", Version: "0.1.0"}})
|
||||
if got[0].State != Unsupported || !strings.Contains(got[0].Detail, "knows the source digest of ComfyUI 0.39.2") {
|
||||
t.Errorf("a version without a digest: %s %q", got[0].State, got[0].Detail)
|
||||
}
|
||||
}
|
||||
|
||||
func TestARequirementsFileIsReadForPackagesAndNothingElse(t *testing.T) {
|
||||
got, err := requirementsIn("torch\ntorchvision>=0.20\nPillow\n# a comment\naiohttp>=3.11 # inline\n", pytorchPackages)
|
||||
if err != nil || strings.Join(got, ",") != "Pillow,aiohttp>=3.11" {
|
||||
t.Errorf("%v %v", got, err)
|
||||
}
|
||||
for _, bad := range []string{"-r other.txt", "--index-url https://evil", "x @ https://evil/x.whl", "-e ."} {
|
||||
if _, err := requirementsIn(bad+"\n", nil); err == nil {
|
||||
t.Errorf("%q was accepted", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -66,12 +66,17 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
return fmt.Errorf("installing uv, which installs %s: %s", w.Engine, uv.Detail)
|
||||
}
|
||||
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
var asOf string
|
||||
if !b.AsOf.IsZero() {
|
||||
asOf = b.AsOf.UTC().Format(time.RFC3339)
|
||||
} else {
|
||||
in.progress(st, "asking PyPI when %s %s was published", r.PyPI, w.Version)
|
||||
published, err := in.published(ctx, r.PyPI, w.Version)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
asOf = published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
}
|
||||
asOf := published.Add(24 * time.Hour).UTC().Format(time.RFC3339)
|
||||
|
||||
cache := filepath.Join(in.root(), ".uv-cache")
|
||||
env := []string{
|
||||
@@ -96,7 +101,14 @@ func (in *Installer) installPython(ctx context.Context, r Recipe, w Want, b Buil
|
||||
}
|
||||
python := filepath.Join(tmp, "bin", "python")
|
||||
|
||||
reqs := b.Requirements
|
||||
reqs := slices.Clone(b.Requirements)
|
||||
if b.Source != nil {
|
||||
more, err := in.unpackSource(ctx, w, b, tmp, st)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
reqs = append(reqs, more...)
|
||||
}
|
||||
if b.Wheel != nil {
|
||||
// The engine from its GitHub release, pinned to the digest GitHub
|
||||
// publishes for the file: the same trust as an archive, and uv
|
||||
@@ -320,3 +332,68 @@ func (in *Installer) gpuCheck(ctx context.Context, python string, w Want, b Buil
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// unpackSource fetches an engine's source archive, checks it against the
|
||||
// digest the recipe holds, puts its code in the environment, and returns the
|
||||
// requirements its requirements file names — less the packages a second
|
||||
// index gives, which the build names itself.
|
||||
func (in *Installer) unpackSource(ctx context.Context, w Want, b Build, env string, st *Status) ([]string, error) {
|
||||
src := b.Source
|
||||
url := src.URL
|
||||
if strings.HasPrefix(url, "/") {
|
||||
url = strings.TrimSuffix(in.downloadBase(), "/") + url
|
||||
}
|
||||
scratch := env + ".source"
|
||||
_ = os.RemoveAll(scratch)
|
||||
if err := os.MkdirAll(scratch, 0o755); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(scratch) //nolint:errcheck // scratch
|
||||
archive := filepath.Join(scratch, "source.tar.gz")
|
||||
name := w.Engine + " " + w.Version + "'s source"
|
||||
if err := in.download(ctx, url, name, archive, src.SHA256, st); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
in.progress(st, "unpacking %s", name)
|
||||
if err := untarGz(archive, scratch); err != nil {
|
||||
return nil, fmt.Errorf("unpacking %s: %w", name, err)
|
||||
}
|
||||
dst := filepath.Join(env, src.Dir)
|
||||
if err := os.Rename(filepath.Join(scratch, src.Top), dst); err != nil {
|
||||
return nil, fmt.Errorf("%s has no %s at its top: %w", name, src.Top, err)
|
||||
}
|
||||
if src.Requirements == "" {
|
||||
return nil, nil
|
||||
}
|
||||
f, err := os.ReadFile(filepath.Join(dst, src.Requirements))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%s: %w", name, err)
|
||||
}
|
||||
return requirementsIn(string(f), b.IndexPackages)
|
||||
}
|
||||
|
||||
// requirementsIn reads a requirements file's lines. Options (-r, -e, --index-url
|
||||
// and the like) are refused: they would fetch from somewhere other than PyPI,
|
||||
// or install what the lock never saw.
|
||||
func requirementsIn(text string, skip []string) ([]string, error) {
|
||||
var out []string
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line, _, _ = strings.Cut(line, "#")
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if strings.HasPrefix(line, "-") || strings.Contains(line, "://") || strings.Contains(line, " @ ") {
|
||||
return nil, fmt.Errorf("the requirements name %q, which is not a package from PyPI", line)
|
||||
}
|
||||
name := strings.ToLower(line)
|
||||
if i := strings.IndexAny(name, "<>=!~;[ "); i >= 0 {
|
||||
name = name[:i]
|
||||
}
|
||||
if slices.Contains(skip, name) {
|
||||
continue
|
||||
}
|
||||
out = append(out, line)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -53,6 +53,7 @@ func (f *fakePython) run(ctx context.Context, env []string, name string, args ..
|
||||
return []byte("error: Hash mismatch for torch\n"), errExit
|
||||
}
|
||||
venv := filepath.Dir(filepath.Dir(args[slices.Index(args, "--python")+1]))
|
||||
_ = os.WriteFile(filepath.Join(venv, "bin", "python"), []byte("#!/bin/sh\n"), 0o755)
|
||||
return nil, os.WriteFile(filepath.Join(venv, "bin", "vllm"), []byte("#!/bin/sh\n"), 0o755)
|
||||
case strings.HasSuffix(name, "/bin/vllm") && args[0] == "--version":
|
||||
return []byte("0.30.0\n"), nil
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"fmt"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
)
|
||||
@@ -68,12 +69,31 @@ type Build struct {
|
||||
// Note says why this build, when it is not the plain one: shown beside
|
||||
// it wherever the installation is.
|
||||
Note string
|
||||
// Source, for a Python engine that is not on PyPI, is its own code: an
|
||||
// archive pinned by its digest, unpacked into the environment, whose
|
||||
// requirements file is installed with Requirements. AsOf is then the
|
||||
// date its dependencies are resolved as of — the day after its version
|
||||
// was published, which PyPI cannot be asked for.
|
||||
Source *Source
|
||||
AsOf time.Time
|
||||
// FreeGB is the disk space the installation needs before it starts: a
|
||||
// Python engine is several gigabytes, and one that fills the disk halfway
|
||||
// leaves the node worse off than one that never began.
|
||||
FreeGB int
|
||||
}
|
||||
|
||||
// Source is an engine's code as an archive.
|
||||
type Source struct {
|
||||
// URL is fetched as is; SHA256 is what it must hash to, written into
|
||||
// the recipe because a project's source archives come with no published
|
||||
// digest of their own.
|
||||
URL, SHA256 string
|
||||
// Top is the archive's one top directory, put in the environment as Dir.
|
||||
Top, Dir string
|
||||
// Requirements is the requirements file, in Dir.
|
||||
Requirements string
|
||||
}
|
||||
|
||||
// Wheel is a package published as a GitHub release asset: the asset whose
|
||||
// name starts with Prefix and ends with Suffix.
|
||||
type Wheel struct {
|
||||
@@ -104,6 +124,7 @@ var Recipes = map[string]Recipe{
|
||||
"llamacpp": {Repo: "ggml-org/llama.cpp", Candidates: llamaCppCandidates},
|
||||
"vllm": {PyPI: "vllm", Candidates: vllmCandidates},
|
||||
"sglang": {PyPI: "sglang", Candidates: sglangCandidates},
|
||||
"comfyui": {Candidates: comfyuiCandidates},
|
||||
"uv": {Repo: "astral-sh/uv", Candidates: uvCandidates},
|
||||
}
|
||||
|
||||
@@ -556,3 +577,78 @@ func orUnknown(s string) string {
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// --- ComfyUI ---
|
||||
|
||||
// comfyuiReleases are the ComfyUI versions this agent installs: each source
|
||||
// archive's digest, taken when it was added here and confirmed from a second
|
||||
// machine, and when the version was published. GitHub publishes no digest
|
||||
// for a tag's archive, so a version not listed is refused rather than
|
||||
// installed unchecked.
|
||||
var comfyuiReleases = map[string]struct {
|
||||
sha256 string
|
||||
published string
|
||||
}{
|
||||
"0.39.2": {"1d655e22cf546700fc1d6d2536e27701695c5832a51f63bb16c41866ac09be0b", "2026-10-07T21:58:07Z"},
|
||||
}
|
||||
|
||||
// ComfyUIRepo is where ComfyUI's source is published.
|
||||
const ComfyUIRepo = "Comfy-Org/ComfyUI"
|
||||
|
||||
// comfyuiDir is where ComfyUI's code is put in its environment: main.py is
|
||||
// <environment>/comfyui/main.py, beside bin/python.
|
||||
const comfyuiDir = "comfyui"
|
||||
|
||||
// comfyuiCandidates are the ComfyUI builds for a machine: its source, in a
|
||||
// Python environment with the torch for the node's driver — PyPI's, built
|
||||
// for CUDA 13, or PyTorch's CUDA 12.9 build for a driver CUDA 13 does not
|
||||
// run on.
|
||||
func comfyuiCandidates(version, variant string, h discovery.Host) ([]Build, error) {
|
||||
if _, err := gpuArch("ComfyUI", h); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
rel, ok := comfyuiReleases[version]
|
||||
if !ok {
|
||||
known := make([]string, 0, len(comfyuiReleases))
|
||||
for v := range comfyuiReleases {
|
||||
known = append(known, v)
|
||||
}
|
||||
slices.Sort(known)
|
||||
return nil, fmt.Errorf("this agent knows the source digest of ComfyUI %s, not of %s; "+
|
||||
"install that one by hand and name it in agent.yaml (engines.comfyui.dir)",
|
||||
strings.Join(known, ", "), version)
|
||||
}
|
||||
published, err := time.Parse(time.RFC3339, rel.published)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
build := func(variant string) Build {
|
||||
return Build{
|
||||
Variant: variant,
|
||||
Python: pythonForEngines,
|
||||
Source: &Source{
|
||||
URL: "/" + ComfyUIRepo + "/archive/refs/tags/v" + version + ".tar.gz",
|
||||
SHA256: rel.sha256, Top: "ComfyUI-" + version, Dir: comfyuiDir,
|
||||
Requirements: "requirements.txt",
|
||||
},
|
||||
AsOf: published.Add(24 * time.Hour),
|
||||
// ComfyUI is started as a script, so what the agent is given is
|
||||
// the environment's interpreter; the proof is importing what it
|
||||
// imports first, from where its code is.
|
||||
Binary: "python",
|
||||
VersionArgs: []string{"-c", `import os, sys, torch, aiohttp, safetensors
|
||||
d = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(sys.executable))), "` + comfyuiDir + `")
|
||||
assert os.path.isfile(os.path.join(d, "main.py")), d + " has no main.py"
|
||||
print("comfyui version", "` + version + `", "on torch", torch.__version__)`},
|
||||
Packages: compilerPackages,
|
||||
FreeGB: 15,
|
||||
GPUCheck: true,
|
||||
}
|
||||
}
|
||||
cu129 := build("cu129")
|
||||
cu129.ExtraIndex, cu129.IndexPackages = pytorchCU129, pytorchPackages
|
||||
return pick("ComfyUI "+version, variant, []candidate{
|
||||
{build("cu13"), gpuNeeds{minDriverForCUDA13, 7.5}, "ComfyUI " + version + " on CUDA 13's torch"},
|
||||
{cu129, gpuNeeds{minDriverForCUDA12, 7.0}, "ComfyUI " + version + " on CUDA 12.9's torch"},
|
||||
}, h)
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package supervisor
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/user"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
@@ -90,3 +91,25 @@ func credentialFor(u *user.User, workDir string) (*runAs, error) {
|
||||
func (r *runAs) env() []string {
|
||||
return []string{"HOME=" + r.home, "XDG_CACHE_HOME=" + filepath.Join(r.home, ".cache"), "USER=" + r.name, "LOGNAME=" + r.name}
|
||||
}
|
||||
|
||||
// AsEngineUser has cmd run as the engine user, as an engine would, with its
|
||||
// home under workDir: for work the agent does that is not an engine but
|
||||
// reads what an engine reads — training an adapter. It answers that home
|
||||
// and a way to hand a path to the user, or why it runs as this process.
|
||||
func AsEngineUser(cmd *exec.Cmd, name, workDir string) (home string, give func(path string) error, why string) {
|
||||
r, why := resolveRunAs(name, workDir)
|
||||
if r == nil {
|
||||
return "", func(string) error { return nil }, why
|
||||
}
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Credential: r.cred, Setpgid: true}
|
||||
cmd.Env = append(cmd.Env, r.env()...)
|
||||
uid, gid := int(r.cred.Uid), int(r.cred.Gid)
|
||||
return r.home, func(path string) error {
|
||||
return filepath.WalkDir(path, func(p string, _ os.DirEntry, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return os.Lchown(p, uid, gid)
|
||||
})
|
||||
}, ""
|
||||
}
|
||||
@@ -0,0 +1,574 @@
|
||||
// Package train runs LoRA training jobs on a node (BACKLOG F4): with an
|
||||
// installed engine's Python — its torch and transformers, already matched to
|
||||
// the node's CUDA — and a small overlay of what training needs besides, so a
|
||||
// node that serves can train without a second multi-gigabyte environment.
|
||||
package train
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
_ "embed"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/supervisor"
|
||||
)
|
||||
|
||||
//go:embed trainer.py
|
||||
var trainerPy []byte
|
||||
|
||||
// overlayPackages are what training needs besides an engine's Python, pinned
|
||||
// and installed without their dependencies: those are the engine's own, and
|
||||
// letting an installer resolve them pulled a second torch.
|
||||
var overlayPackages = []string{"peft==0.21.2", "accelerate==1.15.0"}
|
||||
|
||||
// Job is one training job, as the control plane sent it.
|
||||
type Job struct {
|
||||
ID string
|
||||
BaseRepo string
|
||||
BaseRevision string
|
||||
// Data is NDJSON of {"q", "a"}.
|
||||
Data []byte
|
||||
// Params is the trainer's settings as JSON.
|
||||
Params string
|
||||
GPUMemoryMiB int
|
||||
GGUF bool
|
||||
}
|
||||
|
||||
// Status is how a job is going.
|
||||
type Status struct {
|
||||
State string
|
||||
Message string
|
||||
Step int
|
||||
Steps int
|
||||
Loss float64
|
||||
PeakMiB int
|
||||
}
|
||||
|
||||
// Env is where a node keeps what training needs.
|
||||
type Env struct {
|
||||
// EnginesDir is where engines are installed (the agent's engines_dir).
|
||||
EnginesDir string
|
||||
// WorkDir is the trainer's own directory: its overlay, the llama.cpp
|
||||
// conversion code, and each job's files while it runs.
|
||||
WorkDir string
|
||||
// RunAs is the user jobs run as: the engines' user.
|
||||
RunAs string
|
||||
Log *slog.Logger
|
||||
// Python, UV, ConvDir and GPUs replace what would be found, for tests.
|
||||
Python, UV, ConvDir string
|
||||
GPUs func(ctx context.Context) ([]GPU, error)
|
||||
}
|
||||
|
||||
// GPU is one card and how much of it is free.
|
||||
type GPU struct {
|
||||
Index int
|
||||
FreeMiB int
|
||||
}
|
||||
|
||||
// Runner runs one job at a time.
|
||||
type Runner struct {
|
||||
env Env
|
||||
mu sync.Mutex
|
||||
cancel context.CancelFunc
|
||||
job string
|
||||
}
|
||||
|
||||
// New is a runner.
|
||||
func New(env Env) *Runner {
|
||||
if env.Log == nil {
|
||||
env.Log = slog.New(slog.DiscardHandler)
|
||||
}
|
||||
return &Runner{env: env}
|
||||
}
|
||||
|
||||
// Cancel stops a job if it is the one running.
|
||||
func (r *Runner) Cancel(id string) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.job == id && r.cancel != nil {
|
||||
r.cancel()
|
||||
}
|
||||
}
|
||||
|
||||
// ErrBusy is a node already training.
|
||||
var ErrBusy = errors.New("this node is already training")
|
||||
|
||||
// Run runs a job to its end, telling status as it goes and artifact each
|
||||
// piece of what it made. It answers the error it failed with.
|
||||
func (r *Runner) Run(ctx context.Context, job Job, status func(Status), artifact func(kind string, seq int, data []byte, last bool) error) error {
|
||||
r.mu.Lock()
|
||||
if r.job != "" {
|
||||
r.mu.Unlock()
|
||||
return ErrBusy
|
||||
}
|
||||
ctx, cancel := context.WithCancel(ctx)
|
||||
r.job, r.cancel = job.ID, cancel
|
||||
r.mu.Unlock()
|
||||
defer func() {
|
||||
cancel()
|
||||
r.mu.Lock()
|
||||
r.job, r.cancel = "", nil
|
||||
r.mu.Unlock()
|
||||
}()
|
||||
|
||||
status(Status{State: "preparing", Message: "preparing the trainer"})
|
||||
py, err := r.python()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
overlay, err := r.overlay(ctx, py)
|
||||
if err != nil {
|
||||
return fmt.Errorf("installing %s: %w", strings.Join(overlayPackages, ", "), err)
|
||||
}
|
||||
conv := ""
|
||||
if job.GGUF {
|
||||
if conv, err = r.conversion(ctx); err != nil {
|
||||
return fmt.Errorf("fetching llama.cpp's conversion code: %w", err)
|
||||
}
|
||||
}
|
||||
gpu, err := r.pickGPU(ctx, job.GPUMemoryMiB)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
dir := filepath.Join(r.env.WorkDir, "jobs", job.ID)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
out := filepath.Join(dir, "adapter")
|
||||
var params map[string]any
|
||||
if job.Params != "" {
|
||||
if err := json.Unmarshal([]byte(job.Params), ¶ms); err != nil {
|
||||
return fmt.Errorf("the job's params: %w", err)
|
||||
}
|
||||
}
|
||||
spec, _ := json.Marshal(map[string]any{"base_repo": job.BaseRepo, "base_revision": job.BaseRevision,
|
||||
"data": filepath.Join(dir, "data.jsonl"), "out": out, "params": params})
|
||||
for name, b := range map[string][]byte{"data.jsonl": job.Data, "job.json": spec, "trainer.py": trainerPy} {
|
||||
if err := os.WriteFile(filepath.Join(dir, name), b, 0o644); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
var baseDir string
|
||||
err = r.python3(ctx, py, dir, []string{"PYTHONPATH=" + overlay, "CUDA_VISIBLE_DEVICES=" + strconv.Itoa(gpu)},
|
||||
[]string{filepath.Join(dir, "trainer.py"), filepath.Join(dir, "job.json")},
|
||||
func(line map[string]any) {
|
||||
s := Status{State: str(line["state"]), Message: str(line["message"]), Step: num(line["step"]),
|
||||
Steps: num(line["steps"]), PeakMiB: num(line["peak_mib"])}
|
||||
if v, ok := line["loss"].(float64); ok {
|
||||
s.Loss = v
|
||||
}
|
||||
if b := str(line["base_dir"]); b != "" {
|
||||
baseDir = b
|
||||
}
|
||||
if s.State == "trained" {
|
||||
s.State = "running"
|
||||
s.Message = "trained; packing the adapter"
|
||||
}
|
||||
status(s)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if job.GGUF {
|
||||
status(Status{State: "running", Message: "converting the adapter to GGUF for llama.cpp"})
|
||||
gguf := filepath.Join(dir, "adapter.gguf")
|
||||
err := r.python3(ctx, py, dir, []string{"PYTHONPATH=" + strings.Join([]string{filepath.Join(conv, "gguf-py"), conv, overlay}, ":")},
|
||||
[]string{filepath.Join(conv, "convert_lora_to_gguf.py"), "--base", baseDir, "--outtype", "f16", "--outfile", gguf, out}, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("converting to GGUF: %w", err)
|
||||
}
|
||||
if err := sendFile(gguf, "gguf", artifact); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
var tgz bytes.Buffer
|
||||
if err := tarGz(&tgz, out); err != nil {
|
||||
return err
|
||||
}
|
||||
return sendBytes(tgz.Bytes(), "peft", artifact)
|
||||
}
|
||||
|
||||
func str(v any) string {
|
||||
s, _ := v.(string)
|
||||
return s
|
||||
}
|
||||
|
||||
func num(v any) int {
|
||||
f, _ := v.(float64)
|
||||
return int(f)
|
||||
}
|
||||
|
||||
// python3 runs a Python script as the engines' user, passing each JSON line
|
||||
// it prints to line, and answers the last of what it wrote to stderr when it
|
||||
// fails.
|
||||
func (r *Runner) python3(ctx context.Context, py, dir string, env, args []string, line func(map[string]any)) error {
|
||||
cmd := exec.CommandContext(ctx, py, args...)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True", "TOKENIZERS_PARALLELISM=false",
|
||||
"HF_HUB_DISABLE_PROGRESS_BARS=1", "PYTHONUNBUFFERED=1")
|
||||
cmd.Env = append(cmd.Env, env...)
|
||||
home, give, why := supervisor.AsEngineUser(cmd, r.env.RunAs, r.env.WorkDir)
|
||||
if why != "" {
|
||||
r.env.Log.Warn(why)
|
||||
}
|
||||
if home != "" {
|
||||
cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(home, ".cache", "huggingface"))
|
||||
} else {
|
||||
cmd.Env = append(cmd.Env, "HF_HOME="+filepath.Join(r.env.WorkDir, "hf"))
|
||||
}
|
||||
if err := give(dir); err != nil {
|
||||
return fmt.Errorf("handing %s to %s: %w", dir, r.env.RunAs, err)
|
||||
}
|
||||
// The whole process group goes when the job is cancelled: torch starts
|
||||
// workers of its own.
|
||||
if cmd.SysProcAttr == nil {
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
}
|
||||
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
|
||||
stdout, err := cmd.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var stderr tailBuffer
|
||||
cmd.Stderr = &stderr
|
||||
if err := cmd.Start(); err != nil {
|
||||
return err
|
||||
}
|
||||
sc := bufio.NewScanner(stdout)
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
for sc.Scan() {
|
||||
var m map[string]any
|
||||
if line != nil && json.Unmarshal(sc.Bytes(), &m) == nil {
|
||||
line(m)
|
||||
}
|
||||
}
|
||||
err = cmd.Wait()
|
||||
if ctx.Err() != nil {
|
||||
return errors.New("cancelled")
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("%v: %s", err, stderr.last())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// tailBuffer keeps the end of what a process wrote: the reason it failed is
|
||||
// at the end, and the rest is progress bars.
|
||||
type tailBuffer struct{ b []byte }
|
||||
|
||||
func (t *tailBuffer) Write(p []byte) (int, error) {
|
||||
t.b = append(t.b, p...)
|
||||
if len(t.b) > 16<<10 {
|
||||
t.b = t.b[len(t.b)-16<<10:]
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (t *tailBuffer) last() string {
|
||||
lines := strings.Split(strings.TrimSpace(string(t.b)), "\n")
|
||||
if len(lines) > 3 {
|
||||
lines = lines[len(lines)-3:]
|
||||
}
|
||||
return strings.Join(lines, " | ")
|
||||
}
|
||||
|
||||
// python is the newest installed vLLM's interpreter: its torch is built for
|
||||
// this node's CUDA.
|
||||
func (r *Runner) python() (string, error) {
|
||||
if r.env.Python != "" {
|
||||
return r.env.Python, nil
|
||||
}
|
||||
found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "vllm", "*", "bin", "python3"))
|
||||
if len(found) == 0 {
|
||||
return "", errors.New("training needs vLLM installed on this node (its torch is what trains); " +
|
||||
"llmgw ctl engine-build, or serving any vLLM model here, installs it")
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found[len(found)-1], nil
|
||||
}
|
||||
|
||||
func (r *Runner) uv() (string, error) {
|
||||
if r.env.UV != "" {
|
||||
return r.env.UV, nil
|
||||
}
|
||||
var found []string
|
||||
_ = filepath.WalkDir(filepath.Join(r.env.EnginesDir, "uv"), func(p string, d os.DirEntry, err error) error {
|
||||
if err == nil && !d.IsDir() && d.Name() == "uv" {
|
||||
found = append(found, p)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if len(found) == 0 {
|
||||
return "", errors.New("no uv under " + filepath.Join(r.env.EnginesDir, "uv"))
|
||||
}
|
||||
sort.Strings(found)
|
||||
return found[len(found)-1], nil
|
||||
}
|
||||
|
||||
// overlay installs the pinned packages once per Python and set of pins.
|
||||
func (r *Runner) overlay(ctx context.Context, py string) (string, error) {
|
||||
sum := sha256.Sum256([]byte(py + "\x00" + strings.Join(overlayPackages, " ")))
|
||||
dir := filepath.Join(r.env.WorkDir, "overlay-"+hex.EncodeToString(sum[:6]))
|
||||
if _, err := os.Stat(filepath.Join(dir, ".done")); err == nil {
|
||||
return dir, nil
|
||||
}
|
||||
uv, err := r.uv()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
tmp := dir + ".new"
|
||||
_ = os.RemoveAll(tmp)
|
||||
args := append([]string{"pip", "install", "--quiet", "--python", py, "--no-deps", "--target", tmp}, overlayPackages...)
|
||||
cmd := exec.CommandContext(ctx, uv, args...)
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
return "", fmt.Errorf("%v: %s", err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(tmp, ".done"), nil, 0o644)
|
||||
_ = os.RemoveAll(dir)
|
||||
if err := os.Rename(tmp, dir); err != nil {
|
||||
return "", err
|
||||
}
|
||||
// Readable by the engines' user, which runs the trainer.
|
||||
_ = filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
|
||||
if err == nil {
|
||||
mode := os.FileMode(0o644)
|
||||
if d.IsDir() {
|
||||
mode = 0o755
|
||||
}
|
||||
_ = os.Chmod(p, mode)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
return dir, nil
|
||||
}
|
||||
|
||||
// llamaCppTag is the release of the newest llama.cpp installed: its
|
||||
// conversion code is the one whose GGUF that llama.cpp reads.
|
||||
func (r *Runner) llamaCppTag() (string, error) {
|
||||
found, _ := filepath.Glob(filepath.Join(r.env.EnginesDir, "llamacpp", "b*"))
|
||||
if len(found) == 0 {
|
||||
return "", errors.New("a GGUF adapter needs llama.cpp installed on this node, to match its release")
|
||||
}
|
||||
sort.Strings(found)
|
||||
tag, _, _ := strings.Cut(filepath.Base(found[len(found)-1]), "-")
|
||||
return tag, nil
|
||||
}
|
||||
|
||||
// conversionSource is where llama.cpp's source is fetched from, by tag.
|
||||
var conversionSource = "https://codeload.github.com/ggml-org/llama.cpp/tar.gz/refs/tags/"
|
||||
|
||||
// conversion fetches llama.cpp's LoRA conversion code for the installed
|
||||
// release, once.
|
||||
func (r *Runner) conversion(ctx context.Context) (string, error) {
|
||||
if r.env.ConvDir != "" {
|
||||
return r.env.ConvDir, nil
|
||||
}
|
||||
tag, err := r.llamaCppTag()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
dir := filepath.Join(r.env.WorkDir, "llama.cpp-"+tag)
|
||||
if _, err := os.Stat(filepath.Join(dir, "convert_lora_to_gguf.py")); err == nil {
|
||||
return dir, nil
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, conversionSource+tag, nil)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
res, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer res.Body.Close()
|
||||
if res.StatusCode != http.StatusOK {
|
||||
return "", fmt.Errorf("%s answered %s", conversionSource+tag, res.Status)
|
||||
}
|
||||
tmp := dir + ".new"
|
||||
_ = os.RemoveAll(tmp)
|
||||
if err := untarConversion(res.Body, tmp); err != nil {
|
||||
return "", err
|
||||
}
|
||||
_ = os.RemoveAll(dir)
|
||||
return dir, os.Rename(tmp, dir)
|
||||
}
|
||||
|
||||
// untarConversion keeps, of llama.cpp's source, the conversion scripts and
|
||||
// the Python packages they import.
|
||||
func untarConversion(r io.Reader, dir string) error {
|
||||
gz, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
kept := 0
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, rel, ok := strings.Cut(h.Name, "/")
|
||||
if !ok || h.Typeflag != tar.TypeReg || strings.Contains(rel, "..") {
|
||||
continue
|
||||
}
|
||||
if rel != "convert_lora_to_gguf.py" && rel != "convert_hf_to_gguf.py" &&
|
||||
!strings.HasPrefix(rel, "conversion/") && !strings.HasPrefix(rel, "gguf-py/") {
|
||||
continue
|
||||
}
|
||||
p := filepath.Join(dir, filepath.FromSlash(rel))
|
||||
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.OpenFile(p, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o644)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, err = io.Copy(f, io.LimitReader(tr, 64<<20))
|
||||
f.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
kept++
|
||||
}
|
||||
if kept == 0 {
|
||||
return errors.New("no conversion code in llama.cpp's source")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// pickGPU is the card with the most memory free, if that is enough.
|
||||
func (r *Runner) pickGPU(ctx context.Context, need int) (int, error) {
|
||||
list := r.env.GPUs
|
||||
if list == nil {
|
||||
list = nvidiaSMI
|
||||
}
|
||||
gpus, err := list(ctx)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("reading the cards' free memory: %w", err)
|
||||
}
|
||||
if len(gpus) == 0 {
|
||||
return 0, errors.New("this node has no GPU to train on")
|
||||
}
|
||||
best := gpus[0]
|
||||
for _, g := range gpus[1:] {
|
||||
if g.FreeMiB > best.FreeMiB {
|
||||
best = g
|
||||
}
|
||||
}
|
||||
if best.FreeMiB < need {
|
||||
return 0, fmt.Errorf("no card here has %d MiB free (the most free is GPU %d with %d MiB); "+
|
||||
"a smaller base, or a card freed for it, would do", need, best.Index, best.FreeMiB)
|
||||
}
|
||||
return best.Index, nil
|
||||
}
|
||||
|
||||
func nvidiaSMI(ctx context.Context) ([]GPU, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 15*time.Second)
|
||||
defer cancel()
|
||||
out, err := exec.CommandContext(ctx, "nvidia-smi", "--query-gpu=index,memory.free", "--format=csv,noheader,nounits").Output()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return parseGPUs(string(out)), nil
|
||||
}
|
||||
|
||||
func parseGPUs(s string) []GPU {
|
||||
var out []GPU
|
||||
for _, line := range strings.Split(strings.TrimSpace(s), "\n") {
|
||||
a, b, ok := strings.Cut(line, ",")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
i, err1 := strconv.Atoi(strings.TrimSpace(a))
|
||||
f, err2 := strconv.Atoi(strings.TrimSpace(b))
|
||||
if err1 == nil && err2 == nil {
|
||||
out = append(out, GPU{Index: i, FreeMiB: f})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// tarGz packs a directory.
|
||||
func tarGz(w io.Writer, dir string) error {
|
||||
gz := gzip.NewWriter(w)
|
||||
tw := tar.NewWriter(gz)
|
||||
err := filepath.WalkDir(dir, func(p string, d os.DirEntry, err error) error {
|
||||
if err != nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
rel, _ := filepath.Rel(dir, p)
|
||||
info, err := d.Info()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.WriteHeader(&tar.Header{Name: filepath.ToSlash(rel), Mode: 0o644, Size: info.Size(), ModTime: info.ModTime()}); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := os.Open(p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
_, err = io.Copy(tw, f)
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tw.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
return gz.Close()
|
||||
}
|
||||
|
||||
// artifactChunk is how much of an artifact goes in one message.
|
||||
const artifactChunk = 1 << 20
|
||||
|
||||
func sendFile(path, kind string, send func(string, int, []byte, bool) error) error {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return sendBytes(b, kind, send)
|
||||
}
|
||||
|
||||
func sendBytes(b []byte, kind string, send func(string, int, []byte, bool) error) error {
|
||||
seq := 0
|
||||
for {
|
||||
n := min(artifactChunk, len(b))
|
||||
last := n == len(b)
|
||||
if err := send(kind, seq, b[:n], last); err != nil {
|
||||
return err
|
||||
}
|
||||
if last {
|
||||
return nil
|
||||
}
|
||||
b = b[n:]
|
||||
seq++
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,164 @@
|
||||
package train
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// fakePython stands in for an engine's interpreter: as the trainer it
|
||||
// prints progress and writes an adapter; as the converter it writes a GGUF;
|
||||
// told to fail, it fails.
|
||||
const fakePython = `#!/bin/sh
|
||||
case "$1" in
|
||||
*trainer.py)
|
||||
out=$(sed -n 's/.*"out": *"\([^"]*\)".*/\1/p' "$2")
|
||||
grep -q FAIL "$(dirname "$2")/data.jsonl" && { echo "CUDA out of memory" >&2; exit 1; }
|
||||
echo '{"state":"preparing","message":"fetching base"}'
|
||||
echo '{"state":"running","step":5,"steps":10,"loss":0.9,"peak_mib":1900}'
|
||||
mkdir -p "$out"; echo '{"r":16}' > "$out/adapter_config.json"; echo weights > "$out/adapter_model.safetensors"
|
||||
echo '{"state":"trained","step":10,"steps":10,"peak_mib":2100,"base_dir":"/base"}'
|
||||
;;
|
||||
*convert_lora_to_gguf.py)
|
||||
while [ $# -gt 0 ]; do [ "$1" = "--outfile" ] && echo GGUF > "$2"; shift; done
|
||||
;;
|
||||
esac
|
||||
`
|
||||
|
||||
func runner(t *testing.T, free int) *Runner {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
py := filepath.Join(dir, "python3")
|
||||
_ = os.WriteFile(py, []byte(fakePython), 0o755)
|
||||
uv := filepath.Join(dir, "uv")
|
||||
_ = os.WriteFile(uv, []byte("#!/bin/sh\nwhile [ $# -gt 0 ]; do [ \"$1\" = \"--target\" ] && mkdir -p \"$2\"; shift; done\n"), 0o755)
|
||||
return New(Env{WorkDir: filepath.Join(dir, "work"), Python: py, UV: uv, ConvDir: dir,
|
||||
GPUs: func(context.Context) ([]GPU, error) { return []GPU{{0, 500}, {1, free}}, nil }})
|
||||
}
|
||||
|
||||
type sink struct {
|
||||
mu sync.Mutex
|
||||
statuses []Status
|
||||
parts map[string][]byte
|
||||
lasts int
|
||||
}
|
||||
|
||||
func (s *sink) status(st Status) { s.mu.Lock(); s.statuses = append(s.statuses, st); s.mu.Unlock() }
|
||||
func (s *sink) artifact(kind string, _ int, b []byte, last bool) error {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if s.parts == nil {
|
||||
s.parts = map[string][]byte{}
|
||||
}
|
||||
s.parts[kind] = append(s.parts[kind], b...)
|
||||
if last {
|
||||
s.lasts++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestAJobTrainsOnTheFreestCardAndSendsWhatItMade(t *testing.T) {
|
||||
r := runner(t, 3000)
|
||||
s := &sink{}
|
||||
err := r.Run(context.Background(), Job{ID: "j1", BaseRepo: "Qwen/x", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500, GGUF: true},
|
||||
s.status, s.artifact)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var running bool
|
||||
for _, st := range s.statuses {
|
||||
if st.State == "running" && st.Step == 5 && st.Steps == 10 && st.PeakMiB == 1900 {
|
||||
running = true
|
||||
}
|
||||
}
|
||||
if !running {
|
||||
t.Errorf("statuses %+v", s.statuses)
|
||||
}
|
||||
if string(s.parts["gguf"]) != "GGUF\n" || s.lasts != 2 {
|
||||
t.Errorf("gguf %q, lasts %d", s.parts["gguf"], s.lasts)
|
||||
}
|
||||
files := map[string]string{}
|
||||
gz, err := gzip.NewReader(bytes.NewReader(s.parts["peft"]))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
b, _ := io.ReadAll(tr)
|
||||
files[h.Name] = string(b)
|
||||
}
|
||||
if files["adapter_model.safetensors"] != "weights\n" || files["adapter_config.json"] == "" {
|
||||
t.Errorf("peft archive %v", files)
|
||||
}
|
||||
// Nothing of the job is left on the node.
|
||||
if entries, _ := os.ReadDir(filepath.Join(r.env.WorkDir, "jobs")); len(entries) != 0 {
|
||||
t.Errorf("left behind: %v", entries)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobWaitsForNoCardItDoesNotFit(t *testing.T) {
|
||||
r := runner(t, 1000)
|
||||
err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"a","a":"b"}`), GPUMemoryMiB: 2500}, (&sink{}).status, (&sink{}).artifact)
|
||||
if err == nil || !strings.Contains(err.Error(), "no card here has 2500 MiB free") {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestATrainerThatFailsSaysWhy(t *testing.T) {
|
||||
r := runner(t, 3000)
|
||||
err := r.Run(context.Background(), Job{ID: "j1", Data: []byte(`{"q":"FAIL","a":"b"}`)}, (&sink{}).status, (&sink{}).artifact)
|
||||
if err == nil || !strings.Contains(err.Error(), "CUDA out of memory") {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOneJobAtATime(t *testing.T) {
|
||||
r := runner(t, 3000)
|
||||
r.job = "other"
|
||||
if err := r.Run(context.Background(), Job{ID: "j2"}, (&sink{}).status, (&sink{}).artifact); err != ErrBusy {
|
||||
t.Errorf("got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCardsAreReadFromNvidiaSMI(t *testing.T) {
|
||||
got := parseGPUs("0, 2310\n1, 80000\nbad line\n")
|
||||
if len(got) != 2 || got[1].FreeMiB != 80000 || got[0].Index != 0 {
|
||||
t.Errorf("parsed %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheConversionCodeIsKeptOfLlamaCpp(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
gz := gzip.NewWriter(&buf)
|
||||
tw := tar.NewWriter(gz)
|
||||
for _, name := range []string{"llama.cpp-b1/convert_lora_to_gguf.py", "llama.cpp-b1/conversion/qwen.py",
|
||||
"llama.cpp-b1/gguf-py/gguf/__init__.py", "llama.cpp-b1/src/llama.cpp", "llama.cpp-b1/../evil"} {
|
||||
_ = tw.WriteHeader(&tar.Header{Name: name, Mode: 0o644, Size: 1, Typeflag: tar.TypeReg})
|
||||
_, _ = tw.Write([]byte("x"))
|
||||
}
|
||||
_ = tw.Close()
|
||||
_ = gz.Close()
|
||||
dir := filepath.Join(t.TempDir(), "conv")
|
||||
if err := untarConversion(&buf, dir); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, want := range []string{"convert_lora_to_gguf.py", "conversion/qwen.py", "gguf-py/gguf/__init__.py"} {
|
||||
if _, err := os.Stat(filepath.Join(dir, want)); err != nil {
|
||||
t.Errorf("%s not kept", want)
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(dir, "src")); err == nil {
|
||||
t.Error("the rest of llama.cpp was kept")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
# llmgw's LoRA trainer (BACKLOG F4), run by the node agent with an installed
|
||||
# engine's Python (its torch and transformers) and a small overlay (peft,
|
||||
# accelerate). Reads a job, trains, writes the PEFT adapter, and says how it
|
||||
# goes as one JSON object a line on standard output.
|
||||
import json, math, os, random, sys, threading, subprocess, time
|
||||
|
||||
def say(**kw):
|
||||
print(json.dumps(kw), flush=True)
|
||||
|
||||
job = json.load(open(sys.argv[1], encoding="utf-8"))
|
||||
p = {"rank": 16, "alpha": 32, "dropout": 0.05, "epochs": 6, "lr": 2e-4, "batch": 4, "max_len": 512, "seed": 0}
|
||||
p.update(job.get("params") or {})
|
||||
|
||||
say(state="preparing", message="loading libraries")
|
||||
import torch
|
||||
from huggingface_hub import snapshot_download
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from peft import LoraConfig, get_peft_model
|
||||
|
||||
random.seed(p["seed"]); torch.manual_seed(p["seed"])
|
||||
if not torch.cuda.is_available():
|
||||
raise SystemExit("no CUDA device is visible to the trainer")
|
||||
|
||||
say(state="preparing", message="fetching " + job["base_repo"])
|
||||
base_dir = snapshot_download(job["base_repo"], revision=job.get("base_revision") or None,
|
||||
allow_patterns=["*.json", "*.safetensors", "*.txt", "*.model", "*.tiktoken"])
|
||||
say(state="preparing", message="loading the base model", base_dir=base_dir)
|
||||
|
||||
# bf16 where the card has it; fp16 with a loss scaler where it does not
|
||||
# (Turing and older).
|
||||
bf16 = torch.cuda.is_bf16_supported()
|
||||
dtype = torch.bfloat16 if bf16 else torch.float16
|
||||
tok = AutoTokenizer.from_pretrained(base_dir)
|
||||
if tok.pad_token_id is None:
|
||||
tok.pad_token = tok.eos_token
|
||||
model = AutoModelForCausalLM.from_pretrained(base_dir, dtype=dtype).to("cuda")
|
||||
|
||||
rows = [json.loads(l) for l in open(job["data"], encoding="utf-8") if l.strip()]
|
||||
rows = [r for r in rows if r.get("q") and r.get("a")]
|
||||
if not rows:
|
||||
raise SystemExit("no {q, a} pairs in the data")
|
||||
|
||||
def encode(r):
|
||||
prompt = tok.apply_chat_template([{"role": "user", "content": r["q"]}], add_generation_prompt=True, tokenize=False)
|
||||
full = tok.apply_chat_template([{"role": "user", "content": r["q"]}, {"role": "assistant", "content": r["a"]}], tokenize=False)
|
||||
pi = tok(prompt, add_special_tokens=False)["input_ids"]
|
||||
fi = tok(full, add_special_tokens=False)["input_ids"][: p["max_len"]]
|
||||
return fi, ([-100] * len(pi) + fi[len(pi):])[: len(fi)]
|
||||
|
||||
data = [encode(r) for r in rows]
|
||||
|
||||
model.gradient_checkpointing_enable()
|
||||
model.enable_input_require_grads()
|
||||
model = get_peft_model(model, LoraConfig(r=int(p["rank"]), lora_alpha=int(p["alpha"]), lora_dropout=float(p["dropout"]),
|
||||
task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]))
|
||||
trainable = [x for x in model.parameters() if x.requires_grad]
|
||||
for x in trainable: # fp32 trainable weights, so a loss scaler can unscale them
|
||||
x.data = x.data.float()
|
||||
|
||||
peak = {"mib": 0}
|
||||
stop = threading.Event()
|
||||
def watch():
|
||||
while not stop.is_set():
|
||||
try:
|
||||
out = subprocess.run(["nvidia-smi", "--query-compute-apps=pid,used_memory", "--format=csv,noheader,nounits"],
|
||||
capture_output=True, text=True, timeout=5).stdout
|
||||
for line in out.splitlines():
|
||||
pid, mib = [v.strip() for v in line.split(",")]
|
||||
if int(pid) == os.getpid():
|
||||
peak["mib"] = max(peak["mib"], int(mib))
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
threading.Thread(target=watch, daemon=True).start()
|
||||
|
||||
opt = torch.optim.AdamW(trainable, lr=float(p["lr"]), weight_decay=0.0)
|
||||
scaler = torch.amp.GradScaler("cuda", enabled=not bf16)
|
||||
bs, epochs = int(p["batch"]), int(p["epochs"])
|
||||
steps = epochs * math.ceil(len(data) / bs)
|
||||
warm = max(1, steps // 10)
|
||||
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda s: min(1.0, (s + 1) / warm) * max(0.0, 1 - s / steps))
|
||||
model.train(); step = 0; last = time.time()
|
||||
for ep in range(epochs):
|
||||
random.shuffle(data)
|
||||
for i in range(0, len(data), bs):
|
||||
batch = data[i:i + bs]; n = max(len(x[0]) for x in batch)
|
||||
ids = torch.tensor([x[0] + [tok.pad_token_id] * (n - len(x[0])) for x in batch], device="cuda")
|
||||
lab = torch.tensor([x[1] + [-100] * (n - len(x[1])) for x in batch], device="cuda")
|
||||
att = torch.tensor([[1] * len(x[0]) + [0] * (n - len(x[0])) for x in batch], device="cuda")
|
||||
with torch.autocast("cuda", dtype=dtype):
|
||||
loss = model(input_ids=ids, attention_mask=att, labels=lab).loss
|
||||
scaler.scale(loss).backward(); scaler.unscale_(opt)
|
||||
torch.nn.utils.clip_grad_norm_(trainable, 1.0)
|
||||
scaler.step(opt); scaler.update(); opt.zero_grad(set_to_none=True); sched.step(); step += 1
|
||||
if time.time() - last > 2 or step == steps:
|
||||
last = time.time()
|
||||
say(state="running", message="training", step=step, steps=steps, loss=round(loss.item(), 4), peak_mib=peak["mib"])
|
||||
|
||||
model.save_pretrained(job["out"])
|
||||
stop.set()
|
||||
say(state="trained", step=step, steps=steps, peak_mib=peak["mib"], base_dir=base_dir)
|
||||
@@ -0,0 +1,46 @@
|
||||
package agent
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/train"
|
||||
)
|
||||
|
||||
// trainJob runs a training job the control plane sent, telling it how the
|
||||
// job goes and sending what it made. A stream that ends ends the job: the
|
||||
// control plane, which sees the node go, marks it failed, and it is started
|
||||
// again from the beginning rather than resumed.
|
||||
func (a *Agent) trainJob(ctx context.Context, m *llmgwv1.TrainStart, out chan<- *llmgwv1.AgentMessage) {
|
||||
id := m.GetJobId()
|
||||
send := func(msg *llmgwv1.AgentMessage) bool {
|
||||
select {
|
||||
case out <- msg:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
}
|
||||
}
|
||||
status := func(s train.Status) {
|
||||
send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainStatus{TrainStatus: &llmgwv1.TrainStatus{
|
||||
JobId: id, State: s.State, Message: s.Message, Step: int32(s.Step), Steps: int32(s.Steps), //nolint:gosec // counts
|
||||
Loss: s.Loss, PeakMib: int32(s.PeakMiB)}}}) //nolint:gosec // MiB
|
||||
}
|
||||
artifact := func(kind string, seq int, data []byte, last bool) error {
|
||||
if !send(&llmgwv1.AgentMessage{Msg: &llmgwv1.AgentMessage_TrainArtifact{TrainArtifact: &llmgwv1.TrainArtifact{
|
||||
JobId: id, Kind: kind, Seq: int32(seq), Data: data, Last: last}}}) { //nolint:gosec // chunk count
|
||||
return ctx.Err()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
a.log.Info("training an adapter", "job", id, "base", m.GetBaseRepo())
|
||||
err := a.trainer.Run(ctx, train.Job{ID: id, BaseRepo: m.GetBaseRepo(), BaseRevision: m.GetBaseRevision(),
|
||||
Data: m.GetData(), Params: m.GetParams(), GPUMemoryMiB: int(m.GetGpuMemoryMib()), GGUF: m.GetGguf()}, status, artifact)
|
||||
if err != nil {
|
||||
a.log.Warn("training failed", "job", id, "err", err)
|
||||
status(train.Status{State: "failed", Message: err.Error()})
|
||||
return
|
||||
}
|
||||
a.log.Info("trained an adapter", "job", id)
|
||||
status(train.Status{State: "done"})
|
||||
}
|
||||
@@ -249,7 +249,14 @@ func pgEnv(dsn string) ([]string, error) {
|
||||
set("PGPASSWORD", pw)
|
||||
}
|
||||
set("PGDATABASE", strings.TrimPrefix(u.Path, "/"))
|
||||
set("PGSSLMODE", u.Query().Get("sslmode"))
|
||||
// The TLS settings go with the mode: verify-full without the cluster's
|
||||
// authority sent libpq looking for ~/.postgresql/root.crt, and the dump
|
||||
// failed on every control plane whose store is replicated.
|
||||
q := u.Query()
|
||||
set("PGSSLMODE", q.Get("sslmode"))
|
||||
set("PGSSLROOTCERT", q.Get("sslrootcert"))
|
||||
set("PGSSLCERT", q.Get("sslcert"))
|
||||
set("PGSSLKEY", q.Get("sslkey"))
|
||||
// A standby if there is one, else whichever answers — not the address's
|
||||
// read-write: a dump only reads, it need not load the primary, and one
|
||||
// that insists on the primary fails exactly while the cluster is failing
|
||||
|
||||
@@ -238,3 +238,17 @@ func TestAStoreOfSeveralHostsIsGivenToPgDumpAsLibpqLists(t *testing.T) {
|
||||
t.Error("an empty host in the list was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPgDumpIsGivenTheClusterAuthorityAlongWithVerifyFull(t *testing.T) {
|
||||
// Without it libpq looked for ~/.postgresql/root.crt and the dump failed.
|
||||
env, err := pgEnv("postgres://llmgw:p@10.0.0.149:5432,10.0.0.150:5432/llmgw?sslmode=verify-full&sslrootcert=%2Fetc%2Fllmgw%2Fpki%2Fca.crt&target_session_attrs=read-write")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := strings.Join(env, " ")
|
||||
for _, want := range []string{"PGSSLMODE=verify-full", "PGSSLROOTCERT=/etc/llmgw/pki/ca.crt"} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("%s is not in %s", want, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
)
|
||||
|
||||
// comfyKeyEnv carries the replica's key to the gate: the environment, which
|
||||
// only the engine's user and root can read, rather than the command line,
|
||||
// which every user on the node can.
|
||||
const comfyKeyEnv = "LLMGW_ENGINE_KEY"
|
||||
|
||||
func newComfyUIGateCmd() *cobra.Command {
|
||||
var cfg comfy.GateConfig
|
||||
cmd := &cobra.Command{
|
||||
Use: "comfyui-gate [flags] [-- main.py flags]",
|
||||
Short: "Run ComfyUI as an engine: behind a key, on the loopback (started by the agent)",
|
||||
Hidden: true,
|
||||
Long: "Started by the node agent for a replica whose engine is comfyui. ComfyUI asks\n" +
|
||||
"nobody for a key, so it listens on the loopback only and this gate on the\n" +
|
||||
"replica's port, letting through, with the replica's key, only what the router\n" +
|
||||
"asks of it. It also lays the fetched weights out where ComfyUI looks for them.",
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
cfg.Key = os.Getenv(comfyKeyEnv)
|
||||
_ = os.Unsetenv(comfyKeyEnv)
|
||||
cfg.Args = args
|
||||
cfg.Log = logger()
|
||||
return comfy.RunGate(ctx, cfg)
|
||||
},
|
||||
}
|
||||
f := cmd.Flags()
|
||||
f.StringVar(&cfg.Listen, "listen", "", "the replica's address")
|
||||
f.StringVar(&cfg.Model, "model", "", "the model's name, for /v1/models")
|
||||
f.StringVar(&cfg.Weights, "weights", "", "the fetched weights: a file or a directory")
|
||||
f.StringVar(&cfg.State, "state", "", "the replica's own directory")
|
||||
f.StringVar(&cfg.Python, "python", "python3", "the interpreter of ComfyUI's environment")
|
||||
f.StringVar(&cfg.Dir, "comfyui", "", "ComfyUI's directory, where main.py is")
|
||||
for _, n := range []string{"listen", "weights", "state", "comfyui"} {
|
||||
_ = cmd.MarkFlagRequired(n)
|
||||
}
|
||||
return cmd
|
||||
}
|
||||
@@ -116,6 +116,11 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
|
||||
if out == nil {
|
||||
return nil
|
||||
}
|
||||
// A stream, as it comes: NDJSON a long run writes line by line.
|
||||
if w, ok := out.(io.Writer); ok {
|
||||
_, err := io.Copy(w, resp.Body)
|
||||
return err
|
||||
}
|
||||
// An older control plane answers a path it does not know with its
|
||||
// dashboard: a page, which read as JSON said only "invalid character '<'".
|
||||
if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {
|
||||
@@ -181,6 +186,7 @@ func newCtlCmd() *cobra.Command {
|
||||
newQuotaCmd(client, &asJSON),
|
||||
newGuardCmd(client, &asJSON),
|
||||
newKBCmd(client, &asJSON),
|
||||
newTrainCmd(client, &asJSON),
|
||||
newConfigCmd(client, &asJSON),
|
||||
newAuditCmd(client, &asJSON),
|
||||
newNotifyCmd(client, &asJSON),
|
||||
@@ -325,6 +331,9 @@ func modelStatus(m adminapi.ModelView) string {
|
||||
return "stopped"
|
||||
case m.Ready >= m.Min && m.Min > 0:
|
||||
return "ready"
|
||||
case m.Min == 0 && m.Ready > 0 && m.Ready >= m.Total:
|
||||
// Asked for, and up until it goes unused for long enough.
|
||||
return "ready (on demand)"
|
||||
case m.Total > m.Ready:
|
||||
return "starting"
|
||||
default:
|
||||
|
||||
@@ -56,6 +56,9 @@ func newRequestCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
fmt.Fprintf(out, "status %d\n", v.Status)
|
||||
fmt.Fprintf(out, "time %d ms total: %d queued, %d to the first token\n", v.LatencyMs, v.QueueMs, v.TTFTMs)
|
||||
fmt.Fprintf(out, "tokens %d in, %d out\n", v.PromptTokens, v.CompletionTokens)
|
||||
if v.Images > 0 {
|
||||
fmt.Fprintf(out, "images %d\n", v.Images)
|
||||
}
|
||||
if v.CacheHit {
|
||||
fmt.Fprintf(out, "cache answered from the project's answer cache: nothing generated, nothing charged\n")
|
||||
}
|
||||
|
||||
+49
-21
@@ -7,12 +7,14 @@ import (
|
||||
"fmt"
|
||||
"net/netip"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
)
|
||||
|
||||
// firewallStateFile is where a control plane writes what the cluster's
|
||||
@@ -20,11 +22,9 @@ import (
|
||||
// the server itself runs as llmgw and cannot change the ruleset.
|
||||
const firewallStateFile = "/var/lib/llmgw/firewall.json"
|
||||
|
||||
// firewallState is what is written there.
|
||||
type firewallState struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
}
|
||||
// firewallState is what is written there: the same as the mirror is told
|
||||
// and keeps beside its models.
|
||||
type firewallState = mirror.FirewallState
|
||||
|
||||
// writeFirewallState records the settings' firewall for this machine's sync
|
||||
// unit. Without the state directory — a server not run by systemd — there
|
||||
@@ -46,25 +46,34 @@ func writeFirewallState(cfg config.Server) error {
|
||||
|
||||
// The units that apply it: a path unit watching the file, and the oneshot it
|
||||
// starts.
|
||||
const (
|
||||
firewallSyncPathUnit = "llmgw-firewall-sync.path"
|
||||
firewallSyncUnitFile = `[Unit]
|
||||
Description=llmgw firewall: apply the cluster's settings on this control plane
|
||||
const firewallSyncPathUnit = "llmgw-firewall-sync.path"
|
||||
|
||||
// firewallSyncUnits are the two units for a state file: on a control plane
|
||||
// the one its server writes, on a mirror the one the control plane has the
|
||||
// mirror write.
|
||||
func firewallSyncUnits(state string) (service, pathUnit string) {
|
||||
arg := ""
|
||||
if state != firewallStateFile {
|
||||
arg = " --state " + state
|
||||
}
|
||||
service = `[Unit]
|
||||
Description=llmgw firewall: apply the cluster's settings on this machine
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/bin/llmgw firewall sync
|
||||
ExecStart=/usr/local/bin/llmgw firewall sync` + arg + `
|
||||
`
|
||||
firewallSyncPathFile = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this control plane
|
||||
pathUnit = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this machine
|
||||
|
||||
[Path]
|
||||
PathChanged=` + firewallStateFile + `
|
||||
PathChanged=` + state + `
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
`
|
||||
)
|
||||
return service, pathUnit
|
||||
}
|
||||
|
||||
func newFirewallCmd() *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
@@ -76,6 +85,7 @@ func newFirewallCmd() *cobra.Command {
|
||||
"which " + firewallSyncPathUnit + " applies with 'llmgw firewall sync'.",
|
||||
}
|
||||
var install bool
|
||||
state := firewallStateFile
|
||||
sync := &cobra.Command{
|
||||
Use: "sync",
|
||||
Short: "Apply what the cluster's settings say of the firewall, here (root)",
|
||||
@@ -85,21 +95,21 @@ func newFirewallCmd() *cobra.Command {
|
||||
return errors.New("the firewall is changed as root: run it with sudo")
|
||||
}
|
||||
if install {
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}); err != nil {
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintln(cmd.OutOrStdout(), firewallSyncPathUnit+" follows the cluster's settings here")
|
||||
if _, err := os.Stat(firewallStateFile); err != nil {
|
||||
return nil // the server writes it when it starts
|
||||
if _, err := os.Stat(state); err != nil {
|
||||
return nil // written when the settings are first told
|
||||
}
|
||||
}
|
||||
b, err := os.ReadFile(firewallStateFile)
|
||||
b, err := os.ReadFile(state)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var st firewallState
|
||||
if err := json.Unmarshal(b, &st); err != nil {
|
||||
return fmt.Errorf("%s: %w", firewallStateFile, err)
|
||||
return fmt.Errorf("%s: %w", state, err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(cmd.Context(), 2*time.Minute)
|
||||
defer cancel()
|
||||
@@ -112,14 +122,17 @@ func newFirewallCmd() *cobra.Command {
|
||||
},
|
||||
}
|
||||
sync.Flags().BoolVar(&install, "install", false, "first install the units that run this whenever the settings change")
|
||||
sync.Flags().StringVar(&state, "state", firewallStateFile,
|
||||
"the file the settings are written to: a control plane's, or "+mirror.FirewallFile+" in a mirror's directory")
|
||||
c.AddCommand(sync)
|
||||
return c
|
||||
}
|
||||
|
||||
// installFirewallSync writes the sync units on a machine and has the path
|
||||
// unit follow the state file.
|
||||
func installFirewallSync(ctx context.Context, r firewall.Runner) error {
|
||||
body := firewallSyncUnitFile + "\x00" + firewallSyncPathFile
|
||||
func installFirewallSync(ctx context.Context, r firewall.Runner, state string) error {
|
||||
service, pathUnit := firewallSyncUnits(state)
|
||||
body := service + "\x00" + pathUnit
|
||||
out, err := r.Run(ctx, `set -e
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
@@ -172,5 +185,20 @@ func syncFirewall(ctx context.Context, r firewall.Runner, st firewallState) (str
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
// The cluster's machines: a node on a network this one has not let in
|
||||
// yet, which the mirror's files are for.
|
||||
if len(st.Toward) > 0 {
|
||||
added, err := firewall.Admit(ctx, r, st.Toward)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(added) > 0 {
|
||||
shown := make([]string, len(added))
|
||||
for i, p := range added {
|
||||
shown[i] = p.String()
|
||||
}
|
||||
return "llmgw's firewall is on here, and now lets in " + strings.Join(shown, ", "), nil
|
||||
}
|
||||
}
|
||||
return "llmgw's firewall is on here", nil
|
||||
}
|
||||
@@ -2,8 +2,11 @@ package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/netip"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/firewall"
|
||||
)
|
||||
|
||||
type scriptLog struct {
|
||||
@@ -49,3 +52,32 @@ func TestAControlPlaneFollowsTheFirewallSetting(t *testing.T) {
|
||||
t.Errorf("on: %q %v", msg, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMachineLetsInTheClusterMachinesItIsTold(t *testing.T) {
|
||||
// The rules hold 10.0.0.0/24; a node on 192.168.7.0/24 is let in, as
|
||||
// the route to it says, and one already in is not asked about.
|
||||
r := &scriptLog{reply: func(s string) string {
|
||||
switch {
|
||||
case strings.Contains(s, "echo loaded"):
|
||||
return "loaded\n"
|
||||
case strings.Contains(s, "cat /etc/llmgw/firewall.nft"):
|
||||
return firewallRulesFor("10.0.0.0/24")
|
||||
case strings.Contains(s, "ip -o route get 192.168.7.20"):
|
||||
return "192.168.7.20 dev eth1 src 192.168.7.1 uid 0\n2: eth1 inet 192.168.7.1/24 brd 192.168.7.255 scope global eth1\n"
|
||||
case strings.Contains(s, "nft -f"):
|
||||
return "changed\n"
|
||||
}
|
||||
return ""
|
||||
}}
|
||||
msg, err := syncFirewall(context.Background(), r, firewallState{Enabled: true, Toward: []string{"10.0.0.147", "192.168.7.20"}})
|
||||
if err != nil || !strings.Contains(msg, "192.168.7.0/24") {
|
||||
t.Fatalf("%q %v", msg, err)
|
||||
}
|
||||
if strings.Contains(strings.Join(r.scripts, "\n"), "route get 10.0.0.147") {
|
||||
t.Error("asked the route to a machine already let in")
|
||||
}
|
||||
}
|
||||
|
||||
func firewallRulesFor(nets string) string {
|
||||
return firewall.Script(firewall.Spec{Ports: []firewall.Port{firewall.MirrorFiles}, Allow: []netip.Prefix{netip.MustParsePrefix(nets)}})
|
||||
}
|
||||
@@ -0,0 +1,146 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/netip"
|
||||
"net/url"
|
||||
"slices"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// firewallStateKey is where, in the store, the firewall's state is kept
|
||||
// for the store's members to read.
|
||||
const firewallStateKey = "firewall"
|
||||
|
||||
// firewallPublisher tells the machines that cannot ask a control plane
|
||||
// what the cluster's settings say of their firewall: the mirror, over its
|
||||
// API, and the store's members, through the store they hold a copy of.
|
||||
// Nodes are told over their agents' streams and control planes by their
|
||||
// own servers. On the leader only, at start, at every change to the
|
||||
// firewall's settings, and every few minutes for nodes come and gone.
|
||||
type firewallPublisher struct {
|
||||
settings func() config.Server
|
||||
st *store.DB
|
||||
nodes func() []string
|
||||
leader func() bool
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
mirror *mirror.Client
|
||||
kick chan struct{}
|
||||
}
|
||||
|
||||
func newFirewallPublisher(settings func() config.Server, st *store.DB, nodes func() []string,
|
||||
leader func() bool, log *slog.Logger) *firewallPublisher {
|
||||
return &firewallPublisher{settings: settings, st: st, nodes: nodes, leader: leader, log: log,
|
||||
kick: make(chan struct{}, 1)}
|
||||
}
|
||||
|
||||
// setMirror has the mirror told too.
|
||||
func (p *firewallPublisher) setMirror(m *mirror.Client) {
|
||||
p.mu.Lock()
|
||||
p.mirror = m
|
||||
p.mu.Unlock()
|
||||
p.Kick()
|
||||
}
|
||||
|
||||
// Kick has the state worked out and told again now.
|
||||
func (p *firewallPublisher) Kick() {
|
||||
select {
|
||||
case p.kick <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
func (p *firewallPublisher) run(ctx context.Context, every time.Duration) {
|
||||
t := time.NewTicker(every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
p.once(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
case <-p.kick:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// state is what the settings say now, with the cluster's machines.
|
||||
func (p *firewallPublisher) state(ctx context.Context) mirror.FirewallState {
|
||||
cfg := p.settings()
|
||||
st := mirror.FirewallState{Enabled: cfg.Firewall.Enabled, Networks: slices.Clone(cfg.FirewallNetworks)}
|
||||
var toward []string
|
||||
if p.nodes != nil {
|
||||
toward = append(toward, p.nodes()...)
|
||||
}
|
||||
if p.st != nil {
|
||||
if cps, err := p.st.ControlPlanes(ctx); err == nil {
|
||||
for _, c := range cps {
|
||||
toward = append(toward, c.URL)
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, t := range toward {
|
||||
if a, ok := hostAddr(t); ok {
|
||||
st.Toward = append(st.Toward, a)
|
||||
}
|
||||
}
|
||||
slices.Sort(st.Toward)
|
||||
st.Toward = slices.Compact(st.Toward)
|
||||
return st
|
||||
}
|
||||
|
||||
// hostAddr is the address in an address, host:port or URL; names are not
|
||||
// looked up here.
|
||||
func hostAddr(s string) (string, bool) {
|
||||
if u, err := url.Parse(s); err == nil && u.Host != "" {
|
||||
s = u.Hostname()
|
||||
} else if h, _, err := net.SplitHostPort(s); err == nil {
|
||||
s = h
|
||||
}
|
||||
a, err := netip.ParseAddr(s)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
return a.Unmap().String(), true
|
||||
}
|
||||
|
||||
func (p *firewallPublisher) once(ctx context.Context) {
|
||||
if p.leader != nil && !p.leader() {
|
||||
return
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
st := p.state(ctx)
|
||||
p.mu.Lock()
|
||||
m := p.mirror
|
||||
p.mu.Unlock()
|
||||
if m != nil {
|
||||
changed, err := m.SetFirewall(ctx, st)
|
||||
switch {
|
||||
case err != nil:
|
||||
p.log.Warn("could not tell the mirror what the settings say of its firewall", "err", err)
|
||||
case changed:
|
||||
p.log.Info("the mirror's firewall follows the cluster's settings", "enabled", st.Enabled)
|
||||
}
|
||||
}
|
||||
if p.st != nil {
|
||||
b, _ := json.Marshal(st)
|
||||
changed, err := p.st.PutClusterState(ctx, firewallStateKey, string(b))
|
||||
switch {
|
||||
case err != nil:
|
||||
p.log.Warn("could not record the firewall's state for the store's members", "err", err)
|
||||
case changed:
|
||||
p.log.Info("the store's members' firewall follows the cluster's settings", "enabled", st.Enabled)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestTheLeaderTellsTheMirrorAndTheStoreWhatTheFirewallSettingsSay(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
st, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer st.Close()
|
||||
if err := st.PutControlPlane(ctx, store.ControlPlane{URL: "https://10.0.0.150:7443", Name: "ctl02"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
root := t.TempDir()
|
||||
srv := httptest.NewServer(mirror.NewAPI(mirror.NewManager(root, slog.New(slog.DiscardHandler)), "http://m:8081").Handler())
|
||||
defer srv.Close()
|
||||
mc, _ := mirror.NewClient(srv.URL, tlsconf.Config{}, 0)
|
||||
|
||||
cfg := config.Server{Firewall: config.FirewallSettings{Enabled: false}, FirewallNetworks: []string{"192.168.5.0/24"}}
|
||||
leader := false
|
||||
p := newFirewallPublisher(func() config.Server { return cfg }, st,
|
||||
func() []string { return []string{"10.0.0.147", "", "10.0.0.147", "node-by-name"} },
|
||||
func() bool { return leader }, slog.New(slog.DiscardHandler))
|
||||
p.setMirror(mc)
|
||||
|
||||
p.once(ctx)
|
||||
if _, err := st.ClusterState(ctx, firewallStateKey); err == nil {
|
||||
t.Error("a standby told the store")
|
||||
}
|
||||
leader = true
|
||||
p.once(ctx)
|
||||
var got mirror.FirewallState
|
||||
v, err := st.ClusterState(ctx, firewallStateKey)
|
||||
if err != nil || json.Unmarshal([]byte(v), &got) != nil {
|
||||
t.Fatalf("the store holds %q %v", v, err)
|
||||
}
|
||||
if got.Enabled || got.Networks[0] != "192.168.5.0/24" || len(got.Toward) != 2 ||
|
||||
got.Toward[0] != "10.0.0.147" || got.Toward[1] != "10.0.0.150" {
|
||||
t.Errorf("state %+v", got)
|
||||
}
|
||||
b, err := os.ReadFile(filepath.Join(root, mirror.FirewallFile))
|
||||
if err != nil || string(b) != v {
|
||||
t.Errorf("the mirror holds %s, the store %s", b, v)
|
||||
}
|
||||
}
|
||||
@@ -1,10 +1,14 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
@@ -212,6 +216,45 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
def.Flags().BoolVar(&clear, "clear", false, "no default any more")
|
||||
c.AddCommand(def)
|
||||
|
||||
var dsModel, dsOut string
|
||||
var dsPer, dsMax int
|
||||
dataset := &cobra.Command{
|
||||
Use: "dataset COLLECTION --model MODEL",
|
||||
Short: "Write training data from a collection: questions and answers, for an adapter",
|
||||
Long: "A chat model of the cluster reads each chunk of the collection and writes\n" +
|
||||
"questions a person might ask of it, with the answers it holds, one JSON object\n" +
|
||||
"a line ({\"q\", \"a\", \"doc\", \"ord\"}). Read it through and correct it before\n" +
|
||||
"training on it (llmgw ctl train): what the model got wrong, an adapter learns.\n\n" +
|
||||
"An adapter learns how answers sound — the organisation's words, format and\n" +
|
||||
"tone — far better than what they say; the facts stay the knowledge base's.",
|
||||
Example: " llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
w := cmd.OutOrStdout()
|
||||
if dsOut != "" {
|
||||
f, err := os.Create(dsOut)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
w = f
|
||||
}
|
||||
lw := &datasetWriter{pairs: w, problems: cmd.ErrOrStderr()}
|
||||
body, _ := json.Marshal(adminapi.KBDatasetRequest{Model: dsModel, PerChunk: dsPer, MaxChunks: dsMax})
|
||||
err := client().patient(6*time.Hour).do(cmd.Context(), "POST",
|
||||
"/api/v1/kb/collections/"+url.PathEscape(args[0])+"/dataset", json.RawMessage(body), lw)
|
||||
lw.flush()
|
||||
fmt.Fprintf(cmd.ErrOrStderr(), "%d pairs from %d chunks; %d chunks gave none\n", lw.n, len(lw.chunks), lw.failed)
|
||||
return err
|
||||
},
|
||||
}
|
||||
dataset.Flags().StringVar(&dsModel, "model", "", "the chat model that writes the questions and answers")
|
||||
dataset.Flags().StringVar(&dsOut, "out", "", "the file to write (default: standard output)")
|
||||
dataset.Flags().IntVar(&dsPer, "per-chunk", 8, "pairs from each chunk")
|
||||
dataset.Flags().IntVar(&dsMax, "max-chunks", 500, "the most chunks read")
|
||||
_ = dataset.MarkFlagRequired("model")
|
||||
c.AddCommand(dataset)
|
||||
|
||||
var k int
|
||||
search := &cobra.Command{
|
||||
Use: "search COLLECTION QUERY...",
|
||||
@@ -240,3 +283,50 @@ func newKBCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c.AddCommand(search)
|
||||
return c
|
||||
}
|
||||
|
||||
// datasetWriter splits the dataset stream: pairs to the file, problems to
|
||||
// the terminal, counted as they pass.
|
||||
type datasetWriter struct {
|
||||
pairs, problems io.Writer
|
||||
buf []byte
|
||||
n, failed int
|
||||
chunks map[string]bool
|
||||
}
|
||||
|
||||
func (d *datasetWriter) Write(p []byte) (int, error) {
|
||||
d.buf = append(d.buf, p...)
|
||||
for {
|
||||
i := bytes.IndexByte(d.buf, '\n')
|
||||
if i < 0 {
|
||||
return len(p), nil
|
||||
}
|
||||
d.line(d.buf[:i])
|
||||
d.buf = d.buf[i+1:]
|
||||
}
|
||||
}
|
||||
|
||||
func (d *datasetWriter) flush() {
|
||||
if len(bytes.TrimSpace(d.buf)) > 0 {
|
||||
d.line(d.buf)
|
||||
}
|
||||
d.buf = nil
|
||||
}
|
||||
|
||||
func (d *datasetWriter) line(b []byte) {
|
||||
var l adminapi.KBDatasetLine
|
||||
if json.Unmarshal(b, &l) != nil {
|
||||
return
|
||||
}
|
||||
if d.chunks == nil {
|
||||
d.chunks = map[string]bool{}
|
||||
}
|
||||
d.chunks[fmt.Sprintf("%s#%d", l.Doc, l.Ord)] = true
|
||||
if l.Error != "" {
|
||||
d.failed++
|
||||
fmt.Fprintf(d.problems, " %s #%d: %s\n", l.Doc, l.Ord, l.Error)
|
||||
return
|
||||
}
|
||||
d.n++
|
||||
out, _ := json.Marshal(l)
|
||||
fmt.Fprintln(d.pairs, string(out))
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net"
|
||||
"net/netip"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -65,6 +66,14 @@ Safe to run again: it changes only what differs.`,
|
||||
}
|
||||
if !noFirewall {
|
||||
mirrorFirewall(cmd.Context(), cmd.OutOrStdout(), controlPlanes)
|
||||
// And after, as the cluster's settings say: the control plane
|
||||
// tells the mirror, which writes it beside its models.
|
||||
state := filepath.Join(*dir, mirror.FirewallFile)
|
||||
if err := installFirewallSync(cmd.Context(), firewall.Local{}, state); err != nil {
|
||||
fmt.Fprintf(cmd.OutOrStdout(), " !! firewall: %v; the cluster's settings will not change it here\n", err)
|
||||
} else {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), " ok firewall: follows the cluster's settings ("+firewallSyncPathUnit+")")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
},
|
||||
|
||||
@@ -54,6 +54,7 @@ func Root() *cobra.Command {
|
||||
newMirrorCmd(),
|
||||
newBundleCmd(),
|
||||
newSimEngineCmd(),
|
||||
newComfyUIGateCmd(),
|
||||
newDoctorCmd(),
|
||||
newNodeCmd(),
|
||||
newSelfUpgradeCmd(),
|
||||
|
||||
@@ -49,6 +49,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kvevents"
|
||||
@@ -232,6 +233,9 @@ func newServerCmd() *cobra.Command {
|
||||
var agentHandler http.Handler
|
||||
var agentPath string
|
||||
var admin *control.Controller
|
||||
// What the settings say of the firewall, for the mirror and the
|
||||
// store's members; nil without a control plane.
|
||||
var fwPub *firewallPublisher
|
||||
updateMetrics := &modelserve.UpdateCollector{}
|
||||
// The store's cluster, when it is one; read by the metrics and
|
||||
// by `ctl status`.
|
||||
@@ -275,6 +279,19 @@ func newServerCmd() *cobra.Command {
|
||||
ctl.Reconciler().SetNotifier(notifier)
|
||||
admin = ctl
|
||||
agentPath, agentHandler = svc.Handler()
|
||||
fwPub = newFirewallPublisher(func() config.Server { return *current.Load() }, rt.Store(),
|
||||
func() []string {
|
||||
var out []string
|
||||
for _, n := range ctl.Inventory().Nodes() {
|
||||
out = append(out, n.MgmtAddr)
|
||||
}
|
||||
return out
|
||||
},
|
||||
func() bool { return ctl.Leadership() == nil || ctl.Leadership().IsLeader() }, log)
|
||||
go fwPub.run(ctx, 5*time.Minute)
|
||||
if l := ctl.Leadership(); l != nil {
|
||||
l.OnLead(fwPub.Kick)
|
||||
}
|
||||
|
||||
// Before AdminHandler, which is where the registry is built —
|
||||
// registering after it silently publishes nothing, which is
|
||||
@@ -387,6 +404,13 @@ func newServerCmd() *cobra.Command {
|
||||
// Knowledge bases, with the cluster's own models to embed a
|
||||
// probe (a collection's dimension) and an operator's search.
|
||||
api.SetKB(&adminapi.KBAdmin{Store: st, Embed: embedder(rt.Replay)})
|
||||
// LoRA adapters trained on the nodes.
|
||||
if agentSvc != nil {
|
||||
trainer := train.New(st, agentSvc, log)
|
||||
agentSvc.SetTrain(trainer)
|
||||
go trainer.Run(ctx)
|
||||
api.SetTrain(&adminapi.TrainAdmin{Store: st, Service: trainer})
|
||||
}
|
||||
// The cluster's settings, each change checked against this
|
||||
// machine's own keys before it is kept.
|
||||
api.SetSettings(&adminapi.Settings{Store: st, LocalKeys: config.LocalKeys(),
|
||||
@@ -449,6 +473,7 @@ func newServerCmd() *cobra.Command {
|
||||
api.SetUsage(st)
|
||||
// Capacity quotas, set and read over this API.
|
||||
api.SetQuotas(st)
|
||||
api.SetTenancy(st)
|
||||
}
|
||||
api.Register(adminMux)
|
||||
|
||||
@@ -510,6 +535,16 @@ func newServerCmd() *cobra.Command {
|
||||
return fmt.Errorf("the model mirror: %w", err)
|
||||
}
|
||||
rt.SetMirror(mc)
|
||||
if fwPub != nil {
|
||||
fwPub.setMirror(mc)
|
||||
}
|
||||
// Adapters trained on the nodes are kept there too; sending one
|
||||
// takes longer than the mirror's usual answers.
|
||||
if adminAPI != nil {
|
||||
if up, err := mirror.NewClient(cfg.Mirror.API, cfg.Mirror.TLS, 10*time.Minute); err == nil {
|
||||
adminAPI.SetTrainMirror(up)
|
||||
}
|
||||
}
|
||||
// Browsing the registry is what turns "paste a resolve URL"
|
||||
// into "find the model". It needs a way out to the internet
|
||||
// from this machine, which the mirror's own fetching does not.
|
||||
@@ -727,6 +762,9 @@ func newServerCmd() *cobra.Command {
|
||||
if agentSvc != nil {
|
||||
agentSvc.BroadcastFirewall()
|
||||
}
|
||||
if fwPub != nil {
|
||||
fwPub.Kick()
|
||||
}
|
||||
}
|
||||
if slices.ContainsFunc(changed, func(c string) bool { return strings.HasPrefix(c, "agents") }) && agentSvc != nil {
|
||||
// A moment apart per node, so off this goroutine.
|
||||
|
||||
@@ -160,7 +160,7 @@ func (sa *serverAdd) firewall(ctx context.Context) error {
|
||||
return err
|
||||
}
|
||||
// What applies the cluster's firewall setting there when it changes.
|
||||
if err := installFirewallSync(ctx, sa.remote); err != nil {
|
||||
if err := installFirewallSync(ctx, sa.remote, firewallStateFile); err != nil {
|
||||
fmt.Fprintf(sa.out, " !! firewall: %v\n", err)
|
||||
}
|
||||
if !cfg.Firewall.Enabled {
|
||||
|
||||
@@ -93,6 +93,11 @@ func (in *serverInit) storeFirewall(ctx context.Context, members []pgha.Node, se
|
||||
} else {
|
||||
in.kept("firewall on %s", m.Name)
|
||||
}
|
||||
// And after, as the cluster's settings say, read from the
|
||||
// member's own copy of the store.
|
||||
if err := firewall.InstallFollower(ctx, m.Remote); err != nil {
|
||||
fmt.Fprintf(in.out, " !! firewall on %s: %v; the cluster's settings will not change it\n", m.Name, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
}
|
||||
@@ -135,5 +140,5 @@ func loopbackListen(listen string) bool {
|
||||
// ensureFirewallSync installs the units that apply the cluster's firewall
|
||||
// setting on this control plane.
|
||||
func (in *serverInit) ensureFirewallSync(ctx context.Context) error {
|
||||
return installFirewallSync(ctx, firewall.Local{})
|
||||
return installFirewallSync(ctx, firewall.Local{}, firewallStateFile)
|
||||
}
|
||||
@@ -16,6 +16,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/provision"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/pgha"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
@@ -348,15 +349,23 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
|
||||
}
|
||||
|
||||
if installed {
|
||||
// Its defaults assume it owns the machine. A gateway swapped out to
|
||||
// give the database cache is worse than a smaller cache, and a store
|
||||
// whose working set is megabytes needs little. Only a PostgreSQL init
|
||||
// installed itself: one that was already here may serve something
|
||||
// else that needs its connections.
|
||||
for _, q := range []string{
|
||||
"ALTER SYSTEM SET shared_buffers = '96MB'",
|
||||
"ALTER SYSTEM SET max_connections = 40",
|
||||
} {
|
||||
// Sized to the machine's memory, as one it shares with the gateway:
|
||||
// a gateway swapped out to give the database cache is worse than a
|
||||
// smaller cache, but a knowledge base's vector index is searched
|
||||
// well only from memory. Only a PostgreSQL init installed itself:
|
||||
// one that was already here may serve something else that needs its
|
||||
// connections.
|
||||
meminfo, _ := os.ReadFile("/proc/meminfo")
|
||||
total, ok := pgha.MemTotal(string(meminfo))
|
||||
if !ok {
|
||||
total = 1 << 30
|
||||
}
|
||||
mem := pgha.MemorySettings(total, true)
|
||||
qs := []string{"ALTER SYSTEM SET max_connections = 40"}
|
||||
for _, k := range []string{"shared_buffers", "effective_cache_size", "maintenance_work_mem", "work_mem"} {
|
||||
qs = append(qs, "ALTER SYSTEM SET "+k+" = '"+mem[k]+"'")
|
||||
}
|
||||
for _, q := range qs {
|
||||
if _, err := in.psql(ctx, q); err != nil {
|
||||
return "", err
|
||||
}
|
||||
@@ -367,7 +376,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
|
||||
if err := in.waitForPostgres(ctx); err != nil {
|
||||
return "", err
|
||||
}
|
||||
in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
|
||||
in.done("PostgreSQL sized for a machine it shares, with %d MB of memory (shared_buffers %s, work_mem %s, 40 connections)",
|
||||
total>>20, mem["shared_buffers"], mem["work_mem"])
|
||||
}
|
||||
|
||||
in.setupPgvector(ctx, discovery.DiscoverHost())
|
||||
|
||||
@@ -52,6 +52,8 @@ func TestModelStatusSeparatesWaitingFromBroken(t *testing.T) {
|
||||
{"nothing could be placed", adminapi.ModelView{Min: 1, Ready: 0, Total: 0}, "not placed"},
|
||||
{"deliberately not running", adminapi.ModelView{Min: 0, Ready: 0, Total: 0}, "stopped"},
|
||||
{"short but rebuilding", adminapi.ModelView{Min: 2, Ready: 1, Total: 2}, "starting"},
|
||||
{"woken by a request", adminapi.ModelView{Min: 0, Ready: 1, Total: 1}, "ready (on demand)"},
|
||||
{"waking", adminapi.ModelView{Min: 0, Ready: 0, Total: 1}, "starting"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := modelStatus(tc.view); got != tc.want {
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
package cli
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func newTrainCmd(client func() *ctlClient, asJSON *bool) *cobra.Command {
|
||||
c := &cobra.Command{
|
||||
Use: "train",
|
||||
Short: "Train LoRA adapters on the nodes, on question/answer pairs",
|
||||
Long: "A node trains an adapter for a catalog model on pairs (llmgw ctl kb dataset\n" +
|
||||
"writes them from a knowledge base; read them through first). It trains with\n" +
|
||||
"its installed vLLM's torch and a small overlay, as the engines' user, on a card\n" +
|
||||
"with --gpu-memory free — beside an engine, if it leaves that much. What it\n" +
|
||||
"makes is kept by the control plane: llmgw ctl train download, or publish it\n" +
|
||||
"as the model's adapter.\n\n" +
|
||||
"An adapter learns how answers sound — words, format, tone — far better than\n" +
|
||||
"what they say. Facts belong to the knowledge base.",
|
||||
}
|
||||
|
||||
var model, base, rev, node, data, owner string
|
||||
var public bool
|
||||
var gpuMem int
|
||||
var epochs, rank int
|
||||
var lr float64
|
||||
var noGGUF bool
|
||||
start := &cobra.Command{
|
||||
Use: "start NAME --model MODEL --base-repo OWNER/NAME --node NODE --data FILE",
|
||||
Short: "Queue an adapter's training on a node",
|
||||
Example: " llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \\\n" +
|
||||
" --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
b, err := os.ReadFile(data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
req := adminapi.TrainRequest{Name: args[0], Model: model, BaseRepo: base, BaseRevision: rev, Node: node,
|
||||
GPUMemoryMiB: gpuMem, Data: string(b), Params: map[string]any{}, Owner: owner, Public: public}
|
||||
if cmd.Flags().Changed("epochs") {
|
||||
req.Params["epochs"] = epochs
|
||||
}
|
||||
if cmd.Flags().Changed("rank") {
|
||||
req.Params["rank"] = rank
|
||||
}
|
||||
if cmd.Flags().Changed("lr") {
|
||||
req.Params["lr"] = lr
|
||||
}
|
||||
if noGGUF {
|
||||
f := false
|
||||
req.GGUF = &f
|
||||
}
|
||||
body, _ := json.Marshal(req)
|
||||
var v adminapi.TrainJobView
|
||||
if err := client().patient(2*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs", json.RawMessage(body), &v); err != nil {
|
||||
return err
|
||||
}
|
||||
who := "every caller of " + v.Model
|
||||
if v.Owner != "" {
|
||||
who = v.Owner + " only"
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "queued %s: %s for %s on %s, %d pairs; it will be seen by %s\n",
|
||||
v.ID, v.Name, v.Model, v.Node, v.Pairs, who)
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "follow it with: llmgw ctl train show %s --follow\n", v.ID)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
f := start.Flags()
|
||||
f.StringVar(&model, "model", "", "the catalog model the adapter is for")
|
||||
f.StringVar(&base, "base-repo", "", "the weights to train on, as safetensors on Hugging Face (owner/name)")
|
||||
f.StringVar(&rev, "base-revision", "", "a revision of them (default: the newest)")
|
||||
f.StringVar(&node, "node", "", "the node that trains")
|
||||
f.IntVar(&gpuMem, "gpu-memory", 4096, "MiB of a card the job needs free")
|
||||
f.StringVar(&data, "data", "", "the pairs, NDJSON of {\"q\", \"a\"}")
|
||||
f.IntVar(&epochs, "epochs", 6, "passes over the pairs")
|
||||
f.IntVar(&rank, "rank", 16, "the adapter's rank")
|
||||
f.Float64Var(&lr, "lr", 2e-4, "the learning rate")
|
||||
f.BoolVar(&noGGUF, "no-gguf", false, "skip the llama.cpp GGUF, even for a model llama.cpp serves")
|
||||
f.StringVar(&owner, "owner", "", "who the adapter belongs to — org:, team:, project: (path or ID), user: or key: — and only they see it "+
|
||||
"(default: the knowledge base's owner, when the pairs come from one)")
|
||||
f.BoolVar(&public, "public", false, "every caller of the model sees it; refused for pairs from a knowledge base")
|
||||
for _, req := range []string{"model", "base-repo", "node", "data"} {
|
||||
_ = start.MarkFlagRequired(req)
|
||||
}
|
||||
c.AddCommand(start)
|
||||
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "list",
|
||||
Short: "Training jobs, newest first",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||
var jobs []adminapi.TrainJobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/train/jobs", &jobs); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(cmd.OutOrStdout()).Encode(jobs)
|
||||
}
|
||||
if len(jobs) == 0 {
|
||||
fmt.Fprintln(cmd.OutOrStdout(), "No training jobs (llmgw ctl train start).")
|
||||
return nil
|
||||
}
|
||||
w := tab()
|
||||
fmt.Fprintln(w, "JOB\tADAPTER\tMODEL\tOWNER\tNODE\tSTATE\tPROGRESS\tLOSS\tPEAK\tCREATED")
|
||||
for _, j := range jobs {
|
||||
fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", j.ID, j.Name, j.Model, or(j.Owner, "everyone"), j.Node, j.State,
|
||||
trainProgress(j), trainLoss(j), trainPeak(j), j.CreatedAt.Local().Format("01-02 15:04"))
|
||||
}
|
||||
return w.Flush()
|
||||
},
|
||||
})
|
||||
|
||||
var follow bool
|
||||
show := &cobra.Command{
|
||||
Use: "show JOB",
|
||||
Short: "How a training job goes, and what it made",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
out := cmd.OutOrStdout()
|
||||
last := ""
|
||||
for {
|
||||
var j adminapi.TrainJobView
|
||||
if err := client().get(cmd.Context(), "/api/v1/train/jobs/"+args[0], &j); err != nil {
|
||||
return err
|
||||
}
|
||||
if *asJSON {
|
||||
return json.NewEncoder(out).Encode(j)
|
||||
}
|
||||
line := fmt.Sprintf("%s %s %s loss %s peak %s %s", j.State, trainProgress(j), j.Node, trainLoss(j), trainPeak(j), j.Message)
|
||||
if !follow {
|
||||
fmt.Fprintf(out, "%s: adapter %s for %s, owned by %s, on %s (base %s), %d pairs\n", j.ID, j.Name, j.Model,
|
||||
or(j.Owner, "everyone"), j.Node, j.BaseRepo, j.Pairs)
|
||||
fmt.Fprintln(out, line)
|
||||
if len(j.Artifacts) > 0 {
|
||||
fmt.Fprintf(out, "made: %s (llmgw ctl train download %s --kind KIND)\n", strings.Join(j.Artifacts, ", "), j.ID)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if line != last {
|
||||
fmt.Fprintf(out, "%s %s\n", time.Now().Format("15:04:05"), line)
|
||||
last = line
|
||||
}
|
||||
if store.TrainOver(j.State) {
|
||||
if j.State != store.TrainDone {
|
||||
return fmt.Errorf("%s %s", j.ID, j.State)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
select {
|
||||
case <-cmd.Context().Done():
|
||||
return nil
|
||||
case <-time.After(3 * time.Second):
|
||||
}
|
||||
}
|
||||
},
|
||||
}
|
||||
show.Flags().BoolVarP(&follow, "follow", "f", false, "follow it until it ends")
|
||||
c.AddCommand(show)
|
||||
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "cancel JOB",
|
||||
Short: "Stop a training job",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := client().do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/cancel", nil, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s cancelled\n", args[0])
|
||||
return nil
|
||||
},
|
||||
})
|
||||
c.AddCommand(&cobra.Command{
|
||||
Use: "rm JOB",
|
||||
Short: "Remove an ended training job and what it made",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if err := client().do(cmd.Context(), "DELETE", "/api/v1/train/jobs/"+args[0], nil, nil); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s removed\n", args[0])
|
||||
return nil
|
||||
},
|
||||
})
|
||||
|
||||
var restart bool
|
||||
publish := &cobra.Command{
|
||||
Use: "publish JOB",
|
||||
Short: "Make what a finished job made its model's adapter, kept on the mirror",
|
||||
Long: "The GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves, are kept\n" +
|
||||
"on the mirror and added to the model's adapters; a request then asks for the\n" +
|
||||
"adapter by its name, as a model. llama.cpp loads adapters when a replica starts:\n" +
|
||||
"--restart replaces the model's replicas, one at a time, to load it.",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
var v adminapi.TrainPublishView
|
||||
if err := client().patient(15*time.Minute).do(cmd.Context(), "POST", "/api/v1/train/jobs/"+args[0]+"/publish", nil, &v); err != nil {
|
||||
return err
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
fmt.Fprintf(out, "%s is %s's adapter, from %s\n", v.Adapter, v.Model, v.Source)
|
||||
if v.Owner != "" {
|
||||
fmt.Fprintf(out, "only %s sees it; to anyone else it is a model that does not exist\n", v.Owner)
|
||||
}
|
||||
if restart && strings.Contains(v.Note, "restart") {
|
||||
body, _ := json.Marshal(adminapi.RestartRequest{Models: []string{v.Model}})
|
||||
var res adminapi.RestartResult
|
||||
if err := client().post(cmd.Context(), "/api/v1/restart", body, &res); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(out, "%d replica(s) of %s restarting to load it; then ask for model %s\n", len(res.Queued), v.Model, v.Adapter)
|
||||
return nil
|
||||
}
|
||||
fmt.Fprintln(out, v.Note)
|
||||
return nil
|
||||
},
|
||||
}
|
||||
publish.Flags().BoolVar(&restart, "restart", false, "restart the model's replicas to load it (llama.cpp)")
|
||||
c.AddCommand(publish)
|
||||
|
||||
var kind, outFile string
|
||||
download := &cobra.Command{
|
||||
Use: "download JOB",
|
||||
Short: "Save what a training job made: the PEFT adapter (peft) or the GGUF (gguf)",
|
||||
Args: cobra.ExactArgs(1),
|
||||
RunE: func(cmd *cobra.Command, args []string) error {
|
||||
if outFile == "" {
|
||||
outFile = args[0] + ".tar.gz"
|
||||
if kind == "gguf" {
|
||||
outFile = args[0] + ".gguf"
|
||||
}
|
||||
}
|
||||
f, err := os.Create(outFile)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer f.Close()
|
||||
cw := &countWriter{w: f}
|
||||
if err := client().patient(10*time.Minute).do(cmd.Context(), "GET",
|
||||
"/api/v1/train/jobs/"+args[0]+"/artifacts/"+kind, nil, io.Writer(cw)); err != nil {
|
||||
_ = os.Remove(outFile)
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(cmd.OutOrStdout(), "%s: %s\n", outFile, humanBytes(cw.n))
|
||||
return nil
|
||||
},
|
||||
}
|
||||
download.Flags().StringVar(&kind, "kind", "peft", "peft (a gzipped tar of the PEFT adapter) or gguf")
|
||||
download.Flags().StringVarP(&outFile, "out", "o", "", "the file to write")
|
||||
c.AddCommand(download)
|
||||
return c
|
||||
}
|
||||
|
||||
type countWriter struct {
|
||||
w io.Writer
|
||||
n int64
|
||||
}
|
||||
|
||||
func (c *countWriter) Write(p []byte) (int, error) {
|
||||
n, err := c.w.Write(p)
|
||||
c.n += int64(n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
func trainProgress(j adminapi.TrainJobView) string {
|
||||
if j.Steps == 0 {
|
||||
return "-"
|
||||
}
|
||||
return fmt.Sprintf("%d/%d", j.Step, j.Steps)
|
||||
}
|
||||
|
||||
func trainLoss(j adminapi.TrainJobView) string {
|
||||
if j.Loss == 0 {
|
||||
return "-"
|
||||
}
|
||||
return fmt.Sprintf("%.3f", j.Loss)
|
||||
}
|
||||
|
||||
func trainPeak(j adminapi.TrainJobView) string {
|
||||
if j.PeakMiB == 0 {
|
||||
return "-"
|
||||
}
|
||||
return fmt.Sprintf("%d MiB", j.PeakMiB)
|
||||
}
|
||||
@@ -0,0 +1,529 @@
|
||||
// Package comfy has images made by ComfyUI: a workflow filled in with what
|
||||
// was asked for, queued, waited on, and its pictures fetched.
|
||||
//
|
||||
// ComfyUI speaks no OpenAI. A workflow is a graph of nodes that is queued with
|
||||
// POST /prompt, is done when GET /history/{id} lists it, and leaves its
|
||||
// pictures to be read one by one from GET /view. What the graph is depends on
|
||||
// the model — FLUX schnell is not FLUX dev, and neither is SDXL — so the graph
|
||||
// is a template the catalog names, with placeholders where a request's
|
||||
// prompt, size and seed go.
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"embed"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/big"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed workflows/*.json
|
||||
var builtin embed.FS
|
||||
|
||||
// DefaultWorkflow is the template used when the catalog names none.
|
||||
const DefaultWorkflow = "flux-schnell"
|
||||
|
||||
// Builtins are the workflows that come with llmgw, by name.
|
||||
func Builtins() []string {
|
||||
entries, _ := builtin.ReadDir("workflows")
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Template is a workflow in ComfyUI's API format, with placeholders.
|
||||
//
|
||||
// A placeholder is a string value "{{name}}", or "{{name|default}}" for one a
|
||||
// request may leave out. One that is the whole value becomes the value
|
||||
// itself, a number where the node wants a number; one inside a longer string
|
||||
// is written into it. The names are those of Params.
|
||||
type Template struct {
|
||||
graph map[string]any
|
||||
}
|
||||
|
||||
// Load reads a template: a built-in's name, or the workflow's JSON itself.
|
||||
func Load(nameOrJSON string) (*Template, error) {
|
||||
s := strings.TrimSpace(nameOrJSON)
|
||||
if s == "" {
|
||||
s = DefaultWorkflow
|
||||
}
|
||||
raw := []byte(s)
|
||||
if !strings.HasPrefix(s, "{") {
|
||||
b, err := builtin.ReadFile("workflows/" + s + ".json")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("no built-in workflow %q (there are %s)", s, strings.Join(Builtins(), ", "))
|
||||
}
|
||||
raw = b
|
||||
}
|
||||
var g map[string]any
|
||||
if err := json.Unmarshal(raw, &g); err != nil {
|
||||
return nil, fmt.Errorf("the workflow is not JSON: %w", err)
|
||||
}
|
||||
if len(g) == 0 {
|
||||
return nil, errors.New("the workflow has no nodes")
|
||||
}
|
||||
for id, n := range g {
|
||||
node, ok := n.(map[string]any)
|
||||
if !ok || node["class_type"] == nil {
|
||||
// The UI's own save format has a "nodes" list instead, and is
|
||||
// not something /prompt accepts.
|
||||
return nil, fmt.Errorf("node %q has no class_type: the workflow must be saved in API format", id)
|
||||
}
|
||||
}
|
||||
return &Template{graph: g}, nil
|
||||
}
|
||||
|
||||
// Params are what a request decides. A zero number is one not given, which
|
||||
// leaves it to the template's default.
|
||||
type Params struct {
|
||||
Checkpoint string
|
||||
Prompt string
|
||||
NegativePrompt string
|
||||
Width, Height int
|
||||
N int
|
||||
Seed int64
|
||||
Steps int
|
||||
Guidance float64
|
||||
}
|
||||
|
||||
func (p Params) value(name string) (any, bool) {
|
||||
switch name {
|
||||
case "checkpoint":
|
||||
return p.Checkpoint, p.Checkpoint != ""
|
||||
case "prompt":
|
||||
return p.Prompt, true
|
||||
case "negative_prompt":
|
||||
return p.NegativePrompt, true
|
||||
case "width":
|
||||
return p.Width, p.Width > 0
|
||||
case "height":
|
||||
return p.Height, p.Height > 0
|
||||
case "n":
|
||||
return max(p.N, 1), true
|
||||
case "seed":
|
||||
return p.Seed, true
|
||||
case "steps":
|
||||
return p.Steps, p.Steps > 0
|
||||
case "guidance":
|
||||
return p.Guidance, p.Guidance > 0
|
||||
}
|
||||
return nil, false
|
||||
}
|
||||
|
||||
// Fill is the workflow for one request.
|
||||
func (t *Template) Fill(p Params) (map[string]any, error) {
|
||||
var missing []string
|
||||
var fill func(v any) any
|
||||
fill = func(v any) any {
|
||||
switch v := v.(type) {
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(v))
|
||||
for k, x := range v {
|
||||
out[k] = fill(x)
|
||||
}
|
||||
return out
|
||||
case []any:
|
||||
out := make([]any, len(v))
|
||||
for i, x := range v {
|
||||
out[i] = fill(x)
|
||||
}
|
||||
return out
|
||||
case string:
|
||||
return fillString(v, p, &missing)
|
||||
}
|
||||
return v
|
||||
}
|
||||
g := fill(t.graph).(map[string]any)
|
||||
if len(missing) > 0 {
|
||||
slices.Sort(missing)
|
||||
return nil, fmt.Errorf("the workflow wants %s, which the request did not give and the workflow has no default for",
|
||||
strings.Join(slices.Compact(missing), ", "))
|
||||
}
|
||||
return g, nil
|
||||
}
|
||||
|
||||
func fillString(s string, p Params, missing *[]string) any {
|
||||
if !strings.Contains(s, "{{") {
|
||||
return s
|
||||
}
|
||||
resolve := func(ph string) (any, bool) {
|
||||
name, def, hasDef := strings.Cut(ph, "|")
|
||||
name = strings.TrimSpace(name)
|
||||
if v, ok := p.value(name); ok {
|
||||
return v, true
|
||||
}
|
||||
if hasDef {
|
||||
def = strings.TrimSpace(def)
|
||||
if f, err := strconv.ParseFloat(def, 64); err == nil {
|
||||
return f, true
|
||||
}
|
||||
return def, true
|
||||
}
|
||||
*missing = append(*missing, name)
|
||||
return nil, false
|
||||
}
|
||||
// The whole value: what the node gets is the value, not its text.
|
||||
if strings.HasPrefix(s, "{{") && strings.HasSuffix(s, "}}") && strings.Count(s, "{{") == 1 {
|
||||
v, _ := resolve(s[2 : len(s)-2])
|
||||
return v
|
||||
}
|
||||
var b strings.Builder
|
||||
for {
|
||||
i := strings.Index(s, "{{")
|
||||
j := strings.Index(s, "}}")
|
||||
if i < 0 || j < i {
|
||||
b.WriteString(s)
|
||||
return b.String()
|
||||
}
|
||||
b.WriteString(s[:i])
|
||||
if v, ok := resolve(s[i+2 : j]); ok {
|
||||
fmt.Fprint(&b, v)
|
||||
}
|
||||
s = s[j+2:]
|
||||
}
|
||||
}
|
||||
|
||||
// RandomSeed is a seed for a request that named none. ComfyUI's are 64-bit,
|
||||
// but JSON numbers are read as doubles on the way: kept to 2^53.
|
||||
func RandomSeed() int64 {
|
||||
n, err := rand.Int(rand.Reader, big.NewInt(1<<53))
|
||||
if err != nil {
|
||||
return time.Now().UnixNano() & (1<<53 - 1)
|
||||
}
|
||||
return n.Int64()
|
||||
}
|
||||
|
||||
// Image is one picture made.
|
||||
type Image struct {
|
||||
Data []byte
|
||||
ContentType string
|
||||
}
|
||||
|
||||
// Error is a failure, said as what it means for the request.
|
||||
type Error struct {
|
||||
Msg string
|
||||
// Retryable is a failure of this server, which another one might not
|
||||
// have: it could not be reached, or it answered 5xx.
|
||||
Retryable bool
|
||||
}
|
||||
|
||||
func (e *Error) Error() string { return e.Msg }
|
||||
|
||||
// Client queues workflows and waits for them.
|
||||
type Client struct {
|
||||
HTTP *http.Client
|
||||
// Poll is how often a queued workflow is asked after.
|
||||
Poll time.Duration
|
||||
}
|
||||
|
||||
// Generate runs a workflow on the ComfyUI at base and returns its pictures,
|
||||
// in the order of the nodes that made them. A context that ends takes the
|
||||
// workflow out of the queue, or interrupts it if it is running, so a client
|
||||
// that went away does not leave the card working for nobody.
|
||||
func (c *Client) Generate(ctx context.Context, base, apiKey string, workflow map[string]any) ([]Image, error) {
|
||||
base = strings.TrimRight(base, "/")
|
||||
client := make([]byte, 8)
|
||||
_, _ = rand.Read(client)
|
||||
body, err := json.Marshal(map[string]any{"prompt": workflow, "client_id": "llmgw-" + hex.EncodeToString(client)})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var queued struct {
|
||||
PromptID string `json:"prompt_id"`
|
||||
Error json.RawMessage `json:"error"`
|
||||
NodeErrors json.RawMessage `json:"node_errors"`
|
||||
}
|
||||
status, err := c.do(ctx, http.MethodPost, base+"/prompt", apiKey, body, &queued)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if status != http.StatusOK || queued.PromptID == "" {
|
||||
// A workflow ComfyUI will not run: a checkpoint it does not have, a
|
||||
// node it does not know. The catalog's to fix, not the client's, and
|
||||
// no other replica set up the same way would take it either.
|
||||
return nil, &Error{Msg: "ComfyUI refused the workflow: " + refusal(queued.Error, queued.NodeErrors)}
|
||||
}
|
||||
id := queued.PromptID
|
||||
|
||||
poll := c.Poll
|
||||
if poll <= 0 {
|
||||
poll = 250 * time.Millisecond
|
||||
}
|
||||
t := time.NewTicker(poll)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
c.cancel(context.WithoutCancel(ctx), base, apiKey, id)
|
||||
return nil, ctx.Err()
|
||||
case <-t.C:
|
||||
}
|
||||
var hist map[string]historyEntry
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/history/"+url.PathEscape(id), apiKey, nil, &hist); err != nil {
|
||||
if ctx.Err() != nil {
|
||||
continue // said by the select
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
h, ok := hist[id]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if h.Status.StatusStr == "error" {
|
||||
return nil, &Error{Msg: "the workflow failed in ComfyUI: " + h.failure()}
|
||||
}
|
||||
if !h.Status.Completed {
|
||||
continue
|
||||
}
|
||||
return c.fetch(ctx, base, apiKey, h)
|
||||
}
|
||||
}
|
||||
|
||||
type historyEntry struct {
|
||||
Status struct {
|
||||
StatusStr string `json:"status_str"`
|
||||
Completed bool `json:"completed"`
|
||||
Messages [][]json.RawMessage `json:"messages"`
|
||||
} `json:"status"`
|
||||
Outputs map[string]struct {
|
||||
Images []struct {
|
||||
Filename string `json:"filename"`
|
||||
Subfolder string `json:"subfolder"`
|
||||
Type string `json:"type"`
|
||||
} `json:"images"`
|
||||
} `json:"outputs"`
|
||||
}
|
||||
|
||||
// failure is what ComfyUI said went wrong, from its execution_error.
|
||||
func (h historyEntry) failure() string {
|
||||
for _, m := range h.Status.Messages {
|
||||
if len(m) < 2 || string(m[0]) != `"execution_error"` {
|
||||
continue
|
||||
}
|
||||
var e struct {
|
||||
NodeType string `json:"node_type"`
|
||||
Message string `json:"exception_message"`
|
||||
}
|
||||
if json.Unmarshal(m[1], &e) == nil && e.Message != "" {
|
||||
return strings.TrimSpace(e.NodeType + ": " + strings.TrimSpace(e.Message))
|
||||
}
|
||||
}
|
||||
return "no reason given"
|
||||
}
|
||||
|
||||
func (c *Client) fetch(ctx context.Context, base, apiKey string, h historyEntry) ([]Image, error) {
|
||||
nodes := make([]string, 0, len(h.Outputs))
|
||||
for id := range h.Outputs {
|
||||
nodes = append(nodes, id)
|
||||
}
|
||||
// Numerically where they are numbers, which is how ComfyUI names them.
|
||||
slices.SortFunc(nodes, func(a, b string) int {
|
||||
x, ea := strconv.Atoi(a)
|
||||
y, eb := strconv.Atoi(b)
|
||||
if ea == nil && eb == nil {
|
||||
return x - y
|
||||
}
|
||||
return strings.Compare(a, b)
|
||||
})
|
||||
var out []Image
|
||||
for _, n := range nodes {
|
||||
for _, im := range h.Outputs[n].Images {
|
||||
q := url.Values{"filename": {im.Filename}, "subfolder": {im.Subfolder}, "type": {im.Type}}
|
||||
img, err := c.get(ctx, base+"/view?"+q.Encode(), apiKey)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, img)
|
||||
}
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, &Error{Msg: "the workflow finished without a picture: it has no SaveImage or PreviewImage node"}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// cancel takes a workflow out of the queue, or stops it if it is the one
|
||||
// running. Best effort: the server may be gone, which also stops it.
|
||||
func (c *Client) cancel(ctx context.Context, base, apiKey, id string) {
|
||||
ctx, done := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer done()
|
||||
body, _ := json.Marshal(map[string]any{"delete": []string{id}})
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/queue", apiKey, body, nil)
|
||||
var q struct {
|
||||
Running [][]json.RawMessage `json:"queue_running"`
|
||||
}
|
||||
if _, err := c.do(ctx, http.MethodGet, base+"/queue", apiKey, nil, &q); err != nil {
|
||||
return
|
||||
}
|
||||
for _, r := range q.Running {
|
||||
if len(r) > 1 && string(r[1]) == strconv.Quote(id) {
|
||||
_, _ = c.do(ctx, http.MethodPost, base+"/interrupt", apiKey, []byte("{}"), nil)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Client) client() *http.Client {
|
||||
if c.HTTP != nil {
|
||||
return c.HTTP
|
||||
}
|
||||
return http.DefaultClient
|
||||
}
|
||||
|
||||
func (c *Client) request(ctx context.Context, method, u, apiKey string, body []byte) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, method, u, bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if body != nil {
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
if apiKey != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+apiKey)
|
||||
}
|
||||
resp, err := c.client().Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return nil, &Error{Msg: "ComfyUI could not be reached: " + err.Error(), Retryable: true}
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
// do sends a request and reads a JSON answer into out. A 5xx is an error;
|
||||
// any other status is returned with whatever body could be read.
|
||||
func (c *Client) do(ctx context.Context, method, u, apiKey string, body []byte, out any) (int, error) {
|
||||
resp, err := c.request(ctx, method, u, apiKey, body)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 16<<20))
|
||||
if err != nil {
|
||||
return 0, &Error{Msg: "reading ComfyUI's answer: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode >= 500 {
|
||||
return resp.StatusCode, &Error{Msg: fmt.Sprintf("ComfyUI answered %d: %s", resp.StatusCode, snippet(b)), Retryable: true}
|
||||
}
|
||||
if out != nil && len(b) > 0 {
|
||||
if err := json.Unmarshal(b, out); err != nil && resp.StatusCode == http.StatusOK {
|
||||
return resp.StatusCode, &Error{Msg: "ComfyUI's answer is not JSON: " + snippet(b)}
|
||||
}
|
||||
}
|
||||
return resp.StatusCode, nil
|
||||
}
|
||||
|
||||
func (c *Client) get(ctx context.Context, u, apiKey string) (Image, error) {
|
||||
resp, err := c.request(ctx, http.MethodGet, u, apiKey, nil)
|
||||
if err != nil {
|
||||
return Image{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(io.LimitReader(resp.Body, 64<<20))
|
||||
if err != nil {
|
||||
return Image{}, &Error{Msg: "reading a picture from ComfyUI: " + err.Error(), Retryable: true}
|
||||
}
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return Image{}, &Error{Msg: fmt.Sprintf("ComfyUI answered %d for a picture it made", resp.StatusCode)}
|
||||
}
|
||||
ct := resp.Header.Get("Content-Type")
|
||||
if ct == "" || ct == "application/octet-stream" {
|
||||
ct = http.DetectContentType(b)
|
||||
}
|
||||
if ct == "image/png" {
|
||||
b = withoutPNGText(b)
|
||||
}
|
||||
return Image{Data: b, ContentType: ct}, nil
|
||||
}
|
||||
|
||||
// withoutPNGText is a PNG without its text chunks. ComfyUI writes the whole
|
||||
// workflow into every picture it saves — the prompt, and the checkpoint and
|
||||
// nodes behind it — and what a picture says about how it was made is the
|
||||
// operator's to publish, not something to hand every client. A file that is
|
||||
// not a well-formed PNG is returned as it came.
|
||||
func withoutPNGText(b []byte) []byte {
|
||||
const sig = "\x89PNG\r\n\x1a\n"
|
||||
if len(b) < len(sig) || string(b[:len(sig)]) != sig {
|
||||
return b
|
||||
}
|
||||
out := make([]byte, 0, len(b))
|
||||
out = append(out, b[:len(sig)]...)
|
||||
for i := len(sig); i < len(b); {
|
||||
if i+12 > len(b) {
|
||||
return b
|
||||
}
|
||||
n := int(binary.BigEndian.Uint32(b[i:]))
|
||||
end := i + 12 + n
|
||||
if n < 0 || end > len(b) {
|
||||
return b
|
||||
}
|
||||
switch string(b[i+4 : i+8]) {
|
||||
case "tEXt", "iTXt", "zTXt":
|
||||
default:
|
||||
out = append(out, b[i:end]...)
|
||||
}
|
||||
i = end
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// refusal is what /prompt said when it would not queue a workflow.
|
||||
func refusal(e, nodes json.RawMessage) string {
|
||||
var top struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
}
|
||||
msg := snippet(e)
|
||||
if json.Unmarshal(e, &top) == nil && top.Message != "" {
|
||||
msg = top.Message
|
||||
if top.Details != "" {
|
||||
msg += ": " + top.Details
|
||||
}
|
||||
}
|
||||
var perNode map[string]struct {
|
||||
ClassType string `json:"class_type"`
|
||||
Errors []struct {
|
||||
Message string `json:"message"`
|
||||
Details string `json:"details"`
|
||||
} `json:"errors"`
|
||||
}
|
||||
if json.Unmarshal(nodes, &perNode) == nil {
|
||||
var parts []string
|
||||
for _, n := range perNode {
|
||||
for _, er := range n.Errors {
|
||||
parts = append(parts, strings.TrimSpace(n.ClassType+": "+er.Message+" "+er.Details))
|
||||
}
|
||||
}
|
||||
slices.Sort(parts)
|
||||
if len(parts) > 0 {
|
||||
msg += " (" + strings.Join(parts, "; ") + ")"
|
||||
}
|
||||
}
|
||||
if msg == "" {
|
||||
msg = "no reason given"
|
||||
}
|
||||
return msg
|
||||
}
|
||||
|
||||
func snippet(b []byte) string {
|
||||
s := strings.TrimSpace(string(b))
|
||||
if len(s) > 300 {
|
||||
s = s[:300] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,227 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheBuiltInWorkflowsFillIn(t *testing.T) {
|
||||
for _, name := range Builtins() {
|
||||
tpl, err := Load(name)
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "flux.safetensors", Prompt: `a "red" fox`, Width: 768, Height: 512, N: 2, Seed: 42})
|
||||
if err != nil {
|
||||
t.Fatalf("%s: %v", name, err)
|
||||
}
|
||||
b, _ := json.Marshal(g)
|
||||
s := string(b)
|
||||
for _, want := range []string{`"ckpt_name":"flux.safetensors"`, `"text":"a \"red\" fox"`, `"width":768`,
|
||||
`"height":512`, `"batch_size":2`, `"seed":42`} {
|
||||
if !strings.Contains(s, want) {
|
||||
t.Errorf("%s: %s not in %s", name, want, s)
|
||||
}
|
||||
}
|
||||
if strings.Contains(s, "{{") {
|
||||
t.Errorf("%s: a placeholder was left: %s", name, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStepCountGivenReplacesTheDefault(t *testing.T) {
|
||||
tpl, _ := Load("flux-schnell")
|
||||
g, _ := tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 4.0 {
|
||||
t.Errorf("default steps = %v, want 4", steps)
|
||||
}
|
||||
g, _ = tpl.Fill(Params{Checkpoint: "c", Prompt: "p", Width: 512, Height: 512, Steps: 8})
|
||||
if steps := g["5"].(map[string]any)["inputs"].(map[string]any)["steps"]; steps != 8 {
|
||||
t.Errorf("steps = %v, want 8", steps)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPlaceholderWithoutAValueOrADefaultIsSaid(t *testing.T) {
|
||||
tpl, err := Load(`{"1":{"class_type":"X","inputs":{"a":"{{checkpoint}}","b":"size {{width}}x{{height}}"}}}`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tpl.Fill(Params{Width: 5}); err == nil || !strings.Contains(err.Error(), "checkpoint, height") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
g, err := tpl.Fill(Params{Checkpoint: "c", Width: 5, Height: 6})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b := g["1"].(map[string]any)["inputs"].(map[string]any)["b"]; b != "size 5x6" {
|
||||
t.Errorf("b = %v", b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowInTheUIsFormatIsRefused(t *testing.T) {
|
||||
if _, err := Load(`{"nodes":[{"id":1}],"links":[]}`); err == nil || !strings.Contains(err.Error(), "API format") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
if _, err := Load("sdxl-turbo"); err == nil {
|
||||
t.Fatal("an unknown built-in was loaded")
|
||||
}
|
||||
}
|
||||
|
||||
// fakeComfy is just enough of ComfyUI: /prompt queues, /history answers
|
||||
// after a few asks, /view serves a picture.
|
||||
type fakeComfy struct {
|
||||
mu sync.Mutex
|
||||
asked int
|
||||
fail string
|
||||
refuse bool
|
||||
deleted []string
|
||||
queued map[string]any
|
||||
readyAt int
|
||||
srv *httptest.Server
|
||||
interrupted bool
|
||||
}
|
||||
|
||||
func newFakeComfy(t *testing.T) *fakeComfy {
|
||||
f := &fakeComfy{readyAt: 2}
|
||||
f.srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
switch {
|
||||
case r.Method == http.MethodPost && r.URL.Path == "/prompt":
|
||||
if f.refuse {
|
||||
w.WriteHeader(http.StatusBadRequest)
|
||||
fmt.Fprint(w, `{"error":{"message":"Prompt outputs failed validation","details":""},"node_errors":{"1":{"class_type":"CheckpointLoaderSimple","errors":[{"message":"Value not in list","details":"ckpt_name: 'x' not in []"}]}}}`)
|
||||
return
|
||||
}
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.queued = b.Prompt
|
||||
fmt.Fprint(w, `{"prompt_id":"p1","number":1,"node_errors":{}}`)
|
||||
case r.URL.Path == "/history/p1":
|
||||
f.asked++
|
||||
if f.asked < f.readyAt {
|
||||
fmt.Fprint(w, `{}`)
|
||||
return
|
||||
}
|
||||
if f.fail != "" {
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"error","completed":false,"messages":[["execution_start",{}],["execution_error",{"node_type":"KSampler","exception_message":%q}]]},"outputs":{}}}`, f.fail)
|
||||
return
|
||||
}
|
||||
fmt.Fprint(w, `{"p1":{"status":{"status_str":"success","completed":true,"messages":[]},"outputs":{"7":{"images":[{"filename":"a.png","subfolder":"","type":"temp"},{"filename":"b.png","subfolder":"","type":"temp"}]}}}}`)
|
||||
case r.URL.Path == "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "PNG:"+r.URL.Query().Get("filename"))
|
||||
case r.URL.Path == "/queue" && r.Method == http.MethodPost:
|
||||
var b struct {
|
||||
Delete []string `json:"delete"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
f.deleted = append(f.deleted, b.Delete...)
|
||||
case r.URL.Path == "/queue":
|
||||
fmt.Fprint(w, `{"queue_running":[[1,"p1",{},{},[]]],"queue_pending":[]}`)
|
||||
case r.URL.Path == "/interrupt":
|
||||
f.interrupted = true
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(f.srv.Close)
|
||||
return f
|
||||
}
|
||||
|
||||
func TestGenerateWaitsForTheWorkflowAndFetchesItsPictures(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
c := &Client{Poll: time.Millisecond}
|
||||
images, err := c.Generate(t.Context(), f.srv.URL, "", map[string]any{"1": map[string]any{"class_type": "X"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(images) != 2 || string(images[0].Data) != "PNG:a.png" || images[1].ContentType != "image/png" {
|
||||
t.Fatalf("images = %+v", images)
|
||||
}
|
||||
if f.queued["1"] == nil {
|
||||
t.Error("the workflow was not what was queued")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowThatFailsSaysWhy(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.fail = "CUDA out of memory"
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if err == nil || !strings.Contains(err.Error(), "KSampler: CUDA out of memory") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAWorkflowComfyUIRefusesIsNotRetried(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.refuse = true
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
ce, ok := err.(*Error)
|
||||
if !ok || ce.Retryable || !strings.Contains(ce.Msg, "Value not in list") {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIThatCannotBeReachedIsRetryable(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.srv.Close()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(t.Context(), f.srv.URL, "", map[string]any{})
|
||||
if ce, ok := err.(*Error); !ok || !ce.Retryable {
|
||||
t.Fatalf("err = %#v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAClientThatLeavesTakesItsWorkflowOffTheCard(t *testing.T) {
|
||||
f := newFakeComfy(t)
|
||||
f.readyAt = 1 << 30
|
||||
ctx, cancel := context.WithTimeout(t.Context(), 30*time.Millisecond)
|
||||
defer cancel()
|
||||
_, err := (&Client{Poll: time.Millisecond}).Generate(ctx, f.srv.URL, "", map[string]any{})
|
||||
if err == nil {
|
||||
t.Fatal("no error")
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if len(f.deleted) != 1 || f.deleted[0] != "p1" || !f.interrupted {
|
||||
t.Errorf("deleted %v, interrupted %v", f.deleted, f.interrupted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPictureLosesTheWorkflowComfyUIWroteIntoIt(t *testing.T) {
|
||||
chunk := func(kind, data string) []byte {
|
||||
b := make([]byte, 0, 12+len(data))
|
||||
b = binary.BigEndian.AppendUint32(b, uint32(len(data)))
|
||||
b = append(b, kind...)
|
||||
b = append(b, data...)
|
||||
return append(b, 0, 0, 0, 0) // a CRC nobody here checks
|
||||
}
|
||||
png := []byte("\x89PNG\r\n\x1a\n")
|
||||
png = append(png, chunk("IHDR", "0123456789abc")...)
|
||||
png = append(png, chunk("tEXt", "prompt\x00{\"1\":...}")...)
|
||||
png = append(png, chunk("IDAT", "pixels")...)
|
||||
png = append(png, chunk("iTXt", "workflow\x00...")...)
|
||||
png = append(png, chunk("IEND", "")...)
|
||||
got := string(withoutPNGText(png))
|
||||
if strings.Contains(got, "prompt") || strings.Contains(got, "workflow") {
|
||||
t.Errorf("text left in: %q", got)
|
||||
}
|
||||
for _, kind := range []string{"IHDR", "IDAT", "pixels", "IEND"} {
|
||||
if !strings.Contains(got, kind) {
|
||||
t.Errorf("%s gone: %q", kind, got)
|
||||
}
|
||||
}
|
||||
if broken := png[:30]; string(withoutPNGText(broken)) != string(broken) {
|
||||
t.Error("a truncated PNG was changed")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,334 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/subtle"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httputil"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"os/signal"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The gate is how a node runs ComfyUI as one of llmgw's engines.
|
||||
//
|
||||
// ComfyUI asks nobody for a key: whoever reaches its port can queue work,
|
||||
// upload files and install nodes. Every other engine here demands the key
|
||||
// the control plane made for its replica, so ComfyUI is put behind a gate
|
||||
// that does: ComfyUI listens on the loopback only, and the gate on the
|
||||
// replica's port, letting through only the requests the router makes. The
|
||||
// gate also answers what the agent asks every engine — /health once
|
||||
// ComfyUI is up, and /v1/models naming the model — and lays the weights out
|
||||
// where ComfyUI looks for them.
|
||||
//
|
||||
// ComfyUI runs as the gate's child, in its process group, so stopping the
|
||||
// replica stops both, the card memory ComfyUI holds is counted as the
|
||||
// replica's, and either one exiting ends the other.
|
||||
|
||||
// GateConfig is one replica's gate.
|
||||
type GateConfig struct {
|
||||
// Listen is the replica's address, the one the router is given.
|
||||
Listen string
|
||||
// Model is the name the agent asks /v1/models for.
|
||||
Model string
|
||||
// Key is what every request must carry as a bearer token; empty lets
|
||||
// every request through, as an engine started without one does.
|
||||
Key string
|
||||
// Weights is the model as the agent fetched it: a file, or a directory
|
||||
// of files.
|
||||
Weights string
|
||||
// State is the replica's own directory: ComfyUI's base directory, with
|
||||
// its output, temp and user directories, and the models it is shown.
|
||||
State string
|
||||
// Python runs ComfyUI's main.py, found in Dir.
|
||||
Python, Dir string
|
||||
// Args are passed on to main.py, after the gate's own.
|
||||
Args []string
|
||||
// Output is where ComfyUI's output goes.
|
||||
Output io.Writer
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
// The requests the router makes of ComfyUI: queueing, waiting, fetching and
|
||||
// cancelling. Nothing that uploads, installs or changes ComfyUI.
|
||||
var gatePaths = []string{"/prompt", "/history/", "/view", "/queue", "/interrupt", "/system_stats", "/object_info"}
|
||||
|
||||
// modelFolders are the kinds of file ComfyUI's loaders look for. Each is
|
||||
// shown the same directory: a workflow names its files, so a file being
|
||||
// listed where it is not wanted costs nothing.
|
||||
var modelFolders = []string{"checkpoints", "diffusion_models", "text_encoders", "vae", "loras",
|
||||
"clip_vision", "controlnet", "upscale_models", "embeddings", "style_models"}
|
||||
|
||||
// RunGate lays out the replica, starts ComfyUI and serves the gate until
|
||||
// ctx ends or ComfyUI exits.
|
||||
func RunGate(ctx context.Context, cfg GateConfig) error {
|
||||
log := cfg.Log
|
||||
if log == nil {
|
||||
log = slog.Default()
|
||||
}
|
||||
state, err := filepath.Abs(cfg.State)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// What ComfyUI expects to find in its base directory, and dies
|
||||
// without: custom_nodes is listed before anything else is done.
|
||||
for _, d := range []string{"custom_nodes", "input", "output", "temp", "user"} {
|
||||
if err := os.MkdirAll(filepath.Join(state, d), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
models := filepath.Join(state, "models")
|
||||
if err := LinkWeights(cfg.Weights, models); err != nil {
|
||||
return err
|
||||
}
|
||||
extra := filepath.Join(state, "extra_model_paths.yaml")
|
||||
if err := os.WriteFile(extra, []byte(extraModelPaths(models)), 0o644); err != nil { //nolint:gosec // not secret
|
||||
return err
|
||||
}
|
||||
|
||||
inner, err := freePort()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
args := append([]string{filepath.Join(cfg.Dir, "main.py"),
|
||||
"--listen", "127.0.0.1", "--port", strconv.Itoa(inner),
|
||||
"--base-directory", state, "--extra-model-paths-config", extra,
|
||||
"--disable-auto-launch"}, cfg.Args...)
|
||||
child := exec.Command(cfg.Python, args...) //nolint:gosec // the agent's own configuration
|
||||
child.Dir = cfg.Dir
|
||||
child.Env = os.Environ()
|
||||
out := cfg.Output
|
||||
if out == nil {
|
||||
out = os.Stdout
|
||||
}
|
||||
// ComfyUI writes to a pipe of the gate's, never to what the gate was
|
||||
// given. That is the agent's, and an agent that restarts and adopts the
|
||||
// replica leaves it with no reader: ComfyUI's next progress bar would
|
||||
// fail with a broken pipe, and take the workflow it belongs to with it.
|
||||
// The gate reads the pipe for as long as ComfyUI writes, and passes on
|
||||
// what it can.
|
||||
signal.Ignore(syscall.SIGPIPE)
|
||||
pr, err := child.StdoutPipe()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
child.Stderr = child.Stdout
|
||||
if err := child.Start(); err != nil {
|
||||
return fmt.Errorf("starting ComfyUI: %w", err)
|
||||
}
|
||||
relayed := make(chan struct{})
|
||||
go func() { relay(pr, out); close(relayed) }()
|
||||
exited := make(chan error, 1)
|
||||
go func() {
|
||||
// Wait closes the pipe; what is in it is read first.
|
||||
<-relayed
|
||||
exited <- child.Wait()
|
||||
}()
|
||||
|
||||
upstream := &url.URL{Scheme: "http", Host: net.JoinHostPort("127.0.0.1", strconv.Itoa(inner))}
|
||||
srv := &http.Server{
|
||||
Addr: cfg.Listen,
|
||||
Handler: NewGate(upstream, cfg.Key, cfg.Model),
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
}
|
||||
served := make(chan error, 1)
|
||||
go func() { served <- srv.ListenAndServe() }()
|
||||
log.Info("ComfyUI gate up", "listen", cfg.Listen, "comfyui", upstream.Host, "weights", cfg.Weights)
|
||||
|
||||
stop := func() {
|
||||
_ = child.Process.Signal(syscall.SIGTERM)
|
||||
select {
|
||||
case <-exited:
|
||||
case <-time.After(20 * time.Second):
|
||||
_ = child.Process.Kill()
|
||||
<-exited
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
shutdown(srv)
|
||||
stop()
|
||||
return nil
|
||||
case err := <-exited:
|
||||
shutdown(srv)
|
||||
if err == nil {
|
||||
err = errors.New("exited")
|
||||
}
|
||||
return fmt.Errorf("ComfyUI %w", err)
|
||||
case err := <-served:
|
||||
stop()
|
||||
return fmt.Errorf("the gate stopped serving: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
func shutdown(srv *http.Server) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
_ = srv.Shutdown(ctx)
|
||||
}
|
||||
|
||||
// NewGate is the gate's handler, in front of the ComfyUI at upstream.
|
||||
func NewGate(upstream *url.URL, key, model string) http.Handler {
|
||||
proxy := httputil.NewSingleHostReverseProxy(upstream)
|
||||
base := proxy.Director
|
||||
proxy.Director = func(r *http.Request) {
|
||||
base(r)
|
||||
r.Header.Del("Authorization")
|
||||
// ComfyUI refuses a request whose Origin is not its own host; the
|
||||
// router sends none, and nothing else gets this far.
|
||||
r.Header.Del("Origin")
|
||||
r.Host = upstream.Host
|
||||
}
|
||||
client := &http.Client{Timeout: 3 * time.Second}
|
||||
up := func(ctx context.Context) error {
|
||||
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, upstream.String()+"/system_stats", nil)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("ComfyUI answered %s", resp.Status)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if key != "" {
|
||||
got, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
|
||||
if !ok || subtle.ConstantTimeCompare([]byte(got), []byte(key)) != 1 {
|
||||
http.Error(w, "this engine wants its key", http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
}
|
||||
switch r.URL.Path {
|
||||
case "/health":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return
|
||||
case "/v1/models":
|
||||
if err := up(r.Context()); err != nil {
|
||||
http.Error(w, "ComfyUI is not up yet: "+err.Error(), http.StatusServiceUnavailable)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
fmt.Fprintf(w, `{"object":"list","data":[{"id":%q,"object":"model","owned_by":"comfyui"}]}`, model)
|
||||
return
|
||||
}
|
||||
for _, p := range gatePaths {
|
||||
if r.URL.Path == p || strings.HasSuffix(p, "/") && strings.HasPrefix(r.URL.Path, p) {
|
||||
proxy.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
}
|
||||
http.NotFound(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
// cachedName is a file the agent fetched by itself: its cache names it by
|
||||
// the source's hash, then the name it had.
|
||||
var cachedName = regexp.MustCompile(`^[0-9a-f]{16}-`)
|
||||
|
||||
// FileName is what a weights file is called where ComfyUI sees it: its own
|
||||
// name, without the prefix the agent's cache gives it.
|
||||
func FileName(path string) string {
|
||||
return cachedName.ReplaceAllString(filepath.Base(path), "")
|
||||
}
|
||||
|
||||
// LinkWeights shows ComfyUI the weights: a link in dir to the file, or to
|
||||
// every file of the directory. Links made before are taken away first, so
|
||||
// a replica started again on other weights sees only those.
|
||||
func LinkWeights(weights, dir string) error {
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
old, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, e := range old {
|
||||
if e.Type()&os.ModeSymlink != 0 {
|
||||
_ = os.Remove(filepath.Join(dir, e.Name()))
|
||||
}
|
||||
}
|
||||
weights, err = filepath.Abs(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
info, err := os.Stat(weights)
|
||||
if err != nil {
|
||||
return fmt.Errorf("the weights: %w", err)
|
||||
}
|
||||
files := []string{weights}
|
||||
if info.IsDir() {
|
||||
entries, err := os.ReadDir(weights)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
files = files[:0]
|
||||
for _, e := range entries {
|
||||
if e.Type().IsRegular() && !strings.HasPrefix(e.Name(), ".") {
|
||||
files = append(files, filepath.Join(weights, e.Name()))
|
||||
}
|
||||
}
|
||||
if len(files) == 0 {
|
||||
return fmt.Errorf("the weights directory %s has no files", weights)
|
||||
}
|
||||
}
|
||||
for _, f := range files {
|
||||
if err := os.Symlink(f, filepath.Join(dir, FileName(f))); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func extraModelPaths(models string) string {
|
||||
var b strings.Builder
|
||||
fmt.Fprintf(&b, "llmgw:\n base_path: %q\n is_default: true\n", models)
|
||||
for _, f := range modelFolders {
|
||||
fmt.Fprintf(&b, " %s: .\n", f)
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func freePort() (int, error) {
|
||||
l, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer l.Close()
|
||||
return l.Addr().(*net.TCPAddr).Port, nil
|
||||
}
|
||||
|
||||
// relay copies r to w for as long as r has anything, and goes on reading
|
||||
// after w fails: a writer that stops reading must not stop the one writing.
|
||||
func relay(r io.Reader, w io.Writer) {
|
||||
buf := make([]byte, 32<<10)
|
||||
broken := false
|
||||
for {
|
||||
n, err := r.Read(buf)
|
||||
if n > 0 && !broken {
|
||||
if _, werr := w.Write(buf[:n]); werr != nil {
|
||||
broken = true
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package comfy
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTheGateWantsTheKeyAndLetsThroughOnlyTheRoutersRequests(t *testing.T) {
|
||||
var got []string
|
||||
comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
got = append(got, r.Method+" "+r.URL.Path+" auth="+r.Header.Get("Authorization"))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
}))
|
||||
defer comfy.Close()
|
||||
u, _ := url.Parse(comfy.URL)
|
||||
gate := httptest.NewServer(NewGate(u, "k", "flux"))
|
||||
defer gate.Close()
|
||||
|
||||
do := func(method, path, key string) (int, string) {
|
||||
req, _ := http.NewRequest(method, gate.URL+path, strings.NewReader("{}"))
|
||||
if key != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+key)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return resp.StatusCode, string(b)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", ""); code != http.StatusUnauthorized {
|
||||
t.Errorf("no key: %d", code)
|
||||
}
|
||||
if code, _ := do("POST", "/prompt", "wrong"); code != http.StatusUnauthorized {
|
||||
t.Errorf("wrong key: %d", code)
|
||||
}
|
||||
for _, p := range []string{"/upload/image", "/api/manager/queue/install", "/userdata/x", "/internal/logs"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusNotFound {
|
||||
t.Errorf("%s: %d, want it refused", p, code)
|
||||
}
|
||||
}
|
||||
for _, p := range []string{"/prompt", "/history/p1", "/view?filename=a.png", "/queue", "/interrupt"} {
|
||||
if code, _ := do("POST", p, "k"); code != http.StatusOK {
|
||||
t.Errorf("%s: %d", p, code)
|
||||
}
|
||||
}
|
||||
if code, body := do("GET", "/v1/models", "k"); code != 200 || !strings.Contains(body, `"id":"flux"`) {
|
||||
t.Errorf("/v1/models: %d %s", code, body)
|
||||
}
|
||||
if code, _ := do("GET", "/health", "k"); code != 200 {
|
||||
t.Errorf("/health: %d", code)
|
||||
}
|
||||
for _, g := range got {
|
||||
if strings.Contains(g, "auth=Bearer") {
|
||||
t.Errorf("the key reached ComfyUI: %s", g)
|
||||
}
|
||||
}
|
||||
comfy.Close()
|
||||
if code, _ := do("GET", "/health", "k"); code != http.StatusServiceUnavailable {
|
||||
t.Errorf("/health with ComfyUI down: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheWeightsAreShownByTheirOwnNames(t *testing.T) {
|
||||
cache := t.TempDir()
|
||||
one := filepath.Join(cache, "0123456789abcdef-flux1-schnell-fp8.safetensors")
|
||||
_ = os.WriteFile(one, []byte("w"), 0o644)
|
||||
models := filepath.Join(t.TempDir(), "models")
|
||||
if err := LinkWeights(one, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if target, err := os.Readlink(filepath.Join(models, "flux1-schnell-fp8.safetensors")); err != nil || target != one {
|
||||
t.Fatalf("%q %v", target, err)
|
||||
}
|
||||
|
||||
set := t.TempDir()
|
||||
for _, n := range []string{"flux1-dev.safetensors", "ae.safetensors", "t5xxl_fp8.safetensors", ".complete"} {
|
||||
_ = os.WriteFile(filepath.Join(set, n), []byte("w"), 0o644)
|
||||
}
|
||||
if err := LinkWeights(set, models); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
entries, _ := os.ReadDir(models)
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
if strings.Join(names, ",") != "ae.safetensors,flux1-dev.safetensors,t5xxl_fp8.safetensors" {
|
||||
t.Errorf("shown %v: the old weights' link should be gone and the marker not shown", names)
|
||||
}
|
||||
if !strings.Contains(extraModelPaths(models), "checkpoints: .") {
|
||||
t.Error(extraModelPaths(models))
|
||||
}
|
||||
}
|
||||
|
||||
// TestMain lets the test binary stand in for ComfyUI: started by the gate
|
||||
// as `python main.py --listen … --port …`, it answers /system_stats and
|
||||
// says what it was given.
|
||||
func TestMain(m *testing.M) {
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI") == "1" {
|
||||
fakeComfyUI()
|
||||
return
|
||||
}
|
||||
os.Exit(m.Run())
|
||||
}
|
||||
|
||||
func fakeComfyUI() {
|
||||
var port, base string
|
||||
for i, a := range os.Args {
|
||||
switch a {
|
||||
case "--port":
|
||||
port = os.Args[i+1]
|
||||
case "--base-directory":
|
||||
base = os.Args[i+1]
|
||||
}
|
||||
}
|
||||
_ = os.WriteFile(filepath.Join(base, "args"), []byte(strings.Join(os.Args[1:], " ")+"\nkey="+os.Getenv("LLMGW_ENGINE_KEY")), 0o644)
|
||||
http.HandleFunc("/system_stats", func(w http.ResponseWriter, _ *http.Request) {
|
||||
// A progress bar's worth of output with every answer: a write that
|
||||
// failed would end this process, as it fails ComfyUI's workflow.
|
||||
if _, err := os.Stdout.Write([]byte(strings.Repeat("progress ", 20000) + "\n")); err != nil {
|
||||
os.Exit(4)
|
||||
}
|
||||
_, _ = io.WriteString(w, "{}")
|
||||
})
|
||||
http.HandleFunc("/quit", func(w http.ResponseWriter, _ *http.Request) { os.Exit(3) })
|
||||
if os.Getenv("LLMGW_FAKE_COMFYUI_EXIT") == "1" {
|
||||
time.AfterFunc(200*time.Millisecond, func() { os.Exit(3) })
|
||||
}
|
||||
_ = http.ListenAndServe("127.0.0.1:"+port, nil)
|
||||
}
|
||||
|
||||
func TestTheGateRunsComfyUIAndEndsWithIt(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "flux.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
state := t.TempDir()
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Model: "flux", Weights: weights, State: state,
|
||||
Python: os.Args[0], Dir: t.TempDir(), Args: []string{"--disable-smart-memory"}, Output: io.Discard})
|
||||
}()
|
||||
get := func(path string) int {
|
||||
resp, err := http.Get("http://" + listen + path)
|
||||
if err != nil {
|
||||
return 0
|
||||
}
|
||||
resp.Body.Close()
|
||||
return resp.StatusCode
|
||||
}
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for get("/health") != http.StatusOK {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the gate never said ComfyUI was up")
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(state, "args"))
|
||||
for _, want := range []string{"main.py --listen 127.0.0.1 --port", "--base-directory " + state,
|
||||
"--extra-model-paths-config " + filepath.Join(state, "extra_model_paths.yaml"), "--disable-smart-memory"} {
|
||||
if !strings.Contains(string(b), want) {
|
||||
t.Errorf("%q not in what ComfyUI was given: %s", want, b)
|
||||
}
|
||||
}
|
||||
if _, err := os.Lstat(filepath.Join(state, "models", "flux.safetensors")); err != nil {
|
||||
t.Error("the weights were not shown to ComfyUI:", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(state, "custom_nodes")); err != nil {
|
||||
t.Error("ComfyUI lists custom_nodes in its base directory first thing, and it is not there:", err)
|
||||
}
|
||||
if get("/quit") != http.StatusNotFound {
|
||||
t.Error("a path the router never asks for was let through")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Errorf("stopped, the gate said %v", err)
|
||||
}
|
||||
case <-time.After(25 * time.Second):
|
||||
t.Fatal("the gate did not stop")
|
||||
}
|
||||
}
|
||||
|
||||
func TestComfyUIExitingEndsTheGate(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI_EXIT", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
err := RunGate(t.Context(), GateConfig{Listen: "127.0.0.1:0", Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: io.Discard})
|
||||
if err == nil || !strings.Contains(err.Error(), "exit status 3") {
|
||||
t.Fatalf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
type brokenWriter struct{}
|
||||
|
||||
func (brokenWriter) Write([]byte) (int, error) { return 0, errors.New("broken pipe") }
|
||||
|
||||
func TestComfyUIKeepsWritingWhenWhatTheGateWritesToIsGone(t *testing.T) {
|
||||
t.Setenv("LLMGW_FAKE_COMFYUI", "1")
|
||||
weights := filepath.Join(t.TempDir(), "w.safetensors")
|
||||
_ = os.WriteFile(weights, []byte("w"), 0o644)
|
||||
l, _ := net.Listen("tcp", "127.0.0.1:0")
|
||||
listen := l.Addr().String()
|
||||
l.Close()
|
||||
ctx, cancel := context.WithCancel(t.Context())
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- RunGate(ctx, GateConfig{Listen: listen, Weights: weights, State: t.TempDir(),
|
||||
Python: os.Args[0], Dir: t.TempDir(), Output: brokenWriter{}})
|
||||
}()
|
||||
ok := 0
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for ok < 20 && time.Now().Before(deadline) {
|
||||
resp, err := http.Get("http://" + listen + "/health")
|
||||
if err == nil {
|
||||
if resp.StatusCode == http.StatusOK {
|
||||
ok++
|
||||
}
|
||||
resp.Body.Close()
|
||||
}
|
||||
select {
|
||||
case err := <-done:
|
||||
t.Fatalf("the gate ended: %v", err)
|
||||
default:
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
if ok < 20 {
|
||||
t.Fatalf("ComfyUI answered %d times; output it could not hand on stopped it", ok)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"8": {"class_type": "FluxGuidance", "inputs": {"guidance": "{{guidance|3.5}}", "conditioning": ["2", 0]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|20}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["8", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "{{checkpoint}}"}},
|
||||
"2": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{prompt}}", "clip": ["1", 1]}},
|
||||
"3": {"class_type": "CLIPTextEncode", "inputs": {"text": "{{negative_prompt}}", "clip": ["1", 1]}},
|
||||
"4": {"class_type": "EmptySD3LatentImage", "inputs": {"width": "{{width}}", "height": "{{height}}", "batch_size": "{{n}}"}},
|
||||
"5": {"class_type": "KSampler", "inputs": {
|
||||
"seed": "{{seed}}", "steps": "{{steps|4}}", "cfg": 1, "sampler_name": "euler", "scheduler": "simple", "denoise": 1,
|
||||
"model": ["1", 0], "positive": ["2", 0], "negative": ["3", 0], "latent_image": ["4", 0]}},
|
||||
"6": {"class_type": "VAEDecode", "inputs": {"samples": ["5", 0], "vae": ["1", 2]}},
|
||||
"7": {"class_type": "PreviewImage", "inputs": {"images": ["6", 0]}}
|
||||
}
|
||||
@@ -493,6 +493,9 @@ type Personal struct {
|
||||
type Price struct {
|
||||
PromptPer1k int64 `koanf:"prompt_per_1k"`
|
||||
CompletionPer1k int64 `koanf:"completion_per_1k"`
|
||||
// PerImage is the cost of one picture, for a model that makes them
|
||||
// (/v1/images/generations), on top of its prompt's tokens.
|
||||
PerImage int64 `koanf:"per_image"`
|
||||
}
|
||||
|
||||
// Control configures the control plane.
|
||||
@@ -723,7 +726,10 @@ type Endpoint struct {
|
||||
Models []string `koanf:"models"`
|
||||
// Engine names the inference engine, which decides what its metrics are
|
||||
// called. Empty means vLLM's names.
|
||||
Engine string `koanf:"engine"`
|
||||
Engine string `koanf:"engine"`
|
||||
// Workflow is the ComfyUI workflow for engine: comfyui, a built-in's
|
||||
// name or its JSON; flux-schnell unless set.
|
||||
Workflow string `koanf:"workflow"`
|
||||
Pool string `koanf:"pool"`
|
||||
Weight float64 `koanf:"weight"`
|
||||
Priority int `koanf:"priority"`
|
||||
|
||||
@@ -70,7 +70,11 @@ type API struct {
|
||||
// guard is the guardrail policies' library and attachments.
|
||||
guard *GuardAdmin
|
||||
// kb is knowledge bases; nil without a store.
|
||||
kb *KBAdmin
|
||||
kb *KBAdmin
|
||||
// tenancy is organizations, keys and people; nil without a store.
|
||||
tenancy TenancyStore
|
||||
// train is adapter training; nil without a store and agents.
|
||||
train *TrainAdmin
|
||||
settings *Settings
|
||||
// restartAgents tells every connected agent to start again.
|
||||
restartAgents func(reason string) []string
|
||||
@@ -117,6 +121,8 @@ func (a *API) Register(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/models/{name}/resources", a.setResources)
|
||||
a.registerQuotas(mux)
|
||||
a.registerKB(mux)
|
||||
a.registerTenancy(mux)
|
||||
a.registerTrain(mux)
|
||||
a.registerGuard(mux)
|
||||
a.registerSettings(mux)
|
||||
mux.HandleFunc("POST /api/v1/agents/restart", a.restartAgentsHandler)
|
||||
@@ -1605,6 +1611,7 @@ type RequestView struct {
|
||||
TTFTMs int64 `json:"ttft_ms"`
|
||||
PromptTokens int64 `json:"prompt_tokens"`
|
||||
CompletionTokens int64 `json:"completion_tokens"`
|
||||
Images int64 `json:"images,omitempty"`
|
||||
CostMicros int64 `json:"cost_micros"`
|
||||
// What the guardrails did, by name.
|
||||
GuardAction string `json:"guard_action,omitempty"`
|
||||
@@ -1638,7 +1645,7 @@ func (a *API) request(w http.ResponseWriter, r *http.Request) {
|
||||
RequestID: u.RequestID, At: u.At, KeyID: u.KeyID, UserID: u.UserID, ProjectID: u.ProjectID,
|
||||
Model: u.Model, ServedModel: u.ServedModel, Replica: u.Replica, Status: u.Status,
|
||||
LatencyMs: u.LatencyMs, QueueMs: u.QueueMs, TTFTMs: u.TTFTMs,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, CostMicros: u.CostMicros,
|
||||
PromptTokens: u.PromptTokens, CompletionTokens: u.CompletionTokens, Images: u.Images, CostMicros: u.CostMicros,
|
||||
GuardAction: u.GuardAction, GuardPolicy: u.GuardPolicy, GuardCategory: u.GuardCategory, GuardRule: u.GuardRule, GuardMs: u.GuardMs,
|
||||
EndUser: u.EndUser, Tags: u.Tags, CacheHit: u.CacheHit,
|
||||
})
|
||||
@@ -1963,32 +1970,9 @@ func (a *API) setAdapters(w http.ResponseWriter, r *http.Request) {
|
||||
writeError(w, http.StatusBadRequest, "no model "+name)
|
||||
return
|
||||
}
|
||||
base, baseErr := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), m.Source)
|
||||
have := map[string]bool{}
|
||||
for _, ad := range m.Adapters {
|
||||
have[ad.Source] = true
|
||||
}
|
||||
// The file is read only where llama.cpp will load it: a vLLM adapter is
|
||||
// a PEFT directory, which has no header to read, and vLLM itself refuses
|
||||
// one that does not fit the model when it is loaded.
|
||||
ggufs := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
|
||||
for _, ad := range req.Adapters {
|
||||
if have[ad.Source] || !ggufs {
|
||||
continue // already checked when it was attached
|
||||
}
|
||||
h, err := gguf.KindFromURL(r.Context(), safedial.Client(safedial.ControlPlane, 30*time.Second), ad.Source)
|
||||
switch {
|
||||
case err != nil:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %v", ad.Name, err))
|
||||
return
|
||||
case !h.IsAdapter():
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source))
|
||||
return
|
||||
case baseErr == nil && h.Architecture != base.Architecture:
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("adapter %s is for %s models and %s is %s",
|
||||
ad.Name, h.Architecture, m.Name, base.Architecture))
|
||||
return
|
||||
}
|
||||
if err := checkAdapters(r.Context(), m, req.Adapters); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
out, err := a.models.SetAdapters(r.Context(), m.Name, req.Adapters, actorOf(r))
|
||||
if err != nil {
|
||||
@@ -2070,3 +2054,36 @@ func multiNodeReadiness(n inventory.Node) string {
|
||||
}
|
||||
return "ready"
|
||||
}
|
||||
|
||||
// checkAdapters reads each adapter a model does not have yet, where
|
||||
// llama.cpp will load it, to refuse one that is not a LoRA adapter for the
|
||||
// model's architecture: such an adapter is a replica that fails to start,
|
||||
// found minutes later on a node.
|
||||
func checkAdapters(ctx context.Context, m catalog.Model, adapters []catalog.Adapter) error {
|
||||
// A vLLM adapter is a PEFT directory, which has no header to read, and
|
||||
// vLLM itself refuses one that does not fit the model when it is loaded.
|
||||
if !slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" }) {
|
||||
return nil
|
||||
}
|
||||
client := safedial.Client(safedial.ControlPlane, 30*time.Second)
|
||||
base, baseErr := gguf.KindFromURL(ctx, client, m.Source)
|
||||
have := map[string]bool{}
|
||||
for _, ad := range m.Adapters {
|
||||
have[ad.Source] = true
|
||||
}
|
||||
for _, ad := range adapters {
|
||||
if have[ad.Source] {
|
||||
continue // checked when it was attached
|
||||
}
|
||||
h, err := gguf.KindFromURL(ctx, client, ad.Source)
|
||||
switch {
|
||||
case err != nil:
|
||||
return fmt.Errorf("adapter %s: %v", ad.Name, err)
|
||||
case !h.IsAdapter():
|
||||
return fmt.Errorf("adapter %s: %s is a model, not a LoRA adapter", ad.Name, ad.Source)
|
||||
case baseErr == nil && h.Architecture != base.Architecture:
|
||||
return fmt.Errorf("adapter %s is for %s models and %s is %s", ad.Name, h.Architecture, m.Name, base.Architecture)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -30,6 +30,9 @@ type KBStore interface {
|
||||
DeleteDocument(ctx context.Context, collection, name string) (bool, error)
|
||||
SearchChunks(ctx context.Context, q store.KBSearch) ([]store.KBHit, error)
|
||||
SetKBDefault(ctx context.Context, d store.KBDefault) error
|
||||
CollectionChunks(ctx context.Context, c store.KBCollection, limit int) ([]store.KBHit, error)
|
||||
CollectionsFor(ctx context.Context, e store.Effective, write bool) ([]store.KBCollection, error)
|
||||
EffectiveFor(ctx context.Context, scope store.Scope, id, org string) (store.Effective, error)
|
||||
KBDefaults(ctx context.Context) ([]store.KBDefault, error)
|
||||
}
|
||||
|
||||
@@ -161,6 +164,7 @@ func (a *API) registerKB(mux Mux) {
|
||||
mux.HandleFunc("DELETE /api/v1/kb/grants", a.kbGrant(false))
|
||||
mux.HandleFunc("POST /api/v1/kb/search", a.kbSearch)
|
||||
mux.HandleFunc("POST /api/v1/kb/defaults", a.kbSetDefault)
|
||||
mux.HandleFunc("POST /api/v1/kb/collections/{id}/dataset", a.kbDataset)
|
||||
}
|
||||
|
||||
// kbReady answers for a gateway without knowledge bases, and says whether
|
||||
@@ -495,7 +499,11 @@ func (a *API) kbSearch(w http.ResponseWriter, r *http.Request) {
|
||||
// Said, not swallowed: a search that quietly lost its meaning half
|
||||
// looks like a collection that does not know the answer.
|
||||
if a.kb.Embed != nil {
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, in.Query)
|
||||
question := in.Query
|
||||
if m, ok := a.ctl.Catalog().Resolve(c.EmbedModel); ok {
|
||||
question = m.QueryPrefix + question
|
||||
}
|
||||
v, err := a.kb.Embed(r.Context(), c.EmbedModel, question)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, c.EmbedModel+" did not embed the question: "+err.Error())
|
||||
return
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Training data from a collection (BACKLOG F4): questions a person might
|
||||
// ask of each chunk and the answers it holds, written by a model of the
|
||||
// cluster, for people to read through before an adapter is trained on them.
|
||||
// What an adapter learns well is how the organisation's answers sound; what
|
||||
// they say is still the knowledge base's to give.
|
||||
|
||||
// KBDatasetRequest asks for training data from a collection.
|
||||
type KBDatasetRequest struct {
|
||||
// Model writes the questions and answers: a chat model of the catalog.
|
||||
Model string `json:"model"`
|
||||
// PerChunk is how many pairs each chunk gives; 8 when 0.
|
||||
PerChunk int `json:"per_chunk,omitempty"`
|
||||
// MaxChunks bounds how many chunks are read; 500 when 0.
|
||||
MaxChunks int `json:"max_chunks,omitempty"`
|
||||
}
|
||||
|
||||
// KBDatasetLine is one line of the answer: a pair, or why a chunk gave none.
|
||||
type KBDatasetLine struct {
|
||||
Q string `json:"q,omitempty"`
|
||||
A string `json:"a,omitempty"`
|
||||
Doc string `json:"doc"`
|
||||
Ord int `json:"ord"`
|
||||
// Collection is where the pair came from: an adapter trained on it
|
||||
// belongs to a part of the tree that may read it.
|
||||
Collection string `json:"collection,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
|
||||
const kbDatasetPrompt = `From the passage of an organisation's document below, write %d different questions an employee might ask, each with a short, correct answer.
|
||||
Rules: use only what the passage says. Phrase the questions differently from each other (formal, everyday, short, indirect). Answers are one or two sentences and keep the passage's numbers exactly. Leave out e-mail addresses and people's names. Write in the passage's own language.
|
||||
Output: one JSON object per line, {"q": "...", "a": "..."}, and nothing else.
|
||||
|
||||
DOCUMENT %s:
|
||||
%s`
|
||||
|
||||
var thinkBlock = regexp.MustCompile(`(?s)<think>.*?</think>`)
|
||||
|
||||
func (a *API) kbDataset(w http.ResponseWriter, r *http.Request) {
|
||||
c, ok := a.kbCollection(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if a.replay == nil {
|
||||
writeError(w, http.StatusNotImplemented, "no router here to ask a model")
|
||||
return
|
||||
}
|
||||
var in KBDatasetRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
m, found := a.ctl.Catalog().Resolve(in.Model)
|
||||
if !found {
|
||||
writeError(w, http.StatusBadRequest, "no model "+in.Model+" in the catalog")
|
||||
return
|
||||
}
|
||||
if m.Capabilities.Embeddings {
|
||||
writeError(w, http.StatusBadRequest, m.Name+" embeds; questions are written by a chat model")
|
||||
return
|
||||
}
|
||||
per := in.PerChunk
|
||||
if per <= 0 || per > 30 {
|
||||
per = 8
|
||||
}
|
||||
limit := in.MaxChunks
|
||||
if limit <= 0 || limit > 5000 {
|
||||
limit = 500
|
||||
}
|
||||
chunks, err := a.kb.Store.CollectionChunks(r.Context(), c, limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/x-ndjson")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
enc := json.NewEncoder(w)
|
||||
flush := func() {
|
||||
if f, ok := w.(http.Flusher); ok {
|
||||
f.Flush()
|
||||
}
|
||||
}
|
||||
for _, ch := range chunks {
|
||||
if r.Context().Err() != nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"model": m.Name, "temperature": 0.7, "max_tokens": 200 * per,
|
||||
"messages": []map[string]string{{"role": "user", "content": fmt.Sprintf(kbDatasetPrompt, per, ch.Document, ch.Body)}}})
|
||||
status, out, _, err := a.replay(r.Context(), "/v1/chat/completions", m.Name, body)
|
||||
line := KBDatasetLine{Doc: ch.Document, Ord: ch.Ord, Collection: c.ID}
|
||||
if err == nil && status != http.StatusOK {
|
||||
err = fmt.Errorf("%d: %s", status, strings.TrimSpace(string(out)))
|
||||
}
|
||||
if err != nil {
|
||||
line.Error = err.Error()
|
||||
_ = enc.Encode(line)
|
||||
flush()
|
||||
continue
|
||||
}
|
||||
pairs := datasetPairs(out)
|
||||
if len(pairs) == 0 {
|
||||
line.Error = "the model gave no pairs it could read"
|
||||
_ = enc.Encode(line)
|
||||
}
|
||||
for _, p := range pairs {
|
||||
p.Doc, p.Ord, p.Collection = ch.Document, ch.Ord, c.ID
|
||||
_ = enc.Encode(p)
|
||||
}
|
||||
flush()
|
||||
}
|
||||
}
|
||||
|
||||
// datasetPairs reads the {"q","a"} lines out of a chat completion.
|
||||
func datasetPairs(completion []byte) []KBDatasetLine {
|
||||
var res struct {
|
||||
Choices []struct {
|
||||
Message struct {
|
||||
Content string `json:"content"`
|
||||
} `json:"message"`
|
||||
} `json:"choices"`
|
||||
}
|
||||
if json.Unmarshal(completion, &res) != nil || len(res.Choices) == 0 {
|
||||
return nil
|
||||
}
|
||||
text := thinkBlock.ReplaceAllString(res.Choices[0].Message.Content, "")
|
||||
var out []KBDatasetLine
|
||||
for _, line := range strings.Split(text, "\n") {
|
||||
line = strings.TrimSuffix(strings.TrimSpace(line), ",")
|
||||
if !strings.HasPrefix(line, "{") || !strings.HasSuffix(line, "}") {
|
||||
continue
|
||||
}
|
||||
var p struct{ Q, A string }
|
||||
if json.Unmarshal([]byte(line), &p) == nil && strings.TrimSpace(p.Q) != "" && strings.TrimSpace(p.A) != "" {
|
||||
out = append(out, KBDatasetLine{Q: strings.TrimSpace(p.Q), A: strings.TrimSpace(p.A)})
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestPairsAreReadOutOfWhatAModelWrote(t *testing.T) {
|
||||
completion := `{"choices":[{"message":{"content":"<think>let me see</think>\n{\"q\": \"Babalık izni kaç gün?\", \"a\": \"10 iş günü.\"},\nnot json\n{\"q\": \"\", \"a\": \"empty question\"}\n{\"q\": \"Ne zaman?\", \"a\": \"Doğumdan sonra.\"}"}}]}`
|
||||
got := datasetPairs([]byte(completion))
|
||||
if len(got) != 2 || got[0].Q != "Babalık izni kaç gün?" || got[1].A != "Doğumdan sonra." {
|
||||
t.Errorf("pairs %+v", got)
|
||||
}
|
||||
if datasetPairs([]byte(`{"choices":[]}`)) != nil {
|
||||
t.Error("pairs from no choice")
|
||||
}
|
||||
}
|
||||
|
||||
func TestADatasetIsWrittenChunkByChunk(t *testing.T) {
|
||||
dsn := os.Getenv("LLMGW_TEST_POSTGRES")
|
||||
if dsn == "" {
|
||||
t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
|
||||
}
|
||||
ctx := context.Background()
|
||||
s, err := store.OpenPostgres(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
if s.KnowledgeAvailable() != nil {
|
||||
t.Skip("no pgvector")
|
||||
}
|
||||
models := append(kbModels(), catalog.Model{Name: "writer", Source: "/w",
|
||||
Profiles: []catalog.Profile{{Name: "cpu", GPUClass: "h100-80g", TP: 1}}})
|
||||
a, _ := setup(t, models)
|
||||
a.SetKB(&KBAdmin{Store: s})
|
||||
asked := 0
|
||||
a.SetReplayer(func(_ context.Context, path, model string, body []byte) (int, []byte, time.Duration, error) {
|
||||
asked++
|
||||
if path != "/v1/chat/completions" || model != "writer" || !strings.Contains(string(body), "Write in the passage") {
|
||||
t.Errorf("asked %s %s %s", path, model, body)
|
||||
}
|
||||
if strings.Contains(string(body), "bozuk") {
|
||||
return 200, []byte(`{"choices":[{"message":{"content":"sorry"}}]}`), 0, nil
|
||||
}
|
||||
return 200, []byte(`{"choices":[{"message":{"content":"{\"q\":\"Soru?\",\"a\":\"Cevap.\"}"}}]}`), 0, nil
|
||||
})
|
||||
org, _ := s.CreateOrg(ctx, "ds-"+strings.ToLower(store.NewID("")))
|
||||
c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeOrg, OwnerID: org, Name: "d", EmbedModel: "bge-m3", Dims: 3})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, d := range []string{"a.md", "b.md"} {
|
||||
body := "iyi metin"
|
||||
if d == "b.md" {
|
||||
body = "bozuk metin"
|
||||
}
|
||||
if _, _, err := s.PutDocument(ctx, c, store.KBDocument{Name: d, Digest: d}, []store.KBChunk{{Body: body, Embedding: []float32{1, 0, 0}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"writer","per_chunk":3}`)))
|
||||
var lines []KBDatasetLine
|
||||
for _, l := range strings.Split(strings.TrimSpace(rec.Body.String()), "\n") {
|
||||
var x KBDatasetLine
|
||||
_ = json.Unmarshal([]byte(l), &x)
|
||||
lines = append(lines, x)
|
||||
}
|
||||
if rec.Code != 200 || asked != 2 || len(lines) != 2 || lines[0].Q != "Soru?" || lines[0].Doc != "a.md" ||
|
||||
lines[1].Doc != "b.md" || lines[1].Error == "" {
|
||||
t.Errorf("%d asked %d: %+v", rec.Code, asked, lines)
|
||||
}
|
||||
// An embedding model writes no questions.
|
||||
rec = httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/kb/collections/"+c.ID+"/dataset", strings.NewReader(`{"model":"bge-m3"}`)))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("an embedding model: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -67,6 +67,12 @@ var PathParams = map[string]string{
|
||||
"federation/{peer}": "The federation peer's name, as `llmgw ctl federation list` lists it.",
|
||||
"holds/{domain}": "The domain as level:name, rack:r12-07, as `llmgw ctl facility` lists it.",
|
||||
"collections/{id}": "The collection's ID, its name when that is unique, or ORG/NAME.",
|
||||
"orgs/{id}": "The organization's ID or name.",
|
||||
"teams/{id}": "The team's ID.",
|
||||
"projects/{id}": "The project's ID.",
|
||||
"keys/{id}": "The key's ID, as listed (not the key itself).",
|
||||
"users/{id}": "The person's ID or email.",
|
||||
"artifacts/{kind}": "What the job made: peft, a gzipped tar of the PEFT adapter, or gguf, the adapter for llama.cpp.",
|
||||
}
|
||||
|
||||
// Routes is every endpoint Register mounts.
|
||||
@@ -288,10 +294,77 @@ var Routes = []Route{
|
||||
Description: "On an organization, team, project or key; the narrowest one with a default wins. No collections clears it. " +
|
||||
"A request's own X-Llmgw-Knowledge overrides it, and X-Llmgw-Knowledge: none turns it off.",
|
||||
Request: KBDefaultRequest{}, Response: KBDefaultView{}},
|
||||
{Method: "POST", Path: "/api/v1/kb/collections/{id}/dataset", Summary: "Training data from a collection: questions and answers a chat model writes from each chunk",
|
||||
Description: "NDJSON, a line per pair as each chunk is read, or a line with error for a chunk that gave none. For people to correct before an adapter is trained on it.",
|
||||
Request: KBDatasetRequest{}, Response: KBDatasetLine{}},
|
||||
{Method: "POST", Path: "/api/v1/kb/search", Summary: "Try a search of a collection, as an administrator",
|
||||
Description: "No tenancy applies: this is for checking what a collection holds and finds.",
|
||||
Request: KBSearchRequest{}, Response: []KBHitView{}},
|
||||
|
||||
// LoRA adapters trained on the nodes.
|
||||
// Organizations, teams, projects, keys and people.
|
||||
{Method: "GET", Path: "/api/v1/orgs", Summary: "The tenancy tree: every organization, its teams and their projects",
|
||||
Response: []OrgView{}},
|
||||
{Method: "POST", Path: "/api/v1/orgs", Summary: "Add an organization",
|
||||
Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
|
||||
{Method: "PATCH", Path: "/api/v1/orgs/{id}", Summary: "Rename an organization",
|
||||
Request: TenancyChangeRequest{}, Response: TenancyItem{}},
|
||||
{Method: "DELETE", Path: "/api/v1/orgs/{id}", Summary: "Remove an organization that has no teams",
|
||||
Response: map[string]any{}},
|
||||
{Method: "POST", Path: "/api/v1/teams", Summary: "Add a team to an organization",
|
||||
Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
|
||||
{Method: "PATCH", Path: "/api/v1/teams/{id}", Summary: "Rename a team, or move it to another organization",
|
||||
Description: "Everything under it moves too, and from then on inherits the new organization's limits.",
|
||||
Request: TenancyChangeRequest{}, Response: TenancyItem{}},
|
||||
{Method: "DELETE", Path: "/api/v1/teams/{id}", Summary: "Remove a team that has no projects",
|
||||
Response: map[string]any{}},
|
||||
{Method: "POST", Path: "/api/v1/projects", Summary: "Add a project to a team",
|
||||
Request: TenancyCreateRequest{}, Response: TenancyItem{}, Status: http.StatusCreated},
|
||||
{Method: "PATCH", Path: "/api/v1/projects/{id}", Summary: "Rename a project, or move it to another team",
|
||||
Request: TenancyChangeRequest{}, Response: TenancyItem{}},
|
||||
{Method: "DELETE", Path: "/api/v1/projects/{id}", Summary: "Remove a project with no people, keys or collections",
|
||||
Description: "A project that ever had a key keeps it, revoked, because usage refers to it; rename such a project instead.",
|
||||
Response: map[string]any{}},
|
||||
{Method: "GET", Path: "/api/v1/keys", Summary: "API keys, with their project and owner; never the keys themselves",
|
||||
Query: []Param{{Name: "project", Description: "Only this project's keys: ORG/TEAM/PROJECT or its ID."}},
|
||||
Response: []KeyView{}},
|
||||
{Method: "POST", Path: "/api/v1/keys", Summary: "Issue an API key",
|
||||
Description: "The answer holds the key, once: only its hash is kept. A lifetime longer than a scope above it allows is refused.",
|
||||
Request: KeyCreateRequest{}, Response: KeyCreated{}, Status: http.StatusCreated},
|
||||
{Method: "GET", Path: "/api/v1/keys/{id}", Summary: "One API key", Response: KeyView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/keys/{id}", Summary: "Revoke an API key",
|
||||
Description: "A soft delete: usage refers to it. A gateway that has it cached accepts it until auth.cache_ttl.",
|
||||
Response: map[string]any{}},
|
||||
{Method: "GET", Path: "/api/v1/users", Summary: "People, with where each is a member",
|
||||
Query: []Param{{Name: "state", Description: "Only people in this state: pending, active or disabled."}},
|
||||
Response: []UserView{}},
|
||||
{Method: "GET", Path: "/api/v1/users/{id}", Summary: "One person, with where they are a member", Response: UserView{}},
|
||||
{Method: "POST", Path: "/api/v1/users/{id}/memberships", Summary: "Put a person in an organization, team or project",
|
||||
Description: "Already there, only the role changes.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/users/{id}/memberships", Summary: "Take a person out of an organization, team or project",
|
||||
Description: "Their keys in projects they no longer reach are revoked.",
|
||||
Request: MembershipRequest{}, Response: UserView{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/train/jobs", Summary: "Queue a LoRA adapter's training on a node",
|
||||
Description: "The node trains when its agent is there, nothing else trains on it, and a card has gpu_memory_mib free; " +
|
||||
"the pairs are NDJSON of {\"q\", \"a\"}. An adapter learns how answers sound far better than what they say.",
|
||||
Request: TrainRequest{}, Response: TrainJobView{}, Status: http.StatusCreated},
|
||||
{Method: "GET", Path: "/api/v1/train/jobs", Summary: "Training jobs, newest first",
|
||||
Query: []Param{{Name: "limit", Description: "How many; 50 when absent.", Type: "integer"}},
|
||||
Response: []TrainJobView{}},
|
||||
{Method: "GET", Path: "/api/v1/train/jobs/{id}", Summary: "One training job: how it goes, and what it made",
|
||||
Response: TrainJobView{}},
|
||||
{Method: "POST", Path: "/api/v1/train/jobs/{id}/cancel", Summary: "Stop a training job",
|
||||
Response: TrainJobView{}},
|
||||
{Method: "DELETE", Path: "/api/v1/train/jobs/{id}", Summary: "Remove an ended training job and what it made",
|
||||
Response: map[string]any{}},
|
||||
{Method: "POST", Path: "/api/v1/train/jobs/{id}/publish", Summary: "Make what a finished job made the model's adapter",
|
||||
Description: "Kept on the mirror — the GGUF for a model llama.cpp serves, the PEFT files for one vLLM serves — and added to the model's adapters after the same check as POST /api/v1/models/{name}/adapters.",
|
||||
Response: TrainPublishView{}},
|
||||
{Method: "GET", Path: "/api/v1/train/jobs/{id}/artifacts/{kind}", Summary: "What a training job made: peft (a gzipped tar) or gguf",
|
||||
Response: map[string]any{}},
|
||||
|
||||
{Method: "POST", Path: "/api/v1/agents/restart", Summary: "Have every node's agent start again; their engines keep serving",
|
||||
Response: AgentRestartView{}},
|
||||
|
||||
|
||||
@@ -0,0 +1,658 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// The tenancy tree, keys and people's memberships, administered over the
|
||||
// API as `llmgw org`, `keys` and `users` do against the store: so that
|
||||
// they can be run from elsewhere, and kept in Terraform.
|
||||
|
||||
// TenancyStore is the part of the store the tenancy endpoints need.
|
||||
type TenancyStore interface {
|
||||
Tree(ctx context.Context) ([]store.TreeOrg, error)
|
||||
CreateOrg(ctx context.Context, name string) (string, error)
|
||||
CreateTeam(ctx context.Context, orgID, name string) (string, error)
|
||||
CreateProject(ctx context.Context, teamID, name string) (string, error)
|
||||
RenameOrg(ctx context.Context, id, name string) error
|
||||
RenameTeam(ctx context.Context, id, name string) error
|
||||
RenameProject(ctx context.Context, id, name string) error
|
||||
MoveTeam(ctx context.Context, teamID, orgID string) error
|
||||
MoveProject(ctx context.Context, projectID, teamID string) error
|
||||
DeleteOrg(ctx context.Context, id string) error
|
||||
DeleteTeam(ctx context.Context, id string) error
|
||||
DeleteProject(ctx context.Context, id string) error
|
||||
|
||||
AllKeys(ctx context.Context) ([]store.KeyRow, error)
|
||||
KeyByID(ctx context.Context, id string) (store.Key, error)
|
||||
CreateKey(ctx context.Context, k store.Key) error
|
||||
CreateUserKey(ctx context.Context, userID string, k store.Key) error
|
||||
RevokeKey(ctx context.Context, id string, at time.Time) error
|
||||
|
||||
Users(ctx context.Context, state store.UserState) ([]store.User, error)
|
||||
UserByID(ctx context.Context, id string) (store.User, error)
|
||||
UserByEmail(ctx context.Context, email string) (store.User, error)
|
||||
MembershipsOf(ctx context.Context, userID string) ([]store.Membership, error)
|
||||
AddMembership(ctx context.Context, userID string, scope store.Scope, scopeID, role, by string, at time.Time) error
|
||||
RemoveMembership(ctx context.Context, userID string, scope store.Scope, scopeID string, at time.Time) (int, error)
|
||||
}
|
||||
|
||||
// SetTenancy enables the tenancy endpoints. The quota store must be set
|
||||
// too: names are resolved through it.
|
||||
func (a *API) SetTenancy(t TenancyStore) { a.tenancy = t }
|
||||
|
||||
// OrgView is an organization with what is in it.
|
||||
type OrgView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Members int `json:"members"`
|
||||
Teams []TeamView `json:"teams"`
|
||||
}
|
||||
|
||||
// TeamView is a team with its projects.
|
||||
type TeamView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Org string `json:"org"`
|
||||
Members int `json:"members"`
|
||||
Projects []ProjectView `json:"projects"`
|
||||
}
|
||||
|
||||
// ProjectView is a project, with how many people and keys are in it.
|
||||
type ProjectView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Team string `json:"team"`
|
||||
// Path is ORG/TEAM/PROJECT by name.
|
||||
Path string `json:"path"`
|
||||
Users int `json:"users"`
|
||||
Keys int `json:"keys"`
|
||||
ActiveKeys int `json:"active_keys"`
|
||||
}
|
||||
|
||||
// TenancyCreateRequest adds an organization, a team or a project.
|
||||
type TenancyCreateRequest struct {
|
||||
Name string `json:"name"`
|
||||
// In is the organization a team goes in, or the team a project goes in
|
||||
// (ORG/TEAM), by name or ID; nothing for an organization.
|
||||
In string `json:"in,omitempty"`
|
||||
}
|
||||
|
||||
// TenancyChangeRequest renames an organization, team or project, or moves
|
||||
// a team to another organization or a project to another team.
|
||||
type TenancyChangeRequest struct {
|
||||
Name string `json:"name,omitempty"`
|
||||
// In is where it moves to, by name or ID; not for an organization.
|
||||
In string `json:"in,omitempty"`
|
||||
}
|
||||
|
||||
// TenancyItem is an organization, team or project added or changed.
|
||||
type TenancyItem struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name,omitempty"`
|
||||
// In is the organization a team is in, or the team a project is in, by
|
||||
// ID.
|
||||
In string `json:"in,omitempty"`
|
||||
}
|
||||
|
||||
// KeyView is a key as listed: never the key itself, which is not kept.
|
||||
type KeyView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
// Prefix is the key's first characters, to tell it apart.
|
||||
Prefix string `json:"prefix"`
|
||||
Project string `json:"project,omitempty"`
|
||||
// ProjectPath is ORG/TEAM/PROJECT, or empty for a personal key.
|
||||
ProjectPath string `json:"project_path,omitempty"`
|
||||
User string `json:"user,omitempty"`
|
||||
UserEmail string `json:"user_email,omitempty"`
|
||||
Band string `json:"band,omitempty"`
|
||||
RoutingProfile string `json:"routing_profile,omitempty"`
|
||||
State string `json:"state"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
ExpiresAt time.Time `json:"expires_at,omitzero"`
|
||||
RevokedAt time.Time `json:"revoked_at,omitzero"`
|
||||
LastUsedAt time.Time `json:"last_used_at,omitzero"`
|
||||
}
|
||||
|
||||
// KeyCreateRequest issues a key: a project's, a person's in a project they
|
||||
// are a member of, or a person's own with no project.
|
||||
type KeyCreateRequest struct {
|
||||
Name string `json:"name"`
|
||||
// Project is ORG/TEAM/PROJECT or its ID.
|
||||
Project string `json:"project,omitempty"`
|
||||
// User is the person it belongs to, by ID or email.
|
||||
User string `json:"user,omitempty"`
|
||||
Band string `json:"band,omitempty"`
|
||||
RoutingProfile string `json:"routing_profile,omitempty"`
|
||||
// ExpiresIn is its lifetime, as 720h; none never expires, unless a
|
||||
// scope above it limits how long keys live.
|
||||
ExpiresIn string `json:"expires_in,omitempty"`
|
||||
}
|
||||
|
||||
// KeyCreated is a new key, with the key itself: this answer is the only
|
||||
// time it is given.
|
||||
type KeyCreated struct {
|
||||
KeyView
|
||||
Secret string `json:"secret"`
|
||||
}
|
||||
|
||||
// UserView is a person, with where they are a member.
|
||||
type UserView struct {
|
||||
ID string `json:"id"`
|
||||
Email string `json:"email"`
|
||||
Name string `json:"name,omitempty"`
|
||||
State string `json:"state"`
|
||||
// Role is the person's part in running the gateway, if any: viewer,
|
||||
// operator, billing or admin.
|
||||
Role string `json:"role,omitempty"`
|
||||
Memberships []MembershipView `json:"memberships"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastLoginAt time.Time `json:"last_login_at,omitzero"`
|
||||
}
|
||||
|
||||
// MembershipView is one place a person belongs.
|
||||
type MembershipView struct {
|
||||
Target string `json:"target"`
|
||||
Path string `json:"path,omitempty"`
|
||||
Role string `json:"role"`
|
||||
At time.Time `json:"at"`
|
||||
By string `json:"by,omitempty"`
|
||||
}
|
||||
|
||||
// MembershipRequest puts a person in an organization, team or project, or
|
||||
// with DELETE takes them out.
|
||||
type MembershipRequest struct {
|
||||
// Target is org:NAME, team:ORG/TEAM or project:ORG/TEAM/PROJECT, or the
|
||||
// ID after the colon.
|
||||
Target string `json:"target"`
|
||||
// Role is member or manager; member when absent. Ignored by DELETE.
|
||||
Role string `json:"role,omitempty"`
|
||||
}
|
||||
|
||||
func (a *API) registerTenancy(mux Mux) {
|
||||
mux.HandleFunc("GET /api/v1/orgs", a.tenancyTree)
|
||||
for _, level := range []store.Scope{store.ScopeOrg, store.ScopeTeam, store.ScopeProject} {
|
||||
path := "/api/v1/" + string(level) + "s"
|
||||
mux.HandleFunc("POST "+path, a.tenancyCreate(level))
|
||||
mux.HandleFunc("PATCH "+path+"/{id}", a.tenancyChange(level))
|
||||
mux.HandleFunc("DELETE "+path+"/{id}", a.tenancyDelete(level))
|
||||
}
|
||||
mux.HandleFunc("GET /api/v1/keys", a.keyList)
|
||||
mux.HandleFunc("POST /api/v1/keys", a.keyCreate)
|
||||
mux.HandleFunc("GET /api/v1/keys/{id}", a.keyGet)
|
||||
mux.HandleFunc("DELETE /api/v1/keys/{id}", a.keyRevoke)
|
||||
mux.HandleFunc("GET /api/v1/users", a.userList)
|
||||
mux.HandleFunc("GET /api/v1/users/{id}", a.userGet)
|
||||
mux.HandleFunc("POST /api/v1/users/{id}/memberships", a.membership(true))
|
||||
mux.HandleFunc("DELETE /api/v1/users/{id}/memberships", a.membership(false))
|
||||
}
|
||||
|
||||
// tenancyReady answers for a control plane without the store, and says
|
||||
// whether it did.
|
||||
func (a *API) tenancyReady(w http.ResponseWriter) bool {
|
||||
if a.tenancy == nil || a.quotas == nil {
|
||||
writeError(w, http.StatusNotImplemented, "organizations, keys and people need the store, which this control plane runs without")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// tenancyStatus is the status a store error answers with.
|
||||
func tenancyStatus(err error) int {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
return http.StatusNotFound
|
||||
case errors.Is(err, store.ErrDuplicateName), errors.Is(err, store.ErrNotEmpty):
|
||||
return http.StatusConflict
|
||||
}
|
||||
return http.StatusBadRequest
|
||||
}
|
||||
|
||||
func (a *API) tenancyTree(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
tree, err := a.tenancy.Tree(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []OrgView{}
|
||||
for _, o := range tree {
|
||||
ov := OrgView{ID: o.ID, Name: o.Name, Members: o.Members, Teams: []TeamView{}}
|
||||
for _, t := range o.Teams {
|
||||
tv := TeamView{ID: t.ID, Name: t.Name, Org: o.ID, Members: t.Members, Projects: []ProjectView{}}
|
||||
for _, p := range t.Projects {
|
||||
tv.Projects = append(tv.Projects, ProjectView{ID: p.ID, Name: p.Name, Team: t.ID,
|
||||
Path: o.Name + "/" + t.Name + "/" + p.Name, Users: p.Users, Keys: p.Keys, ActiveKeys: p.ActiveKeys})
|
||||
}
|
||||
ov.Teams = append(ov.Teams, tv)
|
||||
}
|
||||
out = append(out, ov)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
// parentOf is the level something at level sits in.
|
||||
func parentOf(level store.Scope) store.Scope {
|
||||
if level == store.ScopeProject {
|
||||
return store.ScopeTeam
|
||||
}
|
||||
return store.ScopeOrg
|
||||
}
|
||||
|
||||
// levelNoun is a level as a sentence says it.
|
||||
func levelNoun(level store.Scope) string {
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
return "an organization"
|
||||
case store.ScopeTeam:
|
||||
return "a team"
|
||||
}
|
||||
return "a project"
|
||||
}
|
||||
|
||||
func (a *API) tenancyCreate(level store.Scope) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in TenancyCreateRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
in.Name = strings.TrimSpace(in.Name)
|
||||
if in.Name == "" {
|
||||
writeError(w, http.StatusBadRequest, "a name is needed")
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
var id, parent string
|
||||
var err error
|
||||
if level != store.ScopeOrg {
|
||||
if in.In == "" {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("%s goes in %s: give in", levelNoun(level), levelNoun(parentOf(level))))
|
||||
return
|
||||
}
|
||||
if _, parent, _, err = a.resolveScope(ctx, string(parentOf(level))+":"+in.In); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
id, err = a.tenancy.CreateOrg(ctx, in.Name)
|
||||
case store.ScopeTeam:
|
||||
id, err = a.tenancy.CreateTeam(ctx, parent, in.Name)
|
||||
default:
|
||||
id, err = a.tenancy.CreateProject(ctx, parent, in.Name)
|
||||
}
|
||||
if errors.Is(err, store.ErrDuplicateName) {
|
||||
writeError(w, http.StatusConflict, fmt.Sprintf("there is already %s named %q there", levelNoun(level), in.Name))
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, TenancyItem{ID: id, Name: in.Name, In: parent})
|
||||
}
|
||||
}
|
||||
|
||||
func (a *API) tenancyChange(level store.Scope) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in TenancyChangeRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
if in.Name == "" && in.In == "" {
|
||||
writeError(w, http.StatusBadRequest, "nothing to change: give name, in or both")
|
||||
return
|
||||
}
|
||||
out := TenancyItem{ID: id}
|
||||
if in.In != "" {
|
||||
if level == store.ScopeOrg {
|
||||
writeError(w, http.StatusBadRequest, "an organization is in nothing; only a team or a project moves")
|
||||
return
|
||||
}
|
||||
_, to, _, err := a.resolveScope(ctx, string(parentOf(level))+":"+in.In)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
if level == store.ScopeTeam {
|
||||
err = a.tenancy.MoveTeam(ctx, id, to)
|
||||
} else {
|
||||
err = a.tenancy.MoveProject(ctx, id, to)
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
out.In = to
|
||||
}
|
||||
if name := strings.TrimSpace(in.Name); name != "" {
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
err = a.tenancy.RenameOrg(ctx, id, name)
|
||||
case store.ScopeTeam:
|
||||
err = a.tenancy.RenameTeam(ctx, id, name)
|
||||
default:
|
||||
err = a.tenancy.RenameProject(ctx, id, name)
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
out.Name = name
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
}
|
||||
|
||||
func (a *API) tenancyDelete(level store.Scope) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
_, id, _, err := a.resolveScope(ctx, string(level)+":"+r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
switch level {
|
||||
case store.ScopeOrg:
|
||||
err = a.tenancy.DeleteOrg(ctx, id)
|
||||
case store.ScopeTeam:
|
||||
err = a.tenancy.DeleteTeam(ctx, id)
|
||||
default:
|
||||
err = a.tenancy.DeleteProject(ctx, id)
|
||||
}
|
||||
if err != nil {
|
||||
// Only what is empty goes: the store says what is still in it.
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"id": id, "deleted": true})
|
||||
}
|
||||
}
|
||||
|
||||
// keyState is a key's state at an instant.
|
||||
func keyState(k store.Key, now time.Time) string {
|
||||
switch {
|
||||
case !k.RevokedAt.IsZero():
|
||||
return "revoked"
|
||||
case !k.ExpiresAt.IsZero() && !now.Before(k.ExpiresAt):
|
||||
return "expired"
|
||||
}
|
||||
return "active"
|
||||
}
|
||||
|
||||
func keyView(k store.Key, path, email string, now time.Time) KeyView {
|
||||
// The store says "Acme / ml / search"; a path given to the API is
|
||||
// written Acme/ml/search, and is answered the same way.
|
||||
path = strings.ReplaceAll(path, " / ", "/")
|
||||
return KeyView{ID: k.ID, Name: k.Name, Prefix: k.Prefix, Project: k.ProjectID, ProjectPath: path,
|
||||
User: k.UserID, UserEmail: email, Band: k.Band, RoutingProfile: k.RoutingProfile, State: keyState(k, now),
|
||||
CreatedAt: k.CreatedAt, ExpiresAt: k.ExpiresAt, RevokedAt: k.RevokedAt, LastUsedAt: k.LastUsedAt}
|
||||
}
|
||||
|
||||
func (a *API) keyList(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
project := ""
|
||||
if p := r.URL.Query().Get("project"); p != "" {
|
||||
var err error
|
||||
if _, project, _, err = a.resolveScope(r.Context(), "project:"+p); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
rows, err := a.tenancy.AllKeys(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
out := []KeyView{}
|
||||
for _, k := range rows {
|
||||
if project == "" || k.ProjectID == project {
|
||||
out = append(out, keyView(k.Key, k.ProjectPath, k.OwnerEmail, now))
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) keyGet(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
rows, err := a.tenancy.AllKeys(r.Context())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
for _, k := range rows {
|
||||
if k.ID == r.PathValue("id") {
|
||||
writeJSON(w, http.StatusOK, keyView(k.Key, k.ProjectPath, k.OwnerEmail, time.Now()))
|
||||
return
|
||||
}
|
||||
}
|
||||
writeError(w, http.StatusNotFound, "no key "+r.PathValue("id"))
|
||||
}
|
||||
|
||||
// findUser reads a person by ID or email.
|
||||
func (a *API) findUser(ctx context.Context, ref string) (store.User, error) {
|
||||
ref = strings.TrimSpace(ref)
|
||||
if u, err := a.tenancy.UserByID(ctx, ref); err == nil {
|
||||
return u, nil
|
||||
}
|
||||
if strings.Contains(ref, "@") {
|
||||
if u, err := a.tenancy.UserByEmail(ctx, ref); err == nil {
|
||||
return u, nil
|
||||
}
|
||||
}
|
||||
return store.User{}, fmt.Errorf("%w: no person %q", store.ErrNotFound, ref)
|
||||
}
|
||||
|
||||
func (a *API) keyCreate(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in KeyCreateRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
if strings.TrimSpace(in.Name) == "" {
|
||||
writeError(w, http.StatusBadRequest, "a key needs a name")
|
||||
return
|
||||
}
|
||||
if in.Project == "" && in.User == "" {
|
||||
writeError(w, http.StatusBadRequest, "a key belongs to a project, a person, or a person in a project: give project, user or both")
|
||||
return
|
||||
}
|
||||
var owner store.User
|
||||
var err error
|
||||
if in.User != "" {
|
||||
if owner, err = a.findUser(ctx, in.User); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
project, path := "", ""
|
||||
if in.Project != "" {
|
||||
if _, project, path, err = a.resolveScope(ctx, "project:"+in.Project); err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
var expires time.Duration
|
||||
if in.ExpiresIn != "" {
|
||||
if expires, err = time.ParseDuration(in.ExpiresIn); err != nil || expires <= 0 {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("expires_in %q is not a positive duration such as 720h", in.ExpiresIn))
|
||||
return
|
||||
}
|
||||
}
|
||||
secret, k, err := store.NewKey(store.NewID("key-"), strings.TrimSpace(in.Name), project)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
k.CreatedAt = time.Now()
|
||||
k.Band, k.RoutingProfile = in.Band, in.RoutingProfile
|
||||
if expires > 0 {
|
||||
k.ExpiresAt = k.CreatedAt.Add(expires)
|
||||
}
|
||||
if owner.ID != "" {
|
||||
err = a.tenancy.CreateUserKey(ctx, owner.ID, k)
|
||||
} else {
|
||||
err = a.tenancy.CreateKey(ctx, k)
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotMember):
|
||||
writeError(w, http.StatusConflict, fmt.Sprintf("%s is not a member of %s; add the membership first", owner.Email, path))
|
||||
return
|
||||
case errors.Is(err, store.ErrKeyTTL):
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
case err != nil:
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
k.UserID = owner.ID
|
||||
writeJSON(w, http.StatusCreated, KeyCreated{KeyView: keyView(k, path, owner.Email, time.Now()), Secret: secret})
|
||||
}
|
||||
|
||||
func (a *API) keyRevoke(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
id := r.PathValue("id")
|
||||
if _, err := a.tenancy.KeyByID(r.Context(), id); err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.RevokeKey(r.Context(), id, time.Now()); err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
// A key held in a gateway's cache keeps working until auth.cache_ttl.
|
||||
writeJSON(w, http.StatusOK, map[string]any{"id": id, "revoked": true})
|
||||
}
|
||||
|
||||
func (a *API) userView(ctx context.Context, u store.User) UserView {
|
||||
role := u.Role
|
||||
if u.IsAdmin {
|
||||
role = "admin"
|
||||
}
|
||||
v := UserView{ID: u.ID, Email: u.Email, Name: u.Name, State: string(u.State), Role: role,
|
||||
Memberships: []MembershipView{}, CreatedAt: u.CreatedAt, LastLoginAt: u.LastLoginAt}
|
||||
if ms, err := a.tenancy.MembershipsOf(ctx, u.ID); err == nil {
|
||||
for _, m := range ms {
|
||||
v.Memberships = append(v.Memberships, MembershipView{Target: string(m.Scope) + ":" + m.ScopeID,
|
||||
Path: m.Path, Role: m.Role, At: m.AddedAt, By: m.AddedBy})
|
||||
}
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func (a *API) userList(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
us, err := a.tenancy.Users(r.Context(), store.UserState(r.URL.Query().Get("state")))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []UserView{}
|
||||
for _, u := range us {
|
||||
out = append(out, a.userView(r.Context(), u))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) userGet(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
u, err := a.findUser(r.Context(), r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.userView(r.Context(), u))
|
||||
}
|
||||
|
||||
func (a *API) membership(add bool) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.tenancyReady(w) {
|
||||
return
|
||||
}
|
||||
var in MembershipRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
ctx := r.Context()
|
||||
u, err := a.findUser(ctx, r.PathValue("id"))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
scope, id, _, err := a.resolveScope(ctx, in.Target)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if add {
|
||||
role := in.Role
|
||||
if role == "" {
|
||||
role = store.RoleMember
|
||||
}
|
||||
if role != store.RoleMember && role != store.RoleManager {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("role %q: member or manager", role))
|
||||
return
|
||||
}
|
||||
if err := a.tenancy.AddMembership(ctx, u.ID, scope, id, role, actorOf(r), time.Now()); err != nil {
|
||||
writeError(w, tenancyStatus(err), err.Error())
|
||||
return
|
||||
}
|
||||
} else {
|
||||
// The person's keys in projects they no longer reach are
|
||||
// revoked with it, by the store.
|
||||
if _, err := a.tenancy.RemoveMembership(ctx, u.ID, scope, id, time.Now()); err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
err = errors.New(u.Email + " is not a member there")
|
||||
}
|
||||
writeError(w, http.StatusNotFound, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, a.userView(ctx, u))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestTheTenancyTreeKeysAndMembershipsAreAdministered(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
a, _ := setup(t, testModels())
|
||||
s, err := store.OpenSQLite(ctx, filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
a.SetQuotas(s)
|
||||
a.SetTenancy(s)
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
do := func(method, path, body string, out any) int {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest(method, path, strings.NewReader(body)))
|
||||
if out != nil && rec.Code < 300 {
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), out); err != nil {
|
||||
t.Fatalf("%s %s: %v (%s)", method, path, err, rec.Body.String())
|
||||
}
|
||||
}
|
||||
if rec.Code >= 300 {
|
||||
t.Logf("%s %s: %d %s", method, path, rec.Code, strings.TrimSpace(rec.Body.String()))
|
||||
}
|
||||
return rec.Code
|
||||
}
|
||||
var made TenancyItem
|
||||
if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, &made); c != http.StatusCreated {
|
||||
t.Fatalf("adding an organization: %d", c)
|
||||
}
|
||||
acme := made.ID
|
||||
if c := do("POST", "/api/v1/orgs", `{"name":"Acme"}`, nil); c != http.StatusConflict {
|
||||
t.Errorf("a second Acme: %d", c)
|
||||
}
|
||||
if c := do("POST", "/api/v1/teams", `{"name":"hr"}`, nil); c != http.StatusBadRequest {
|
||||
t.Errorf("a team in nothing: %d", c)
|
||||
}
|
||||
do("POST", "/api/v1/teams", `{"name":"hr","in":"Acme"}`, &made)
|
||||
hr := made.ID
|
||||
if c := do("POST", "/api/v1/projects", `{"name":"crm","in":"Acme/hr"}`, &made); c != http.StatusCreated || made.In != hr {
|
||||
t.Fatalf("adding a project: %d %v", c, made)
|
||||
}
|
||||
crm := made.ID
|
||||
do("POST", "/api/v1/orgs", `{"name":"Other"}`, &made)
|
||||
other := made.ID
|
||||
|
||||
// Renamed, and moved with what is in it.
|
||||
if c := do("PATCH", "/api/v1/projects/"+crm, `{"name":"crm2"}`, nil); c != http.StatusOK {
|
||||
t.Errorf("renaming: %d", c)
|
||||
}
|
||||
if c := do("PATCH", "/api/v1/teams/"+hr, `{"in":"Other"}`, nil); c != http.StatusOK {
|
||||
t.Errorf("moving: %d", c)
|
||||
}
|
||||
if c := do("PATCH", "/api/v1/orgs/"+acme, `{"in":"Other"}`, nil); c != http.StatusBadRequest {
|
||||
t.Errorf("moving an organization: %d", c)
|
||||
}
|
||||
var tree []OrgView
|
||||
do("GET", "/api/v1/orgs", "", &tree)
|
||||
if len(tree) != 2 || tree[1].ID != other || len(tree[1].Teams) != 1 ||
|
||||
tree[1].Teams[0].Projects[0].Path != "Other/hr/crm2" {
|
||||
t.Fatalf("the tree is %+v", tree)
|
||||
}
|
||||
|
||||
// A project's key: given once, listed without it.
|
||||
var key KeyCreated
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
|
||||
strings.NewReader(`{"name":"svc","project":"Other/hr/crm2","expires_in":"24h"}`)))
|
||||
if rec.Code != http.StatusCreated {
|
||||
t.Fatalf("issuing a key: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &key)
|
||||
if !strings.HasPrefix(key.Secret, "sk-llmgw-") || key.Project != crm || key.ExpiresAt.IsZero() {
|
||||
t.Errorf("the key issued: %+v", key)
|
||||
}
|
||||
if k, err := s.KeyByHash(ctx, store.HashKey(key.Secret)); err != nil || k.ID != key.ID {
|
||||
t.Errorf("the key does not authenticate: %v", err)
|
||||
}
|
||||
var keys []KeyView
|
||||
do("GET", "/api/v1/keys?project="+crm, "", &keys)
|
||||
if len(keys) != 1 || keys[0].ID != key.ID || keys[0].State != "active" || keys[0].ProjectPath != "Other/hr/crm2" {
|
||||
t.Errorf("the project's keys: %+v", keys)
|
||||
}
|
||||
raw := httptest.NewRecorder()
|
||||
mux.ServeHTTP(raw, httptest.NewRequest("GET", "/api/v1/keys", nil))
|
||||
if strings.Contains(raw.Body.String(), key.Secret) || strings.Contains(raw.Body.String(), "secret") {
|
||||
t.Error("a listing holds the key")
|
||||
}
|
||||
if c := do("POST", "/api/v1/keys", `{"name":"x","project":"Other/hr/crm2","expires_in":"soon"}`, nil); c != http.StatusBadRequest {
|
||||
t.Errorf("a lifetime that is not one: %d", c)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/projects/"+crm, "", nil); c != http.StatusConflict {
|
||||
t.Errorf("removing a project with a key: %d", c)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/keys/"+key.ID, "", nil); c != http.StatusOK {
|
||||
t.Errorf("revoking: %d", c)
|
||||
}
|
||||
var one KeyView
|
||||
do("GET", "/api/v1/keys/"+key.ID, "", &one)
|
||||
if one.State != "revoked" {
|
||||
t.Errorf("after revoking, %+v", one)
|
||||
}
|
||||
|
||||
// A person: a key in a project only as its member, revoked when they
|
||||
// leave it.
|
||||
if err := s.CreateUser(ctx, store.User{ID: "u1", Email: "ayse@example.com", PasswordHash: "x",
|
||||
State: store.UserActive, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if c := do("POST", "/api/v1/keys", `{"name":"mine","project":"`+crm+`","user":"ayse@example.com"}`, nil); c != http.StatusConflict {
|
||||
t.Errorf("a key where its person is no member: %d", c)
|
||||
}
|
||||
var u UserView
|
||||
if c := do("POST", "/api/v1/users/ayse@example.com/memberships", `{"target":"project:Other/hr/crm2"}`, &u); c != http.StatusOK ||
|
||||
len(u.Memberships) != 1 || u.Memberships[0].Target != "project:"+crm || u.Memberships[0].Role != "member" {
|
||||
t.Fatalf("joining: %d %+v", c, u)
|
||||
}
|
||||
var mine KeyCreated
|
||||
rec = httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/keys",
|
||||
strings.NewReader(`{"name":"mine","project":"`+crm+`","user":"u1"}`)))
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &mine)
|
||||
if rec.Code != http.StatusCreated || mine.User != "u1" {
|
||||
t.Fatalf("a member's key: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, &u); c != http.StatusOK || len(u.Memberships) != 0 {
|
||||
t.Errorf("leaving: %d %+v", c, u)
|
||||
}
|
||||
do("GET", "/api/v1/keys/"+mine.ID, "", &one)
|
||||
if one.State != "revoked" {
|
||||
t.Errorf("the key of a person who left: %+v", one)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/users/u1/memberships", `{"target":"project:`+crm+`"}`, nil); c != http.StatusNotFound {
|
||||
t.Errorf("leaving twice: %d", c)
|
||||
}
|
||||
var people []UserView
|
||||
do("GET", "/api/v1/users", "", &people)
|
||||
if len(people) != 1 || people[0].Email != "ayse@example.com" {
|
||||
t.Errorf("people: %+v", people)
|
||||
}
|
||||
|
||||
// Only what is empty is removed.
|
||||
if c := do("DELETE", "/api/v1/orgs/Acme", "", nil); c != http.StatusOK {
|
||||
t.Errorf("removing the empty Acme by name: %d", c)
|
||||
}
|
||||
if c := do("DELETE", "/api/v1/orgs/"+other, "", nil); c != http.StatusConflict {
|
||||
t.Errorf("removing an organization with a team: %d", c)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTenancyNeedsTheStore(t *testing.T) {
|
||||
a, _ := setup(t, testModels())
|
||||
if c := call(t, a, "GET", "/api/v1/orgs", "", nil); c != http.StatusNotImplemented {
|
||||
t.Errorf("without a store: %d", c)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,559 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bufio"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Training LoRA adapters (BACKLOG F4), administered: a job names the model
|
||||
// the adapter is for, the weights to train on, the node and how much of a
|
||||
// card it needs, and the pairs; the node trains when it can, and what it
|
||||
// made is kept here to download or publish.
|
||||
|
||||
// TrainStore is where jobs are kept.
|
||||
type TrainStore interface {
|
||||
CreateTrainJob(ctx context.Context, j store.TrainJob) error
|
||||
TrainJobs(ctx context.Context, limit int) ([]store.TrainJob, error)
|
||||
TrainJob(ctx context.Context, id string) (store.TrainJob, error)
|
||||
TrainArtifact(ctx context.Context, job, kind string) ([]byte, error)
|
||||
TrainArtifactKinds(ctx context.Context, job string) ([]string, error)
|
||||
DeleteTrainJob(ctx context.Context, id string) error
|
||||
}
|
||||
|
||||
// TrainAdmin is what the training endpoints need.
|
||||
type TrainAdmin struct {
|
||||
Store TrainStore
|
||||
Service *train.Service
|
||||
// Mirror keeps what a job made where the nodes fetch weights; nil
|
||||
// leaves publishing off.
|
||||
Mirror TrainMirror
|
||||
}
|
||||
|
||||
// TrainMirror is the model mirror, as publishing needs it.
|
||||
type TrainMirror interface {
|
||||
Upload(ctx context.Context, model, version, file string, body []byte, source string) (mirror.FetchResponse, error)
|
||||
}
|
||||
|
||||
// TrainPublishView is an adapter published: where the nodes fetch it from,
|
||||
// and what to do for replicas to load it.
|
||||
type TrainPublishView struct {
|
||||
Adapter string `json:"adapter"`
|
||||
Model string `json:"model"`
|
||||
Source string `json:"source"`
|
||||
// Owner is who sees it; empty is every caller of the model.
|
||||
Owner string `json:"owner,omitempty"`
|
||||
Note string `json:"note,omitempty"`
|
||||
}
|
||||
|
||||
// SetTrain enables the training endpoints.
|
||||
func (a *API) SetTrain(t *TrainAdmin) { a.train = t }
|
||||
|
||||
// SetTrainMirror gives publishing the mirror, which is set up after the
|
||||
// training endpoints are.
|
||||
func (a *API) SetTrainMirror(m TrainMirror) {
|
||||
if a.train != nil {
|
||||
a.train.Mirror = m
|
||||
}
|
||||
}
|
||||
|
||||
// TrainRequest queues a job.
|
||||
type TrainRequest struct {
|
||||
// Name is the adapter's: what requests will ask for it by.
|
||||
Name string `json:"name"`
|
||||
// Model is the catalog model the adapter is for.
|
||||
Model string `json:"model"`
|
||||
// BaseRepo and BaseRevision are the weights trained on: the model's own
|
||||
// on its registry, in safetensors (Hugging Face), not a GGUF.
|
||||
BaseRepo string `json:"base_repo"`
|
||||
BaseRevision string `json:"base_revision,omitempty"`
|
||||
// Node trains it; GPUMemoryMiB is what it needs of a card, which can be
|
||||
// what an engine there leaves free.
|
||||
Node string `json:"node"`
|
||||
GPUMemoryMiB int `json:"gpu_memory_mib"`
|
||||
// Owner is who the adapter will belong to: org:, team:, project: (by
|
||||
// path or ID), user: or key:. Only callers there see it. When the
|
||||
// pairs come from one knowledge base, it defaults to that collection's
|
||||
// owner; it must be able to read every collection they come from.
|
||||
Owner string `json:"owner,omitempty"`
|
||||
// Public makes the adapter every caller's its model allows. Refused for
|
||||
// pairs from a knowledge base.
|
||||
Public bool `json:"public,omitempty"`
|
||||
// Data is the pairs, NDJSON of {"q", "a"} (llmgw ctl kb dataset).
|
||||
Data string `json:"data"`
|
||||
// Params tune the trainer: rank, alpha, dropout, epochs, lr, batch,
|
||||
// max_len, seed.
|
||||
Params map[string]any `json:"params,omitempty"`
|
||||
// GGUF also makes the adapter in llama.cpp's format; when absent, it
|
||||
// does when the model is served by llama.cpp.
|
||||
GGUF *bool `json:"gguf,omitempty"`
|
||||
}
|
||||
|
||||
// TrainJobView is a job as listed.
|
||||
type TrainJobView struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Model string `json:"model"`
|
||||
BaseRepo string `json:"base_repo"`
|
||||
BaseRevision string `json:"base_revision,omitempty"`
|
||||
Node string `json:"node"`
|
||||
Owner string `json:"owner,omitempty"`
|
||||
GPUMemoryMiB int `json:"gpu_memory_mib"`
|
||||
Params string `json:"params,omitempty"`
|
||||
GGUF bool `json:"gguf"`
|
||||
Pairs int `json:"pairs"`
|
||||
State string `json:"state"`
|
||||
Step int `json:"step"`
|
||||
Steps int `json:"steps"`
|
||||
Loss float64 `json:"loss"`
|
||||
Message string `json:"message,omitempty"`
|
||||
PeakMiB int `json:"peak_mib"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
StartedAt time.Time `json:"started_at,omitzero"`
|
||||
FinishedAt time.Time `json:"finished_at,omitzero"`
|
||||
CreatedBy string `json:"created_by,omitempty"`
|
||||
// Artifacts are what it made: peft (a gzipped tar), gguf.
|
||||
Artifacts []string `json:"artifacts,omitempty"`
|
||||
}
|
||||
|
||||
func trainView(j store.TrainJob) TrainJobView {
|
||||
return TrainJobView{ID: j.ID, Name: j.Name, Model: j.Model, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision,
|
||||
Node: j.Node, Owner: j.Owner, GPUMemoryMiB: j.GPUMemoryMiB, Params: j.Params, GGUF: j.GGUF, Pairs: j.Pairs, State: j.State,
|
||||
Step: j.Step, Steps: j.Steps, Loss: j.Loss, Message: j.Message, PeakMiB: j.PeakMiB, CreatedAt: j.CreatedAt,
|
||||
StartedAt: j.StartedAt, FinishedAt: j.FinishedAt, CreatedBy: j.CreatedBy}
|
||||
}
|
||||
|
||||
func (a *API) registerTrain(mux Mux) {
|
||||
mux.HandleFunc("POST /api/v1/train/jobs", a.trainCreate)
|
||||
mux.HandleFunc("GET /api/v1/train/jobs", a.trainList)
|
||||
mux.HandleFunc("GET /api/v1/train/jobs/{id}", a.trainGet)
|
||||
mux.HandleFunc("POST /api/v1/train/jobs/{id}/cancel", a.trainCancel)
|
||||
mux.HandleFunc("DELETE /api/v1/train/jobs/{id}", a.trainDelete)
|
||||
mux.HandleFunc("GET /api/v1/train/jobs/{id}/artifacts/{kind}", a.trainArtifact)
|
||||
mux.HandleFunc("POST /api/v1/train/jobs/{id}/publish", a.trainPublish)
|
||||
}
|
||||
|
||||
func (a *API) trainReady(w http.ResponseWriter) bool {
|
||||
if a.train == nil {
|
||||
writeError(w, http.StatusNotImplemented, "training needs the store and the nodes' agents, which this control plane runs without")
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// trainMaxData bounds a job's pairs.
|
||||
const trainMaxData = 64 << 20
|
||||
|
||||
var adapterName = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,99}$`)
|
||||
|
||||
// trainParams are the trainer's settings, with what each may be.
|
||||
var trainParams = map[string][2]float64{
|
||||
"rank": {1, 256}, "alpha": {1, 1024}, "dropout": {0, 0.9}, "epochs": {1, 100},
|
||||
"lr": {1e-7, 1e-2}, "batch": {1, 128}, "max_len": {16, 32768}, "seed": {0, 1 << 31},
|
||||
}
|
||||
|
||||
func (a *API) trainCreate(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.trainReady(w) {
|
||||
return
|
||||
}
|
||||
var in TrainRequest
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, trainMaxData+1<<20)).Decode(&in); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
bad := func(format string, args ...any) {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf(format, args...))
|
||||
}
|
||||
if !adapterName.MatchString(in.Name) {
|
||||
bad("an adapter's name is letters, digits, . _ and -, up to 100: %q is not", in.Name)
|
||||
return
|
||||
}
|
||||
m, ok := a.ctl.Catalog().Resolve(in.Model)
|
||||
if !ok {
|
||||
bad("no model %s in the catalog", in.Model)
|
||||
return
|
||||
}
|
||||
if _, _, _, taken := a.ctl.Catalog().ResolveAdapter(in.Name); taken || in.Name == m.Name {
|
||||
bad("%s is already a model's or an adapter's name", in.Name)
|
||||
return
|
||||
}
|
||||
if strings.Count(in.BaseRepo, "/") != 1 || strings.ContainsAny(in.BaseRepo, " :") {
|
||||
bad("base_repo is the weights' registry repository, owner/name: %q is not", in.BaseRepo)
|
||||
return
|
||||
}
|
||||
if _, ok := a.ctl.Inventory().Get(in.Node); !ok {
|
||||
bad("no node %s", in.Node)
|
||||
return
|
||||
}
|
||||
if in.GPUMemoryMiB <= 0 {
|
||||
bad("gpu_memory_mib is what the job needs of a card, in MiB")
|
||||
return
|
||||
}
|
||||
pairs, colls, err := countPairs(in.Data)
|
||||
if err != nil {
|
||||
bad("%v", err)
|
||||
return
|
||||
}
|
||||
owner, err := a.trainOwner(r.Context(), in, colls)
|
||||
if err != nil {
|
||||
bad("%v", err)
|
||||
return
|
||||
}
|
||||
for k, v := range in.Params {
|
||||
lim, known := trainParams[k]
|
||||
f, isNum := v.(float64)
|
||||
if !known || !isNum || f < lim[0] || f > lim[1] {
|
||||
bad("params.%s: the trainer takes rank, alpha, dropout, epochs, lr, batch, max_len and seed, each a number in its range", k)
|
||||
return
|
||||
}
|
||||
}
|
||||
params := ""
|
||||
if len(in.Params) > 0 {
|
||||
b, _ := json.Marshal(in.Params)
|
||||
params = string(b)
|
||||
}
|
||||
gguf := false
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == "llamacpp" {
|
||||
gguf = true
|
||||
}
|
||||
}
|
||||
if in.GGUF != nil {
|
||||
gguf = *in.GGUF
|
||||
}
|
||||
j := store.TrainJob{ID: store.NewID("train-"), Name: in.Name, Model: m.Name, BaseRepo: in.BaseRepo,
|
||||
BaseRevision: in.BaseRevision, Node: in.Node, Owner: owner, GPUMemoryMiB: in.GPUMemoryMiB, Params: params, GGUF: gguf,
|
||||
Data: []byte(in.Data), Pairs: pairs, CreatedBy: actorOf(r)}
|
||||
if err := a.train.Store.CreateTrainJob(r.Context(), j); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
// At once rather than at the next look, when the node is here.
|
||||
go a.train.Service.Dispatch(context.WithoutCancel(r.Context()))
|
||||
j.State = store.TrainQueued
|
||||
writeJSON(w, http.StatusCreated, trainView(j))
|
||||
}
|
||||
|
||||
// trainOwner is who a job's adapter will belong to, as scope:ID: one that
|
||||
// may read every knowledge base the pairs come from. Pairs from a knowledge
|
||||
// base never make an adapter everyone's.
|
||||
func (a *API) trainOwner(ctx context.Context, in TrainRequest, colls []string) (string, error) {
|
||||
if in.Public {
|
||||
if len(colls) > 0 {
|
||||
return "", errors.New("these pairs come from a knowledge base, and an adapter trained on them " +
|
||||
"carries some of what it says: it belongs to a part of the tree that may read it, not to everyone")
|
||||
}
|
||||
if in.Owner != "" {
|
||||
return "", errors.New("an adapter is public or has an owner, not both")
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
if len(colls) > 0 && a.kb == nil {
|
||||
return "", errors.New("the pairs name knowledge bases, which this control plane cannot read")
|
||||
}
|
||||
var found []store.KBCollection
|
||||
for _, id := range colls {
|
||||
c, err := a.kb.Store.Collection(ctx, id)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("the pairs come from collection %s, which is not here: %w", id, err)
|
||||
}
|
||||
found = append(found, c)
|
||||
}
|
||||
if in.Owner == "" {
|
||||
if len(found) == 1 {
|
||||
return string(found[0].OwnerScope) + ":" + found[0].OwnerID, nil
|
||||
}
|
||||
return "", errors.New("name the adapter's owner (org:, team:, project:, user: or key:), or make it public " +
|
||||
"when its pairs are not a knowledge base's")
|
||||
}
|
||||
scope, id, err := a.kbTarget(ctx, in.Owner)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
for _, c := range found {
|
||||
e, err := a.kb.Store.EffectiveFor(ctx, scope, id, c.OrgID)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("%s: %w", in.Owner, err)
|
||||
}
|
||||
readable, err := a.kb.Store.CollectionsFor(ctx, e, false)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if !slices.ContainsFunc(readable, func(x store.KBCollection) bool { return x.ID == c.ID }) {
|
||||
return "", fmt.Errorf("%s may not read collection %s, which the pairs come from: "+
|
||||
"an adapter trained on it would tell its callers what it says", in.Owner, c.Name)
|
||||
}
|
||||
}
|
||||
return string(scope) + ":" + id, nil
|
||||
}
|
||||
|
||||
// countPairs checks a job's data: NDJSON of {"q", "a"}, at least one, and
|
||||
// says which knowledge bases the pairs come from.
|
||||
func countPairs(data string) (int, []string, error) {
|
||||
if len(data) > trainMaxData {
|
||||
return 0, nil, fmt.Errorf("the data is over %d MiB", trainMaxData>>20)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
var colls []string
|
||||
n := 0
|
||||
sc := bufio.NewScanner(strings.NewReader(data))
|
||||
sc.Buffer(make([]byte, 64<<10), 1<<20)
|
||||
line := 0
|
||||
for sc.Scan() {
|
||||
line++
|
||||
b := bytes.TrimSpace(sc.Bytes())
|
||||
if len(b) == 0 {
|
||||
continue
|
||||
}
|
||||
var p struct{ Q, A, Collection string }
|
||||
if err := json.Unmarshal(b, &p); err != nil {
|
||||
return 0, nil, fmt.Errorf("data line %d is not JSON: %v", line, err)
|
||||
}
|
||||
if strings.TrimSpace(p.Q) == "" || strings.TrimSpace(p.A) == "" {
|
||||
return 0, nil, fmt.Errorf("data line %d has no q or no a", line)
|
||||
}
|
||||
if p.Collection != "" && !seen[p.Collection] {
|
||||
seen[p.Collection] = true
|
||||
colls = append(colls, p.Collection)
|
||||
}
|
||||
n++
|
||||
}
|
||||
if err := sc.Err(); err != nil {
|
||||
return 0, nil, err
|
||||
}
|
||||
if n == 0 {
|
||||
return 0, nil, errors.New("the data holds no pairs: NDJSON of {\"q\", \"a\"}, as llmgw ctl kb dataset writes")
|
||||
}
|
||||
return n, colls, nil
|
||||
}
|
||||
|
||||
func (a *API) trainList(w http.ResponseWriter, r *http.Request) {
|
||||
if !a.trainReady(w) {
|
||||
return
|
||||
}
|
||||
limit := 50
|
||||
if n, err := strconv.Atoi(r.URL.Query().Get("limit")); err == nil && n > 0 && n <= 500 {
|
||||
limit = n
|
||||
}
|
||||
jobs, err := a.train.Store.TrainJobs(r.Context(), limit)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
out := []TrainJobView{}
|
||||
for _, j := range jobs {
|
||||
out = append(out, trainView(j))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
func (a *API) trainJob(w http.ResponseWriter, r *http.Request) (store.TrainJob, bool) {
|
||||
if !a.trainReady(w) {
|
||||
return store.TrainJob{}, false
|
||||
}
|
||||
j, err := a.train.Store.TrainJob(r.Context(), r.PathValue("id"))
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "no training job "+r.PathValue("id"))
|
||||
return j, false
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return j, false
|
||||
}
|
||||
return j, true
|
||||
}
|
||||
|
||||
func (a *API) trainGet(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
v := trainView(j)
|
||||
v.Artifacts, _ = a.train.Store.TrainArtifactKinds(r.Context(), j.ID)
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
func (a *API) trainCancel(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := a.train.Service.Cancel(r.Context(), j.ID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
j, _ = a.train.Store.TrainJob(r.Context(), j.ID)
|
||||
writeJSON(w, http.StatusOK, trainView(j))
|
||||
}
|
||||
|
||||
func (a *API) trainDelete(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if !store.TrainOver(j.State) {
|
||||
writeError(w, http.StatusConflict, "the job has not ended: cancel it first")
|
||||
return
|
||||
}
|
||||
if err := a.train.Store.DeleteTrainJob(r.Context(), j.ID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"id": j.ID, "deleted": true})
|
||||
}
|
||||
|
||||
func (a *API) trainArtifact(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
kind := r.PathValue("kind")
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, kind)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "the job made no "+kind)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
name := j.Name + ".tar.gz"
|
||||
w.Header().Set("Content-Type", "application/gzip")
|
||||
if kind == "gguf" {
|
||||
name = j.Name + ".gguf"
|
||||
w.Header().Set("Content-Type", "application/octet-stream")
|
||||
}
|
||||
w.Header().Set("Content-Disposition", `attachment; filename="`+name+`"`)
|
||||
w.Header().Set("Content-Length", strconv.Itoa(len(b)))
|
||||
_, _ = w.Write(b)
|
||||
}
|
||||
|
||||
// trainPublish puts what a job made on the mirror and makes it the model's
|
||||
// adapter: the GGUF for a model llama.cpp serves, the PEFT directory for
|
||||
// one vLLM serves.
|
||||
func (a *API) trainPublish(w http.ResponseWriter, r *http.Request) {
|
||||
j, ok := a.trainJob(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if a.train.Mirror == nil {
|
||||
writeError(w, http.StatusNotImplemented, "publishing needs the model mirror (mirror.api in the settings); "+
|
||||
"llmgw ctl train download, then llmgw mirror import there, does it by hand")
|
||||
return
|
||||
}
|
||||
if a.models == nil {
|
||||
writeError(w, http.StatusNotImplemented, "this control plane has no store for the catalog")
|
||||
return
|
||||
}
|
||||
if j.State != store.TrainDone {
|
||||
writeError(w, http.StatusConflict, "the job has not finished: it is "+j.State)
|
||||
return
|
||||
}
|
||||
m, found := a.ctl.Catalog().Resolve(j.Model)
|
||||
if !found {
|
||||
writeError(w, http.StatusConflict, "the model "+j.Model+" is no longer in the catalog")
|
||||
return
|
||||
}
|
||||
for _, ad := range m.Adapters {
|
||||
if ad.Name == j.Name {
|
||||
writeError(w, http.StatusConflict, j.Name+" is already "+m.Name+"'s adapter; train under another name")
|
||||
return
|
||||
}
|
||||
}
|
||||
llama := slices.ContainsFunc(m.Profiles, func(p catalog.Profile) bool { return p.Engine == "llamacpp" })
|
||||
source := "trained by llmgw job " + j.ID + " on " + j.Node + " from " + j.BaseRepo
|
||||
var url string
|
||||
if llama {
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "gguf")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusConflict, "the job made no GGUF, which llama.cpp needs; train again without --no-gguf")
|
||||
return
|
||||
}
|
||||
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, j.Name+".gguf", b, source)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "keeping it on the mirror: "+err.Error())
|
||||
return
|
||||
}
|
||||
url = res.URL
|
||||
} else {
|
||||
b, err := a.train.Store.TrainArtifact(r.Context(), j.ID, "peft")
|
||||
if err != nil {
|
||||
writeError(w, http.StatusConflict, "the job made no PEFT adapter")
|
||||
return
|
||||
}
|
||||
files, err := untarAll(b)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, err.Error())
|
||||
return
|
||||
}
|
||||
for name, data := range files {
|
||||
res, err := a.train.Mirror.Upload(r.Context(), j.Name, j.ID, name, data, source)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "keeping "+name+" on the mirror: "+err.Error())
|
||||
return
|
||||
}
|
||||
url = res.URL[:strings.LastIndex(res.URL, "/")+1]
|
||||
}
|
||||
}
|
||||
ad := catalog.Adapter{Name: j.Name, Source: url, Owner: j.Owner}
|
||||
if err := checkAdapters(r.Context(), m, []catalog.Adapter{ad}); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
if _, err := a.models.SetAdapters(r.Context(), m.Name, append(slices.Clone(m.Adapters), ad), actorOf(r)); err != nil {
|
||||
writeError(w, http.StatusBadRequest, err.Error())
|
||||
return
|
||||
}
|
||||
v := TrainPublishView{Adapter: j.Name, Model: m.Name, Source: url, Owner: j.Owner}
|
||||
if llama {
|
||||
v.Note = "llama.cpp loads adapters when a replica starts: restart " + m.Name + "'s replicas (llmgw ctl restart " +
|
||||
m.Name + ") and ask for model " + j.Name
|
||||
} else {
|
||||
v.Note = "ask for model " + j.Name + "; a replica loads it on the first request"
|
||||
}
|
||||
writeJSON(w, http.StatusOK, v)
|
||||
}
|
||||
|
||||
// untarAll reads a gzipped tar of a PEFT directory's files.
|
||||
func untarAll(b []byte) (map[string][]byte, error) {
|
||||
gz, err := gzip.NewReader(bytes.NewReader(b))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tr := tar.NewReader(gz)
|
||||
out := map[string][]byte{}
|
||||
for {
|
||||
h, err := tr.Next()
|
||||
if errors.Is(err, io.EOF) {
|
||||
return out, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if h.Typeflag != tar.TypeReg || strings.ContainsAny(h.Name, "/\\") {
|
||||
continue
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(tr, 4<<30))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[h.Name] = data
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
package adminapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/train"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type noAgents struct{}
|
||||
|
||||
func (noAgents) Send(string, *llmgwv1.ControlMessage) error { return nil }
|
||||
func (noAgents) Connected(string) bool { return false }
|
||||
|
||||
func TestATrainingJobIsCheckedBeforeItIsQueued(t *testing.T) {
|
||||
models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
|
||||
a, _ := setup(t, models, gpuNode("cuda01", 1))
|
||||
s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
post := func(body string) (int, string) {
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
|
||||
return rec.Code, rec.Body.String()
|
||||
}
|
||||
good := map[string]any{"name": "tone", "model": "small", "base_repo": "Qwen/Qwen2.5-0.5B-Instruct", "node": "cuda01",
|
||||
"gpu_memory_mib": 2300, "data": "{\"q\":\"a\",\"a\":\"b\"}\n{\"q\":\"c\",\"a\":\"d\"}\n", "params": map[string]any{"epochs": 3},
|
||||
"public": true}
|
||||
with := func(k string, v any) string {
|
||||
m := map[string]any{}
|
||||
for kk, vv := range good {
|
||||
m[kk] = vv
|
||||
}
|
||||
m[k] = v
|
||||
b, _ := json.Marshal(m)
|
||||
return string(b)
|
||||
}
|
||||
for _, c := range []struct {
|
||||
field string
|
||||
value any
|
||||
says string
|
||||
}{
|
||||
{"name", "bad name", "letters, digits"},
|
||||
{"model", "nope", "no model nope"},
|
||||
{"name", "small", "already a model"},
|
||||
{"base_repo", "Qwen", "owner/name"},
|
||||
{"node", "gone", "no node gone"},
|
||||
{"data", "{\"q\":\"a\"}\n", "no q or no a"},
|
||||
{"data", "", "no pairs"},
|
||||
{"params", map[string]any{"epochs": 0}, "params.epochs"},
|
||||
{"params", map[string]any{"momentum": 0.9}, "params.momentum"},
|
||||
{"public", false, "name the adapter's owner"},
|
||||
{"data", "{\"q\":\"a\",\"a\":\"b\",\"collection\":\"kb_aaaaaaaaaaaaaaaa\"}\n", "from a knowledge base"},
|
||||
} {
|
||||
if code, body := post(with(c.field, c.value)); code != http.StatusBadRequest || !strings.Contains(body, c.says) {
|
||||
t.Errorf("%s=%v: %d %s", c.field, c.value, code, body)
|
||||
}
|
||||
}
|
||||
code, body := post(with("name", "tone"))
|
||||
var v TrainJobView
|
||||
_ = json.Unmarshal([]byte(body), &v)
|
||||
if code != http.StatusCreated || v.Pairs != 2 || !v.GGUF || v.State != "queued" || v.Params != `{"epochs":3}` {
|
||||
t.Fatalf("%d %s", code, body)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("DELETE", "/api/v1/train/jobs/"+v.ID, nil))
|
||||
if rec.Code != http.StatusConflict {
|
||||
t.Errorf("deleting a queued job: %d", rec.Code)
|
||||
}
|
||||
rec = httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/v1/train/jobs/"+v.ID+"/artifacts/peft", nil))
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Errorf("an artifact not made: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAdapterTrainedOnAKnowledgeBaseBelongsToWhoMayReadIt(t *testing.T) {
|
||||
dsn := os.Getenv("LLMGW_TEST_POSTGRES")
|
||||
if dsn == "" {
|
||||
t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
|
||||
}
|
||||
ctx := context.Background()
|
||||
s, err := store.OpenPostgres(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.Close()
|
||||
if s.KnowledgeAvailable() != nil {
|
||||
t.Skip("no pgvector")
|
||||
}
|
||||
models := append(testModels(), catalog.Model{Name: "small", Source: "/s",
|
||||
Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}})
|
||||
a, _ := setup(t, models, gpuNode("cuda01", 1))
|
||||
a.SetQuotas(s)
|
||||
a.SetKB(&KBAdmin{Store: s})
|
||||
a.SetTrain(&TrainAdmin{Store: s, Service: train.New(s, noAgents{}, nil)})
|
||||
sfx := strings.ToLower(store.NewID(""))
|
||||
org, _ := s.CreateOrg(ctx, "Tr"+sfx)
|
||||
hr, _ := s.CreateTeam(ctx, org, "hr")
|
||||
_, _ = s.CreateProject(ctx, hr, "docs")
|
||||
sales, _ := s.CreateTeam(ctx, org, "sales")
|
||||
_, _ = s.CreateProject(ctx, sales, "crm")
|
||||
c, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeTeam, OwnerID: hr, Name: "p", EmbedModel: "bge-m3", Dims: 3})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
mux := http.NewServeMux()
|
||||
a.Register(mux)
|
||||
post := func(extra string) (int, TrainJobView, string) {
|
||||
body := `{"name":"tone-` + store.NewID("") + `","model":"small","base_repo":"Qwen/Q","node":"cuda01","gpu_memory_mib":2000,` +
|
||||
`"data":"{\"q\":\"a\",\"a\":\"b\",\"collection\":\"` + c.ID + `\"}\n"` + extra + `}`
|
||||
rec := httptest.NewRecorder()
|
||||
mux.ServeHTTP(rec, httptest.NewRequest("POST", "/api/v1/train/jobs", strings.NewReader(body)))
|
||||
var v TrainJobView
|
||||
_ = json.Unmarshal(rec.Body.Bytes(), &v)
|
||||
return rec.Code, v, rec.Body.String()
|
||||
}
|
||||
if code, v, body := post(""); code != http.StatusCreated || v.Owner != "team:"+hr {
|
||||
t.Errorf("by default: %d %s", code, body)
|
||||
}
|
||||
if code, v, body := post(`,"owner":"project:Tr` + sfx + `/hr/docs"`); code != http.StatusCreated || !strings.HasPrefix(v.Owner, "project:") {
|
||||
t.Errorf("a project of the team: %d %s", code, body)
|
||||
}
|
||||
if code, _, body := post(`,"owner":"project:Tr` + sfx + `/sales/crm"`); code != http.StatusBadRequest || !strings.Contains(body, "may not read") {
|
||||
t.Errorf("a project that cannot read it: %d %s", code, body)
|
||||
}
|
||||
if code, _, body := post(`,"public":true`); code != http.StatusBadRequest || !strings.Contains(body, "not to everyone") {
|
||||
t.Errorf("public: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
@@ -82,6 +82,8 @@ type Service struct {
|
||||
// agentSettings is what each node's agent is told to run on; see
|
||||
// SetAgentSettings.
|
||||
agentSettings func(node string) string
|
||||
// train is told how training jobs go; see SetTrain.
|
||||
train TrainHandler
|
||||
|
||||
// logWaits are the log requests awaiting a reply, by request id.
|
||||
logMu sync.Mutex
|
||||
@@ -368,6 +370,9 @@ func (s *Service) readLoop(ctx context.Context,
|
||||
msg, err := stream.Receive()
|
||||
if err != nil {
|
||||
c.close()
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnNodeGone(nodeID)
|
||||
}
|
||||
if ctx.Err() != nil || errors.Is(err, io.EOF) {
|
||||
s.log.Info("agent disconnected", "node", nodeID)
|
||||
return
|
||||
@@ -505,6 +510,16 @@ func (s *Service) handle(nodeID string, msg *llmgwv1.AgentMessage) {
|
||||
ch <- m.Upgrade
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainStatus:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainStatus(nodeID, m.TrainStatus)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_TrainArtifact:
|
||||
if h := s.trainHandler(); h != nil {
|
||||
h.OnTrainArtifact(nodeID, m.TrainArtifact)
|
||||
}
|
||||
|
||||
case *llmgwv1.AgentMessage_Logs:
|
||||
s.logMu.Lock()
|
||||
ch, ok := s.logWaits[m.Logs.GetRequestId()]
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
package agentsvc
|
||||
|
||||
import (
|
||||
"slices"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
)
|
||||
|
||||
// TrainHandler is told how training jobs on the nodes go (BACKLOG F4).
|
||||
type TrainHandler interface {
|
||||
OnTrainStatus(node string, st *llmgwv1.TrainStatus)
|
||||
OnTrainArtifact(node string, a *llmgwv1.TrainArtifact)
|
||||
// OnNodeGone is a node's stream ending: a job running there ended with
|
||||
// it.
|
||||
OnNodeGone(node string)
|
||||
}
|
||||
|
||||
// SetTrain gives the service what training jobs' messages go to.
|
||||
func (s *Service) SetTrain(h TrainHandler) {
|
||||
s.mu.Lock()
|
||||
s.train = h
|
||||
s.mu.Unlock()
|
||||
}
|
||||
|
||||
func (s *Service) trainHandler() TrainHandler {
|
||||
s.mu.RLock()
|
||||
defer s.mu.RUnlock()
|
||||
return s.train
|
||||
}
|
||||
|
||||
// Send gives a connected node's agent a command.
|
||||
func (s *Service) Send(node string, msg *llmgwv1.ControlMessage) error { return s.send(node, msg) }
|
||||
|
||||
// Connected is whether a node's agent holds a stream to this control plane.
|
||||
func (s *Service) Connected(node string) bool { return slices.Contains(s.ConnectedNodes(), node) }
|
||||
@@ -8,12 +8,14 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"path"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/facility"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/placement"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
@@ -321,6 +323,9 @@ type Model struct {
|
||||
// prompts leave the cluster, so which keys may name it is worth an
|
||||
// allowlist.
|
||||
External *External `json:"external,omitempty" koanf:"external"`
|
||||
// ComfyUI is how a model whose profiles run engine comfyui makes its
|
||||
// pictures: the workflow, and the file it loads.
|
||||
ComfyUI *ComfyUI `json:"comfyui,omitempty" koanf:"comfyui"`
|
||||
// Adapters are LoRA adapters served on this model's replicas, each asked
|
||||
// for by its own name as though it were a model. Every replica loads
|
||||
// every adapter, applying one only to a request that names it — unless
|
||||
@@ -363,6 +368,13 @@ type Model struct {
|
||||
// Capabilities is what this model can do beyond plain completion.
|
||||
Capabilities Capabilities `json:"capabilities,omitempty" koanf:"capabilities"`
|
||||
|
||||
// QueryPrefix is put before a question an embedding model embeds for a
|
||||
// knowledge-base search; documents are embedded without it. Models
|
||||
// trained with an instruction on the query side want it, Qwen3-Embedding
|
||||
// "Instruct: Given a web search query, retrieve relevant passages that
|
||||
// answer the query\nQuery: " for one.
|
||||
QueryPrefix string `json:"query_prefix,omitempty" koanf:"query_prefix"`
|
||||
|
||||
// Role "guard" is a model that judges the gateway's traffic for the
|
||||
// guardrails rather than serving clients: hidden from /v1/models and the
|
||||
// chat page, refused to any key that names it, critical by default, and
|
||||
@@ -450,6 +462,98 @@ type Capabilities struct {
|
||||
Tools bool `json:"tools,omitempty" koanf:"tools"`
|
||||
Vision bool `json:"vision,omitempty" koanf:"vision"`
|
||||
Embeddings bool `json:"embeddings,omitempty" koanf:"embeddings"`
|
||||
// Images is a model that makes pictures, asked for at
|
||||
// /v1/images/generations. Implied by an external model whose API is
|
||||
// ComfyUI's.
|
||||
Images bool `json:"images,omitempty" koanf:"images"`
|
||||
}
|
||||
|
||||
// MakesImages reports whether a model answers /v1/images/generations.
|
||||
func (m Model) MakesImages() bool {
|
||||
return m.Capabilities.Images || m.ServesComfyUI()
|
||||
}
|
||||
|
||||
// EngineComfyUI is the engine that runs ComfyUI on a node.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// ServesComfyUI reports whether a model is made by ComfyUI, on the
|
||||
// cluster's own nodes or on a server outside it.
|
||||
func (m Model) ServesComfyUI() bool {
|
||||
if m.External != nil {
|
||||
return m.External.API == APIComfyUI
|
||||
}
|
||||
return len(m.Profiles) > 0 && m.Profiles[0].Engine == EngineComfyUI
|
||||
}
|
||||
|
||||
// ComfyUI is a ComfyUI model's workflow and checkpoint.
|
||||
type ComfyUI struct {
|
||||
// Workflow is a built-in's name (flux-schnell, the default, or
|
||||
// flux-dev) or a JSON file on the control plane, in ComfyUI's API
|
||||
// format.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
// Checkpoint is the file the workflow's {{checkpoint}} names: the
|
||||
// source's own name unless set, which a source that is a set of files
|
||||
// must.
|
||||
Checkpoint string `json:"checkpoint,omitempty" koanf:"checkpoint"`
|
||||
}
|
||||
|
||||
// ComfyWorkflow is what a ComfyUI model's replicas are sent: the workflow
|
||||
// as the catalog names it, and the checkpoint it loads.
|
||||
func (m Model) ComfyWorkflow() (workflow, checkpoint string) {
|
||||
if m.External != nil {
|
||||
return m.External.Workflow, m.External.Model
|
||||
}
|
||||
if m.ComfyUI != nil {
|
||||
workflow, checkpoint = m.ComfyUI.Workflow, m.ComfyUI.Checkpoint
|
||||
}
|
||||
if checkpoint == "" && !strings.HasSuffix(m.Source, "/") {
|
||||
checkpoint = path.Base(strings.TrimPrefix(m.Source, "hf:"))
|
||||
}
|
||||
return workflow, checkpoint
|
||||
}
|
||||
|
||||
// checkWorkflow refuses a workflow that is neither a built-in nor a file.
|
||||
func checkWorkflow(model, w string) error {
|
||||
// A file is read where the router runs; a name is checked here.
|
||||
if w != "" && !strings.ContainsAny(w, "/.") && !slices.Contains(comfy.Builtins(), w) {
|
||||
return fmt.Errorf("model %q: no built-in workflow %q (there are %s; or name a JSON file)",
|
||||
model, w, strings.Join(comfy.Builtins(), ", "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m Model) validateComfyUI() error {
|
||||
comfyProfiles := 0
|
||||
for _, p := range m.Profiles {
|
||||
if p.Engine == EngineComfyUI {
|
||||
comfyProfiles++
|
||||
}
|
||||
}
|
||||
if comfyProfiles == 0 {
|
||||
if m.ComfyUI != nil {
|
||||
return fmt.Errorf("model %q: comfyui: is for a model whose profiles run engine comfyui", m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if comfyProfiles != len(m.Profiles) {
|
||||
return fmt.Errorf("model %q: every profile runs engine comfyui or none does: "+
|
||||
"a request for pictures cannot be answered by an engine that writes text", m.Name)
|
||||
}
|
||||
if len(m.Adapters) > 0 {
|
||||
return fmt.Errorf("model %q: ComfyUI serves no adapters", m.Name)
|
||||
}
|
||||
var w string
|
||||
if m.ComfyUI != nil {
|
||||
w = m.ComfyUI.Workflow
|
||||
}
|
||||
if err := checkWorkflow(m.Name, w); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, ckpt := m.ComfyWorkflow(); ckpt == "" {
|
||||
return fmt.Errorf("model %q: its source is a set of files, so comfyui.checkpoint must name the one the workflow loads",
|
||||
m.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Catalog is the set of models the cluster should serve.
|
||||
@@ -690,6 +794,9 @@ func (m Model) validate() error {
|
||||
} else if len(m.Profiles) == 0 {
|
||||
return fmt.Errorf("model %q: at least one profile is required", m.Name)
|
||||
}
|
||||
if err := m.validateComfyUI(); err != nil {
|
||||
return err
|
||||
}
|
||||
if m.Owner != "" && !validOwner(m.Owner) {
|
||||
return fmt.Errorf("model %q: owner %q is not org:ID, team:ID or project:ID", m.Name, m.Owner)
|
||||
}
|
||||
@@ -1094,8 +1201,20 @@ type External struct {
|
||||
// than one written by hand. Such a model is named PEER/MODEL, comes and
|
||||
// goes with the peer's own list, and is sent only while the peer is up.
|
||||
Peer string `json:"peer,omitempty" koanf:"peer"`
|
||||
// API is what the provider speaks: OpenAI's, unless this says comfyui —
|
||||
// a ComfyUI server, asked for pictures at /v1/images/generations and
|
||||
// sent the workflow below with the request's prompt in it. Model is then
|
||||
// the checkpoint file the workflow loads, as ComfyUI lists it.
|
||||
API string `json:"api,omitempty" koanf:"api"`
|
||||
// Workflow is the ComfyUI workflow, in its API format: a built-in's name
|
||||
// (flux-schnell, the default, or flux-dev) or a JSON file on the control
|
||||
// plane.
|
||||
Workflow string `json:"workflow,omitempty" koanf:"workflow"`
|
||||
}
|
||||
|
||||
// APIComfyUI is External.API for a ComfyUI server.
|
||||
const APIComfyUI = "comfyui"
|
||||
|
||||
// IsPeer reports whether a model is offered by a federated cluster.
|
||||
func (m Model) IsPeer() bool { return m.External != nil && m.External.Peer != "" }
|
||||
|
||||
@@ -1110,6 +1229,21 @@ func (e *External) validate(model string, profiles int) error {
|
||||
if strings.TrimSpace(e.Model) == "" {
|
||||
return fmt.Errorf("model %q: external model needs the name the provider knows it by", model)
|
||||
}
|
||||
switch e.API {
|
||||
case "", "openai":
|
||||
if e.Workflow != "" {
|
||||
return fmt.Errorf("model %q: a workflow is for api: comfyui", model)
|
||||
}
|
||||
case APIComfyUI:
|
||||
if e.Peer != "" {
|
||||
return fmt.Errorf("model %q: a federated model speaks OpenAI's API, not ComfyUI's", model)
|
||||
}
|
||||
if err := checkWorkflow(model, e.Workflow); err != nil {
|
||||
return err
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("model %q: external api %q is not openai or comfyui", model, e.API)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1127,6 +1261,27 @@ type Adapter struct {
|
||||
Source string `json:"source" koanf:"source"`
|
||||
// Scale is how strongly it is applied; zero takes 1.
|
||||
Scale float64 `json:"scale,omitempty" koanf:"scale"`
|
||||
// Owner is the part of the tenancy tree the adapter belongs to —
|
||||
// org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in
|
||||
// it see it: to anyone else it is a model that does not exist. An
|
||||
// adapter trained on an organisation's documents carries some of what
|
||||
// they say in its weights. Empty is every caller the model allows.
|
||||
Owner string `json:"owner,omitempty" koanf:"owner"`
|
||||
}
|
||||
|
||||
// ValidAdapterOwner checks an adapter's owner is scope:ID.
|
||||
func ValidAdapterOwner(owner string) error {
|
||||
if owner == "" {
|
||||
return nil
|
||||
}
|
||||
scope, id, ok := strings.Cut(owner, ":")
|
||||
switch {
|
||||
case !ok || id == "":
|
||||
return fmt.Errorf("an adapter's owner is scope:ID, %q is not", owner)
|
||||
case scope != "org" && scope != "team" && scope != "project" && scope != "user" && scope != "key":
|
||||
return fmt.Errorf("an adapter's owner is an org, team, project, user or key, not %q", scope)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// EffectiveScale is the scale an adapter is applied at.
|
||||
@@ -1175,6 +1330,9 @@ func validateAdapters(m Model) error {
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, a := range m.Adapters {
|
||||
if err := ValidAdapterOwner(a.Owner); err != nil {
|
||||
return fmt.Errorf("model %q adapter %q: %w", m.Name, a.Name, err)
|
||||
}
|
||||
switch {
|
||||
case a.Name == "" || a.Source == "":
|
||||
return fmt.Errorf("model %q: an adapter needs a name and a source", m.Name)
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func comfyModel(source string, c *ComfyUI, profiles ...Profile) Model {
|
||||
if len(profiles) == 0 {
|
||||
profiles = []Profile{{Name: "gpu", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1}}
|
||||
}
|
||||
return Model{Name: "flux", Source: source, ComfyUI: c, Profiles: profiles}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelMakesPicturesFromItsSourcesFile(t *testing.T) {
|
||||
cat, err := New([]Model{comfyModel("hf:Comfy-Org/flux1-schnell/flux1-schnell-fp8.safetensors", nil)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
m, _ := cat.Resolve("flux")
|
||||
if !m.MakesImages() || !m.ServesComfyUI() {
|
||||
t.Error("a ComfyUI model does not say it makes pictures")
|
||||
}
|
||||
if w, ckpt := m.ComfyWorkflow(); w != "" || ckpt != "flux1-schnell-fp8.safetensors" {
|
||||
t.Errorf("workflow %q, checkpoint %q", w, ckpt)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsRefusedWhatItCannotDo(t *testing.T) {
|
||||
for name, c := range map[string]struct {
|
||||
m Model
|
||||
want string
|
||||
}{
|
||||
"a set without a checkpoint": {comfyModel("https://mirror/flux-dev/v1/", nil), "comfyui.checkpoint"},
|
||||
"an unknown workflow": {comfyModel("/w/f.safetensors", &ComfyUI{Workflow: "sdxl"}), "no built-in workflow"},
|
||||
"mixed engines": {comfyModel("/w/f.safetensors", nil,
|
||||
Profile{Name: "a", Engine: EngineComfyUI, GPUClass: "6000-24g", TP: 1},
|
||||
Profile{Name: "b", Engine: "vllm", GPUClass: "6000-24g", TP: 1}), "every profile"},
|
||||
"comfyui settings without the engine": {Model{Name: "x", Source: "/w/x", ComfyUI: &ComfyUI{},
|
||||
Profiles: []Profile{{Name: "a", Engine: "vllm", GPUClass: "6000-24g", TP: 1}}}, "engine comfyui"},
|
||||
"no card": {comfyModel("/w/f.safetensors", nil, Profile{Name: "a", Engine: EngineComfyUI, TP: 1}), "gpu_class"},
|
||||
} {
|
||||
if _, err := New([]Model{c.m}); err == nil || !strings.Contains(err.Error(), c.want) {
|
||||
t.Errorf("%s: %v", name, err)
|
||||
}
|
||||
}
|
||||
ok := comfyModel("https://mirror/flux-dev/v1/", &ComfyUI{Workflow: "flux-dev", Checkpoint: "flux1-dev-fp8.safetensors"})
|
||||
if _, err := New([]Model{ok}); err != nil {
|
||||
t.Errorf("a set with its checkpoint named was refused: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -9,6 +9,7 @@ import (
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
@@ -360,7 +361,7 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
names := append([]string{m.Name}, m.Aliases...)
|
||||
|
||||
prof, _ := m.ProfileByName(rep.Profile)
|
||||
out = append(out, endpoint.Info{
|
||||
info := endpoint.Info{
|
||||
ID: rep.ID,
|
||||
URL: strings.TrimRight(rep.Endpoint, "/"),
|
||||
Models: names,
|
||||
@@ -369,7 +370,18 @@ func (c *Controller) endpoints() []endpoint.Info {
|
||||
Pool: rep.Pool,
|
||||
Weight: rep.Weight,
|
||||
Priority: prof.Priority,
|
||||
})
|
||||
}
|
||||
if prof.Engine == catalog.EngineComfyUI {
|
||||
workflow, ckpt := m.ComfyWorkflow()
|
||||
w, err := c.workflow(workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; its replica is not offered",
|
||||
"model", m.Name, "replica", rep.ID, "err", err)
|
||||
continue
|
||||
}
|
||||
info.Workflow, info.UpstreamModel = w, ckpt
|
||||
}
|
||||
out = append(out, info)
|
||||
}
|
||||
return append(out, c.externalEndpoints()...)
|
||||
}
|
||||
@@ -412,23 +424,45 @@ func (c *Controller) externalEndpoints() []endpoint.Info {
|
||||
if m.IsPeer() {
|
||||
id = "peer:" + m.Name
|
||||
}
|
||||
engine, workflow := endpoint.EngineExternal, ""
|
||||
if m.External.API == catalog.APIComfyUI {
|
||||
w, err := c.workflow(m.External.Workflow)
|
||||
if err != nil {
|
||||
c.log.Warn("a ComfyUI model's workflow could not be read; not offered", "model", m.Name, "err", err)
|
||||
continue
|
||||
}
|
||||
engine, workflow = endpoint.EngineComfyUI, w
|
||||
}
|
||||
out = append(out, endpoint.Info{
|
||||
ID: id, URL: strings.TrimRight(m.External.URL, "/"),
|
||||
Models: append([]string{m.Name}, m.Aliases...), APIKey: key,
|
||||
Engine: endpoint.EngineExternal, Pool: "external", Weight: 1,
|
||||
Engine: engine, Pool: "external", Weight: 1,
|
||||
UpstreamModel: m.External.Model,
|
||||
Peer: m.External.Peer, PeerCluster: cluster, PeerTags: tags,
|
||||
Workflow: workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// workflow is a ComfyUI workflow as the router is sent it: a built-in's
|
||||
// name as it is, a file read here, on the control plane the catalog says it
|
||||
// is on, and sent as the workflow itself.
|
||||
func (c *Controller) workflow(w string) (string, error) {
|
||||
if !strings.ContainsAny(w, "/.") {
|
||||
return w, nil
|
||||
}
|
||||
b, err := os.ReadFile(w)
|
||||
return string(b), err
|
||||
}
|
||||
|
||||
// fingerprint summarizes a list so an unchanged one can be skipped.
|
||||
func fingerprint(infos []endpoint.Info) string {
|
||||
var b strings.Builder
|
||||
for _, i := range infos {
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)))
|
||||
fmt.Fprintf(&b, "%s|%s|%s|%s|%v|%g|%d|%s|%s|%s|%t|%x|%x;", i.ID, i.URL, i.Engine, i.Pool, i.Models, i.Weight,
|
||||
i.Priority, i.UpstreamModel, i.Peer, i.PeerCluster, i.PeerTags, sha256.Sum256([]byte(i.APIKey)),
|
||||
sha256.Sum256([]byte(i.Workflow)))
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
@@ -111,7 +111,14 @@ func (r *Reconciler) Adopt(nodeID string, shards []AdoptedShard) (unclaimed []Ad
|
||||
}
|
||||
r.mergeAdopted(rep, group)
|
||||
r.replicas[replicaID] = rep
|
||||
|
||||
// An on-demand model that is up was asked for, by somebody this
|
||||
// control plane has not heard of: it started before a restart or a
|
||||
// change of leader. Without this it reads as wanted by nobody and is
|
||||
// drained on the next pass — every awake on-demand model, at every
|
||||
// failover. It gets a whole idle timeout from now instead.
|
||||
if _, asked := r.wanted[m.Name]; !asked && m.Scaling.OnDemand() {
|
||||
r.wanted[m.Name] = r.now()
|
||||
}
|
||||
}
|
||||
return unclaimed
|
||||
}
|
||||
|
||||
@@ -1409,6 +1409,29 @@ func TestUseKeepsAnOnDemandModelAlive(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAwakeOnDemandModelSurvivesARestartOfTheControlPlane(t *testing.T) {
|
||||
// A control plane that has just started, or just become leader, knows
|
||||
// nobody asked for anything. Taking that to mean nobody wants the
|
||||
// on-demand models a node reports running drained every one of them at
|
||||
// every failover, under whoever was using them.
|
||||
h := newHarness(t, onDemandModel(10*time.Minute), gpuNode("1", 8))
|
||||
h.r.Adopt("1", []AdoptedShard{adopted("llama-99", "llama-99-r0", "1", 0, []int{0, 1, 2, 3})})
|
||||
h.r.Reconcile(t.Context())
|
||||
h.advance(5 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
if reps := h.r.Replicas(); len(reps) != 1 || !reps[0].State.Counts() {
|
||||
t.Fatalf("the adopted replica of an on-demand model was taken down: %+v", reps)
|
||||
}
|
||||
// Still an on-demand model: unused for a whole idle timeout, it goes.
|
||||
h.advance(6 * time.Minute)
|
||||
h.r.Reconcile(t.Context())
|
||||
for _, rep := range h.r.Replicas() {
|
||||
if rep.State.Counts() {
|
||||
t.Errorf("an adopted replica nobody used outlived its idle timeout: %+v", rep)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAModelWithMaxZeroStaysOff(t *testing.T) {
|
||||
// min 0 max 0 is configured and deliberately not running, which is a
|
||||
// different and equally useful thing from "load it when wanted".
|
||||
|
||||
@@ -0,0 +1,196 @@
|
||||
// Package train runs LoRA training jobs from the control plane (BACKLOG F4):
|
||||
// queued in the store, sent to the node each names when its agent is there
|
||||
// and has nothing else training, followed as the agent reports, and what it
|
||||
// makes kept in the store.
|
||||
package train
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// Store is where jobs are kept.
|
||||
type Store interface {
|
||||
QueuedTrainJobs(ctx context.Context) ([]store.TrainJob, error)
|
||||
ActiveTrainJobs(ctx context.Context) ([]store.TrainJob, error)
|
||||
TrainJob(ctx context.Context, id string) (store.TrainJob, error)
|
||||
UpdateTrainJob(ctx context.Context, j store.TrainJob) error
|
||||
PutTrainArtifact(ctx context.Context, job, kind string, data []byte) error
|
||||
}
|
||||
|
||||
// Agents is how jobs reach the nodes.
|
||||
type Agents interface {
|
||||
Send(node string, msg *llmgwv1.ControlMessage) error
|
||||
Connected(node string) bool
|
||||
}
|
||||
|
||||
// Service sends queued jobs and records how they go.
|
||||
type Service struct {
|
||||
store Store
|
||||
agents Agents
|
||||
log *slog.Logger
|
||||
|
||||
mu sync.Mutex
|
||||
// pieces are artifacts arriving, by job and kind.
|
||||
pieces map[[2]string][]byte
|
||||
}
|
||||
|
||||
// New is a service.
|
||||
func New(s Store, a Agents, log *slog.Logger) *Service {
|
||||
if log == nil {
|
||||
log = slog.New(slog.DiscardHandler)
|
||||
}
|
||||
return &Service{store: s, agents: a, log: log, pieces: map[[2]string][]byte{}}
|
||||
}
|
||||
|
||||
// Every is how often queued jobs are looked at.
|
||||
var Every = 5 * time.Second
|
||||
|
||||
// Run sends queued jobs until ctx ends.
|
||||
func (s *Service) Run(ctx context.Context) {
|
||||
t := time.NewTicker(Every)
|
||||
defer t.Stop()
|
||||
for {
|
||||
s.Dispatch(ctx)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Dispatch sends each queued job whose node is here and not already
|
||||
// training, one per node.
|
||||
func (s *Service) Dispatch(ctx context.Context) {
|
||||
queued, err := s.store.QueuedTrainJobs(ctx)
|
||||
if err != nil || len(queued) == 0 {
|
||||
return
|
||||
}
|
||||
active, err := s.store.ActiveTrainJobs(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
busy := map[string]bool{}
|
||||
for _, j := range active {
|
||||
busy[j.Node] = true
|
||||
}
|
||||
for _, q := range queued {
|
||||
if busy[q.Node] || !s.agents.Connected(q.Node) {
|
||||
continue
|
||||
}
|
||||
j, err := s.store.TrainJob(ctx, q.ID)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
err = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainStart{TrainStart: &llmgwv1.TrainStart{
|
||||
JobId: j.ID, BaseRepo: j.BaseRepo, BaseRevision: j.BaseRevision, Data: j.Data, Params: j.Params,
|
||||
GpuMemoryMib: int32(j.GPUMemoryMiB), Gguf: j.GGUF}}}) //nolint:gosec // MiB
|
||||
if err != nil {
|
||||
s.log.Warn("could not send a training job to its node", "job", j.ID, "node", j.Node, "err", err)
|
||||
continue
|
||||
}
|
||||
j.State, j.StartedAt, j.Message = store.TrainStarting, time.Now(), "sent to "+j.Node
|
||||
_ = s.store.UpdateTrainJob(ctx, j)
|
||||
busy[j.Node] = true
|
||||
s.log.Info("training job sent", "job", j.ID, "node", j.Node, "adapter", j.Name)
|
||||
}
|
||||
}
|
||||
|
||||
// OnTrainStatus records how a job goes, as its node says.
|
||||
func (s *Service) OnTrainStatus(node string, st *llmgwv1.TrainStatus) {
|
||||
ctx := context.Background()
|
||||
j, err := s.store.TrainJob(ctx, st.GetJobId())
|
||||
if err != nil || j.Node != node || store.TrainOver(j.State) {
|
||||
return
|
||||
}
|
||||
j.State = st.GetState()
|
||||
if m := st.GetMessage(); m != "" || j.State == store.TrainDone {
|
||||
j.Message = m
|
||||
}
|
||||
if st.GetSteps() > 0 {
|
||||
j.Step, j.Steps = int(st.GetStep()), int(st.GetSteps())
|
||||
}
|
||||
if st.GetLoss() > 0 {
|
||||
j.Loss = st.GetLoss()
|
||||
}
|
||||
if p := int(st.GetPeakMib()); p > j.PeakMiB {
|
||||
j.PeakMiB = p
|
||||
}
|
||||
if store.TrainOver(j.State) {
|
||||
j.FinishedAt = time.Now()
|
||||
s.mu.Lock()
|
||||
for k := range s.pieces {
|
||||
if k[0] == j.ID {
|
||||
delete(s.pieces, k)
|
||||
}
|
||||
}
|
||||
s.mu.Unlock()
|
||||
s.log.Info("training job ended", "job", j.ID, "state", j.State, "message", j.Message)
|
||||
}
|
||||
if err := s.store.UpdateTrainJob(ctx, j); err != nil {
|
||||
s.log.Warn("could not record a training job's progress", "job", j.ID, "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// OnTrainArtifact gathers what a job made, and keeps it when it is whole.
|
||||
func (s *Service) OnTrainArtifact(node string, a *llmgwv1.TrainArtifact) {
|
||||
key := [2]string{a.GetJobId(), a.GetKind()}
|
||||
s.mu.Lock()
|
||||
if a.GetSeq() == 0 {
|
||||
s.pieces[key] = nil
|
||||
}
|
||||
s.pieces[key] = append(s.pieces[key], a.GetData()...)
|
||||
whole := s.pieces[key]
|
||||
if a.GetLast() {
|
||||
delete(s.pieces, key)
|
||||
}
|
||||
s.mu.Unlock()
|
||||
if !a.GetLast() {
|
||||
return
|
||||
}
|
||||
if err := s.store.PutTrainArtifact(context.Background(), key[0], key[1], whole); err != nil {
|
||||
s.log.Warn("could not keep what a training job made", "job", key[0], "kind", key[1], "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
// OnNodeGone fails the jobs a node was running: its agent's stream ended,
|
||||
// and the job with it.
|
||||
func (s *Service) OnNodeGone(node string) {
|
||||
ctx := context.Background()
|
||||
active, err := s.store.ActiveTrainJobs(ctx)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
for _, j := range active {
|
||||
if j.Node != node {
|
||||
continue
|
||||
}
|
||||
j.State, j.FinishedAt = store.TrainFailed, time.Now()
|
||||
j.Message = "the node's agent went away while it trained; start the job again"
|
||||
_ = s.store.UpdateTrainJob(ctx, j)
|
||||
}
|
||||
}
|
||||
|
||||
// Cancel stops a job: one still queued is not sent, one running is stopped
|
||||
// on its node.
|
||||
func (s *Service) Cancel(ctx context.Context, id string) error {
|
||||
j, err := s.store.TrainJob(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if store.TrainOver(j.State) {
|
||||
return nil
|
||||
}
|
||||
if j.State != store.TrainQueued && s.agents.Connected(j.Node) {
|
||||
_ = s.agents.Send(j.Node, &llmgwv1.ControlMessage{Msg: &llmgwv1.ControlMessage_TrainCancel{
|
||||
TrainCancel: &llmgwv1.TrainCancel{JobId: id}}})
|
||||
}
|
||||
j.State, j.FinishedAt, j.Message = store.TrainCancelled, time.Now(), "cancelled"
|
||||
return s.store.UpdateTrainJob(ctx, j)
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
package train
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
type fakeAgents struct {
|
||||
mu sync.Mutex
|
||||
connected map[string]bool
|
||||
sent []*llmgwv1.ControlMessage
|
||||
}
|
||||
|
||||
func (f *fakeAgents) Send(_ string, m *llmgwv1.ControlMessage) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.sent = append(f.sent, m)
|
||||
return nil
|
||||
}
|
||||
func (f *fakeAgents) Connected(n string) bool {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.connected[n]
|
||||
}
|
||||
|
||||
func setup(t *testing.T) (*Service, *store.DB, *fakeAgents) {
|
||||
t.Helper()
|
||||
s, err := store.OpenSQLite(context.Background(), filepath.Join(t.TempDir(), "s.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { _ = s.Close() })
|
||||
a := &fakeAgents{connected: map[string]bool{}}
|
||||
return New(s, a, nil), s, a
|
||||
}
|
||||
|
||||
func job(t *testing.T, s *store.DB, id, node string) {
|
||||
t.Helper()
|
||||
if err := s.CreateTrainJob(context.Background(), store.TrainJob{ID: id, Name: "a-" + id, Model: "m", BaseRepo: "o/n",
|
||||
Node: node, GPUMemoryMiB: 2000, Data: []byte(`{"q":"x","a":"y"}`), Pairs: 1, GGUF: true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAJobGoesToItsNodeWhenItIsHereAndFree(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, s, a := setup(t)
|
||||
job(t, s, "j1", "cuda01")
|
||||
job(t, s, "j2", "cuda01")
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 0 {
|
||||
t.Fatal("sent to a node that is not here")
|
||||
}
|
||||
a.connected["cuda01"] = true
|
||||
svc.Dispatch(ctx)
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 1 {
|
||||
t.Fatalf("sent %d jobs to one node", len(a.sent))
|
||||
}
|
||||
start := a.sent[0].GetTrainStart()
|
||||
if start.GetJobId() != "j1" || start.GetGpuMemoryMib() != 2000 || !start.GetGguf() || string(start.GetData()) == "" {
|
||||
t.Errorf("sent %+v", start)
|
||||
}
|
||||
if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainStarting || j.StartedAt.IsZero() {
|
||||
t.Errorf("j1 %+v", j)
|
||||
}
|
||||
|
||||
// Progress, then what it made in pieces, then done: the next is sent.
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running", Step: 5, Steps: 10, Loss: 0.8, PeakMib: 1900})
|
||||
svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 0, Data: []byte("GG")})
|
||||
svc.OnTrainArtifact("cuda01", &llmgwv1.TrainArtifact{JobId: "j1", Kind: "gguf", Seq: 1, Data: []byte("UF"), Last: true})
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "done"})
|
||||
j, _ := s.TrainJob(ctx, "j1")
|
||||
if j.State != store.TrainDone || j.Step != 5 || j.PeakMiB != 1900 || j.Loss != 0.8 || j.FinishedAt.IsZero() {
|
||||
t.Errorf("j1 after %+v", j)
|
||||
}
|
||||
if b, _ := s.TrainArtifact(ctx, "j1", "gguf"); string(b) != "GGUF" {
|
||||
t.Errorf("artifact %q", b)
|
||||
}
|
||||
// A late message does not bring an ended job back.
|
||||
svc.OnTrainStatus("cuda01", &llmgwv1.TrainStatus{JobId: "j1", State: "running"})
|
||||
if j, _ := s.TrainJob(ctx, "j1"); j.State != store.TrainDone {
|
||||
t.Errorf("revived: %s", j.State)
|
||||
}
|
||||
// Nor does another node speak for it.
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 2 || a.sent[1].GetTrainStart().GetJobId() != "j2" {
|
||||
t.Fatalf("j2 not sent: %d", len(a.sent))
|
||||
}
|
||||
svc.OnTrainStatus("node01", &llmgwv1.TrainStatus{JobId: "j2", State: "done"})
|
||||
if j, _ := s.TrainJob(ctx, "j2"); j.State == store.TrainDone {
|
||||
t.Error("another node ended the job")
|
||||
}
|
||||
// The node goes: its job fails, saying why.
|
||||
svc.OnNodeGone("cuda01")
|
||||
if j, _ := s.TrainJob(ctx, "j2"); j.State != store.TrainFailed || j.Message == "" {
|
||||
t.Errorf("after the node went: %+v", j)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCancellingAJob(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
svc, s, a := setup(t)
|
||||
job(t, s, "j1", "cuda01")
|
||||
if err := svc.Cancel(ctx, "j1"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a.connected["cuda01"] = true
|
||||
svc.Dispatch(ctx)
|
||||
if len(a.sent) != 0 {
|
||||
t.Error("a cancelled job was sent")
|
||||
}
|
||||
job(t, s, "j2", "cuda01")
|
||||
svc.Dispatch(ctx)
|
||||
_ = svc.Cancel(ctx, "j2")
|
||||
if len(a.sent) != 2 || a.sent[1].GetTrainCancel().GetJobId() != "j2" {
|
||||
t.Errorf("the node was not told: %+v", a.sent)
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ var Reference = []Block{
|
||||
{Key: "name", Type: "string", Doc: "Name is what clients ask for, like a model name.", Nested: ""},
|
||||
{Key: "source", Type: "string", Doc: "Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT\nadapter directory (a mirror version ending in \"/\", a path on the node,\nor a Hugging Face repository id). Fetched by each node as the model's\nweights are.", Nested: ""},
|
||||
{Key: "scale", Type: "float64", Doc: "Scale is how strongly it is applied; zero takes 1.", Nested: ""},
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree the adapter belongs to —\norg:ID, team:ID, project:ID, user:ID or key:ID — and only callers in\nit see it: to anyone else it is a model that does not exist. An\nadapter trained on an organisation's documents carries some of what\nthey say in its weights. Empty is every caller the model allows.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Admission", Title: "Admission", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "bool", Doc: "Enabled turns the layer on. Off, every request goes straight through,\nwhich is the v0.2.0 behaviour.", Nested: ""},
|
||||
@@ -63,6 +64,15 @@ var Reference = []Block{
|
||||
{Key: "tools", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "vision", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "embeddings", Type: "bool", Doc: "", Nested: ""},
|
||||
{Key: "images", Type: "bool", Doc: "Images is a model that makes pictures, asked for at\n/v1/images/generations. Implied by an external model whose API is\nComfyUI's.", Nested: ""},
|
||||
}},
|
||||
{Package: "catalog", Name: "ComfyUI", Title: "ComfyUI", Doc: "", Fields: []Field{
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is a built-in's name (flux-schnell, the default, or\nflux-dev) or a JSON file on the control plane, in ComfyUI's API\nformat.", Nested: ""},
|
||||
{Key: "checkpoint", Type: "string", Doc: "Checkpoint is the file the workflow's {{checkpoint}} names: the\nsource's own name unless set, which a source that is a set of files\nmust.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "ComfyUIConfig", Title: "ComfyUIConfig", Doc: "", Fields: []Field{
|
||||
{Key: "dir", Type: "string", Doc: "Dir is ComfyUI's directory, where main.py is. Readable by the user\nengines run as.", Nested: ""},
|
||||
{Key: "python", Type: "string", Doc: "Python is the interpreter of its environment; Dir's .venv when unset.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Control", Title: "Control", Doc: "", Fields: []Field{
|
||||
{Key: "interval", Type: "time.Duration", Doc: "Interval is how often desired and actual state are compared.", Nested: ""},
|
||||
@@ -83,6 +93,7 @@ var Reference = []Block{
|
||||
{Key: "api_key", Type: "string", Doc: "APIKey is sent to the engine as a bearer token, for an engine started\nwith one.", Nested: ""},
|
||||
{Key: "models", Type: "[]string", Doc: "", Nested: ""},
|
||||
{Key: "engine", Type: "string", Doc: "Engine names the inference engine, which decides what its metrics are\ncalled. Empty means vLLM's names.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow for engine: comfyui, a built-in's\nname or its JSON; flux-schnell unless set.", Nested: ""},
|
||||
{Key: "pool", Type: "string", Doc: "", Nested: ""},
|
||||
{Key: "weight", Type: "float64", Doc: "", Nested: ""},
|
||||
{Key: "priority", Type: "int", Doc: "", Nested: ""},
|
||||
@@ -103,6 +114,8 @@ var Reference = []Block{
|
||||
{Key: "model", Type: "string", Doc: "Model is what the provider calls it; requests are rewritten to it.", Nested: ""},
|
||||
{Key: "api_key_file", Type: "string", Doc: "APIKeyFile holds the provider's key, on the control plane. A file\nrather than the key itself, so the key is in neither the catalog, the\nstore nor the audit log; empty for a provider that asks for none.", Nested: ""},
|
||||
{Key: "peer", Type: "string", Doc: "Peer names the federated llmgw cluster this model is offered by, for\na model that federation added (`llmgw ctl federation add`) rather\nthan one written by hand. Such a model is named PEER/MODEL, comes and\ngoes with the peer's own list, and is sent only while the peer is up.", Nested: ""},
|
||||
{Key: "api", Type: "string", Doc: "API is what the provider speaks: OpenAI's, unless this says comfyui —\na ComfyUI server, asked for pictures at /v1/images/generations and\nsent the workflow below with the request's prompt in it. Model is then\nthe checkpoint file the workflow loads, as ComfyUI lists it.", Nested: ""},
|
||||
{Key: "workflow", Type: "string", Doc: "Workflow is the ComfyUI workflow, in its API format: a built-in's name\n(flux-schnell, the default, or flux-dev) or a JSON file on the control\nplane.", Nested: ""},
|
||||
}},
|
||||
{Package: "config", Name: "Facility", Title: "Facility", Doc: "", Fields: []Field{
|
||||
{Key: "budgets", Type: "[]FacilityBudget", Doc: "Budgets are the domains with a power cap.", Nested: ""},
|
||||
@@ -170,6 +183,7 @@ var Reference = []Block{
|
||||
{Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""},
|
||||
{Key: "owner", Type: "string", Doc: "Owner is the part of the tenancy tree whose capacity quota this model\ncounts against, written org:ID, team:ID or project:ID. Empty is shared:\nheld by the cluster, counted against nobody.", Nested: ""},
|
||||
{Key: "external", Type: "*External", Doc: "External is a model served by someone else — an OpenAI-compatible API\noutside the cluster — rather than by replicas placed here. Such a model\nhas no profiles and nothing is placed for it; the router sends its\nrequests to the provider. Most useful named in another model's\nfallbacks: used only when the cluster has no capacity left. Its\nprompts leave the cluster, so which keys may name it is worth an\nallowlist.", Nested: "catalog.External"},
|
||||
{Key: "comfyui", Type: "*ComfyUI", Doc: "ComfyUI is how a model whose profiles run engine comfyui makes its\npictures: the workflow, and the file it loads.", Nested: "catalog.ComfyUI"},
|
||||
{Key: "adapters", Type: "[]Adapter", Doc: "Adapters are LoRA adapters served on this model's replicas, each asked\nfor by its own name as though it were a model. Every replica loads\nevery adapter, applying one only to a request that names it — unless\nMaxLoadedAdapters is set.", Nested: ""},
|
||||
{Key: "max_loaded_adapters", Type: "int", Doc: "MaxLoadedAdapters, above zero, has each replica hold at most this many\nadapters at once, loaded when a request first asks for one and unloaded\nleast recently used first to make room. The router sends a request for\nan adapter to a replica that already holds it while that one has room\nto take it. vLLM profiles only: llama-server loads adapters at start\nand cannot load one afterwards. Zero loads every adapter on every\nreplica at start.", Nested: ""},
|
||||
{Key: "fallbacks", Type: "[]string", Doc: "", Nested: ""},
|
||||
@@ -180,6 +194,7 @@ var Reference = []Block{
|
||||
{Key: "context_length", Type: "int", Doc: "ContextLength is how many tokens one request may use, prompt and\ncompletion together, and MaxOutput bounds the completion alone. Zero\nmeans unknown, which is what every model was until now.\n\nThe gateway knew neither, so an over-long prompt was not refused: it was\nadmitted, given a replica, and failed at the engine, and a client sizing\nits prompts from /v1/models had nothing to read. Worse, the number is not\nthe one an operator writes in the engine's arguments — llama.cpp divides\n--ctx-size across its parallel slots, so a pool of 8192 across four slots\nis 2048 a request, and nothing said so.", Nested: ""},
|
||||
{Key: "max_output", Type: "int", Doc: "", Nested: ""},
|
||||
{Key: "capabilities", Type: "Capabilities", Doc: "Capabilities is what this model can do beyond plain completion.", Nested: "catalog.Capabilities"},
|
||||
{Key: "query_prefix", Type: "string", Doc: "QueryPrefix is put before a question an embedding model embeds for a\nknowledge-base search; documents are embedded without it. Models\ntrained with an instruction on the query side want it, Qwen3-Embedding\n\"Instruct: Given a web search query, retrieve relevant passages that\nanswer the query\\nQuery: \" for one.", Nested: ""},
|
||||
{Key: "role", Type: "string", Doc: "Role \"guard\" is a model that judges the gateway's traffic for the\nguardrails rather than serving clients: hidden from /v1/models and the\nchat page, refused to any key that names it, critical by default, and\nalways running. Guard says how its answers are read.", Nested: ""},
|
||||
{Key: "guard", Type: "*GuardModel", Doc: "", Nested: "catalog.GuardModel"},
|
||||
{Key: "origin", Type: "*Origin", Doc: "Origin is the registry repository the weights came from, at which\ncommit, and each file's checksum there. Written when a model is served\nfrom the registry; for one that was not, worked out from the mirror's\nrecord of where it fetched the file. It is what lets the control plane\nsay that the repository has published different weights since, and\nmove the model to them.", Nested: "catalog.Origin"},
|
||||
@@ -443,6 +458,7 @@ var Reference = []Block{
|
||||
{Package: "accounting", Name: "Price", Title: "accounting.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is what one picture costs, for a model that makes them: a\npicture's cost is the card's time, which its prompt's few tokens do\nnot measure.", Nested: ""},
|
||||
}},
|
||||
{Package: "admission", Name: "Band", Title: "admission.Band", Doc: "", Fields: []Field{
|
||||
{Key: "name", Type: "string", Doc: "", Nested: ""},
|
||||
@@ -487,6 +503,7 @@ var Reference = []Block{
|
||||
{Package: "config", Name: "Price", Title: "config.Price", Doc: "", Fields: []Field{
|
||||
{Key: "prompt_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "completion_per_1k", Type: "int64", Doc: "", Nested: ""},
|
||||
{Key: "per_image", Type: "int64", Doc: "PerImage is the cost of one picture, for a model that makes them\n(/v1/images/generations), on top of its prompt's tokens.", Nested: ""},
|
||||
}},
|
||||
{Package: "engines", Name: "Config", Title: "engines.Config", Doc: "", Fields: []Field{
|
||||
{Key: "disabled", Type: "[]string", Doc: "Disabled are engines this node never installs or advertises, whatever\nthe control plane asks: vLLM on a node of cards no build of it runs on,\nthat is meant for llama.cpp alone.", Nested: ""},
|
||||
@@ -495,6 +512,7 @@ var Reference = []Block{
|
||||
{Key: "llamacpp", Type: "LlamaCppConfig", Doc: "", Nested: "engines.LlamaCppConfig"},
|
||||
{Key: "trtllm", Type: "TRTLLMConfig", Doc: "TRTLLM is TensorRT-LLM, which the agent does not install: a node\noffers it only when its interpreter is named here.", Nested: "engines.TRTLLMConfig"},
|
||||
{Key: "sim", Type: "SimConfig", Doc: "", Nested: "engines.SimConfig"},
|
||||
{Key: "comfyui", Type: "ComfyUIConfig", Doc: "ComfyUI makes pictures; offered by a node whose ComfyUI is named here.", Nested: "engines.ComfyUIConfig"},
|
||||
}},
|
||||
{Package: "guard", Name: "Config", Title: "guard.Config", Doc: "", Fields: []Field{
|
||||
{Key: "enabled", Type: "*bool", Doc: "Enabled turns every policy on or off at once: false checks nothing,\nwhatever is defined or attached. Unset is on.", Nested: ""},
|
||||
|
||||
@@ -22,6 +22,7 @@ var Sources = []string{
|
||||
"internal/agent/engines/sglang.go",
|
||||
"internal/agent/engines/trtllm.go",
|
||||
"internal/agent/engines/sim.go",
|
||||
"internal/agent/engines/comfyui.go",
|
||||
"internal/agent/supervisor/supervisor.go",
|
||||
"internal/tlsconf/tlsconf.go",
|
||||
"internal/control/catalog/catalog.go",
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// A store member that is nothing else runs no llmgw process: Patroni, etcd
|
||||
// and PostgreSQL only. What the cluster's settings say of its firewall it
|
||||
// reads from its own copy of the store, where the leading control plane
|
||||
// keeps it (the cluster_state table, key firewall), once a minute, by a
|
||||
// root timer running FollowScript. It turns llmgw's table off and on, and
|
||||
// lets in the networks the settings add; what `server init` let in stays.
|
||||
|
||||
// FollowScript is the script, as python3 runs it. LLMGW_FW_STATE and
|
||||
// LLMGW_FW_RULES replace the store and the rules file, for its tests.
|
||||
const FollowScript = `#!/usr/bin/python3
|
||||
# Written by llmgw: llmgw's firewall on this store member, as the cluster's
|
||||
# settings say. Run by llmgw-firewall-follow.timer.
|
||||
import ipaddress, json, os, subprocess, sys, tempfile
|
||||
|
||||
RULES = os.environ.get("LLMGW_FW_RULES", "` + RulesFile + `")
|
||||
|
||||
def run(*a):
|
||||
return subprocess.run(a, capture_output=True, text=True)
|
||||
|
||||
raw = os.environ.get("LLMGW_FW_STATE")
|
||||
if raw is None:
|
||||
r = run("runuser", "-u", "postgres", "--", "psql", "-d", "llmgw", "-tAqc",
|
||||
"SELECT value FROM cluster_state WHERE key = 'firewall'")
|
||||
# No table yet, or nothing told: a control plane older than this, or
|
||||
# none leading. Left as it is.
|
||||
if r.returncode != 0:
|
||||
sys.exit(0)
|
||||
raw = r.stdout
|
||||
if not raw.strip():
|
||||
sys.exit(0)
|
||||
st = json.loads(raw)
|
||||
present = run("nft", "list", "table", "inet", "` + Table + `").returncode == 0
|
||||
|
||||
if not st.get("enabled", True):
|
||||
if present:
|
||||
run("systemctl", "disable", "--now", "` + Unit + `")
|
||||
run("nft", "delete", "table", "inet", "` + Table + `")
|
||||
print("llmgw's firewall is off here")
|
||||
sys.exit(0)
|
||||
|
||||
if not os.path.exists(RULES):
|
||||
sys.exit(0)
|
||||
lines = open(RULES).read().split("\n")
|
||||
want = []
|
||||
for n in st.get("networks") or []:
|
||||
try:
|
||||
want.append(ipaddress.ip_network(n, strict=False))
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
def saddr(l):
|
||||
s = l.strip()
|
||||
return s.startswith("ip saddr {") or s.startswith("ip6 saddr {")
|
||||
|
||||
changed = False
|
||||
for fam, kw in ((4, "ip saddr {"), (6, "ip6 saddr {")):
|
||||
add = [str(n) for n in want if n.version == fam]
|
||||
if not add:
|
||||
continue
|
||||
same = [i for i, l in enumerate(lines) if l.strip().startswith(kw)]
|
||||
if same:
|
||||
i = same[0]
|
||||
l = lines[i]
|
||||
a, b = l.index("{") + 1, l.index("}")
|
||||
have = [x.strip() for x in l[a:b].split(",") if x.strip()]
|
||||
new = [x for x in add if x not in have]
|
||||
if new:
|
||||
lines[i] = l[:a] + " " + ", ".join(have + new) + " " + l[b:]
|
||||
changed = True
|
||||
continue
|
||||
other = [i for i, l in enumerate(lines) if saddr(l)]
|
||||
if not other:
|
||||
continue
|
||||
l = lines[other[0]]
|
||||
indent = l[:len(l) - len(l.lstrip())]
|
||||
lines.insert(other[0] + 1, indent + kw + " " + ", ".join(add) + " }" + l[l.index("}") + 1:])
|
||||
changed = True
|
||||
|
||||
if changed:
|
||||
fd, tmp = tempfile.mkstemp(dir=os.path.dirname(RULES))
|
||||
with os.fdopen(fd, "w") as f:
|
||||
f.write("\n".join(lines))
|
||||
if run("nft", "-c", "-f", tmp).returncode != 0:
|
||||
os.unlink(tmp)
|
||||
print("the settings' networks make a ruleset nft refuses; left as it was", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
os.chmod(tmp, 0o644)
|
||||
os.replace(tmp, RULES)
|
||||
if changed or not present:
|
||||
run("systemctl", "enable", "` + Unit + `")
|
||||
if run("nft", "-f", RULES).returncode != 0:
|
||||
sys.exit(1)
|
||||
print("llmgw's firewall is on here")
|
||||
`
|
||||
|
||||
// FollowerPath is where FollowScript is installed.
|
||||
const FollowerPath = "/usr/local/lib/llmgw/firewall-follow"
|
||||
|
||||
// FollowerTimer is the timer that runs it.
|
||||
const FollowerTimer = "llmgw-firewall-follow.timer"
|
||||
|
||||
const followService = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this store member
|
||||
After=postgresql.service patroni.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/bin/python3 ` + FollowerPath + `
|
||||
`
|
||||
|
||||
const followTimer = `[Unit]
|
||||
Description=llmgw firewall: follow the cluster's settings on this store member
|
||||
|
||||
[Timer]
|
||||
OnBootSec=1min
|
||||
OnUnitActiveSec=1min
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
`
|
||||
|
||||
// InstallFollower puts FollowScript and its timer on a store member.
|
||||
func InstallFollower(ctx context.Context, r Runner) error {
|
||||
body := FollowScript + "\x00" + followService + "\x00" + followTimer
|
||||
out, err := r.Run(ctx, `set -e
|
||||
t=$(mktemp -d); trap 'rm -rf "$t"' EXIT
|
||||
cat > "$t/all"
|
||||
python3 - "$t" <<'PY'
|
||||
import sys
|
||||
d = sys.argv[1]
|
||||
s, svc, tim = open(d + "/all", "rb").read().split(b"\x00")
|
||||
open(d + "/script", "wb").write(s)
|
||||
open(d + "/service", "wb").write(svc)
|
||||
open(d + "/timer", "wb").write(tim)
|
||||
PY
|
||||
sudo install -d -m 755 /usr/local/lib/llmgw
|
||||
sudo install -m 755 "$t/script" `+FollowerPath+`
|
||||
sudo install -m 644 "$t/service" /etc/systemd/system/llmgw-firewall-follow.service
|
||||
sudo install -m 644 "$t/timer" /etc/systemd/system/`+FollowerTimer+`
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now `+FollowerTimer+` >/dev/null 2>&1`, []byte(body))
|
||||
if err != nil {
|
||||
return fmt.Errorf("installing %s: %v: %s", FollowerTimer, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package firewall
|
||||
|
||||
import (
|
||||
"net/netip"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// follow runs FollowScript against a rules file and a state, with nft and
|
||||
// systemctl replaced by scripts that log what they were asked and answer as
|
||||
// a machine whose table is loaded or not.
|
||||
func follow(t *testing.T, rules, state string, loaded bool) (string, string, string) {
|
||||
t.Helper()
|
||||
if _, err := exec.LookPath("python3"); err != nil {
|
||||
t.Skip("no python3")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
bin := filepath.Join(dir, "bin")
|
||||
_ = os.Mkdir(bin, 0o755)
|
||||
log := filepath.Join(dir, "log")
|
||||
list := "exit 1"
|
||||
if loaded {
|
||||
list = "exit 0"
|
||||
}
|
||||
for name, body := range map[string]string{
|
||||
"nft": `echo "nft $*" >> ` + log + "\n" + `[ "$1" = list ] && ` + list + "\nexit 0\n",
|
||||
"systemctl": `echo "systemctl $*" >> ` + log + "\nexit 0\n",
|
||||
} {
|
||||
if err := os.WriteFile(filepath.Join(bin, name), []byte("#!/bin/sh\n"+body), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
rf := filepath.Join(dir, "firewall.nft")
|
||||
if rules != "" {
|
||||
_ = os.WriteFile(rf, []byte(rules), 0o644)
|
||||
}
|
||||
script := filepath.Join(dir, "follow")
|
||||
_ = os.WriteFile(script, []byte(FollowScript), 0o755)
|
||||
cmd := exec.Command("python3", script)
|
||||
cmd.Env = append(os.Environ(), "PATH="+bin+":"+os.Getenv("PATH"), "LLMGW_FW_STATE="+state, "LLMGW_FW_RULES="+rf)
|
||||
out, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatalf("%v: %s", err, out)
|
||||
}
|
||||
got, _ := os.ReadFile(rf)
|
||||
calls, _ := os.ReadFile(log)
|
||||
return string(out), string(got), string(calls)
|
||||
}
|
||||
|
||||
var memberRules = Script(Spec{Ports: []Port{Postgres, EtcdClient}, Allow: mustPrefixes("10.0.0.0/24")})
|
||||
|
||||
func TestAStoreMemberFollowsTheSettings(t *testing.T) {
|
||||
// Off: the table goes, the rules stay for later.
|
||||
out, rules, calls := follow(t, memberRules, `{"enabled":false}`, true)
|
||||
if !strings.Contains(out, "off") || !strings.Contains(calls, "nft delete table inet llmgw") ||
|
||||
!strings.Contains(calls, "systemctl disable --now "+Unit) || rules != memberRules {
|
||||
t.Errorf("off: %q %q", out, calls)
|
||||
}
|
||||
// On again: loaded from the rules kept.
|
||||
out, _, calls = follow(t, memberRules, `{"enabled":true}`, false)
|
||||
if !strings.Contains(out, "on") || !strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("on: %q %q", out, calls)
|
||||
}
|
||||
// On and loaded, nothing new: nothing done.
|
||||
out, _, calls = follow(t, memberRules, `{"enabled":true,"networks":["10.0.0.0/24"]}`, true)
|
||||
if out != "" || strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("unchanged: %q %q", out, calls)
|
||||
}
|
||||
// Networks the settings add, of both families, are let in beside what
|
||||
// init let in; the result reads back as the same ports.
|
||||
_, rules, calls = follow(t, memberRules, `{"enabled":true,"networks":["192.168.5.0/24","fd00::/64","nonsense"]}`, true)
|
||||
if !strings.Contains(calls, "nft -c -f") || !strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("not checked and loaded: %q", calls)
|
||||
}
|
||||
back := parse(rules)
|
||||
var nets []string
|
||||
for _, p := range back.Allow {
|
||||
nets = append(nets, p.String())
|
||||
}
|
||||
if strings.Join(nets, ",") != "10.0.0.0/24,192.168.5.0/24,fd00::/64" || len(back.Ports) != 2 {
|
||||
t.Errorf("rules now:\n%s", rules)
|
||||
}
|
||||
// Nothing told yet, or no rules: left alone.
|
||||
if out, _, calls := follow(t, memberRules, "", true); out != "" || calls != "" {
|
||||
t.Errorf("nothing told: %q %q", out, calls)
|
||||
}
|
||||
if out, _, calls := follow(t, "", `{"enabled":true}`, false); out != "" || strings.Contains(calls, "nft -f") {
|
||||
t.Errorf("no rules: %q %q", out, calls)
|
||||
}
|
||||
}
|
||||
|
||||
func mustPrefixes(ss ...string) []netip.Prefix {
|
||||
var out []netip.Prefix
|
||||
for _, s := range ss {
|
||||
out = append(out, netip.MustParsePrefix(s))
|
||||
}
|
||||
return out
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
@@ -37,7 +38,9 @@ func (a *API) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /v1/models", a.fetch)
|
||||
mux.HandleFunc("GET /v1/models/{model}/{version}", a.status)
|
||||
mux.HandleFunc("DELETE /v1/models/{model}/{version}", a.remove)
|
||||
mux.HandleFunc("PUT /v1/models/{model}/{version}/{file}", a.upload)
|
||||
mux.HandleFunc("GET /v1/health", a.health)
|
||||
mux.HandleFunc("PUT /v1/firewall", a.putFirewall)
|
||||
return mux
|
||||
}
|
||||
|
||||
@@ -199,3 +202,59 @@ func LastPathElement(u string) string {
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
// maxUpload bounds a file sent to the mirror rather than fetched by it.
|
||||
const maxUpload = 8 << 30
|
||||
|
||||
// upload keeps a file the caller sends — an adapter a node trained, which
|
||||
// is on no registry to fetch from — with the same discipline as an import:
|
||||
// its checksum and source beside it, and an existing version never
|
||||
// replaced. X-Llmgw-Source says where it came from.
|
||||
func (a *API) upload(w http.ResponseWriter, r *http.Request) {
|
||||
e := Entry{Model: r.PathValue("model"), Version: r.PathValue("version"), File: r.PathValue("file")}
|
||||
for _, n := range []string{e.Model, e.Version, e.File} {
|
||||
if err := ValidName(n); err != nil {
|
||||
fail(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
if _, err := os.Stat(e.Path(a.m.root)); err == nil {
|
||||
fail(w, http.StatusConflict, fmt.Errorf("%s/%s/%s: %w", e.Model, e.Version, e.File, ErrExists))
|
||||
return
|
||||
}
|
||||
tmp, err := os.CreateTemp(a.m.root, ".upload-*")
|
||||
if err != nil {
|
||||
fail(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
defer os.Remove(tmp.Name())
|
||||
n, err := io.Copy(tmp, http.MaxBytesReader(w, r.Body, maxUpload))
|
||||
if cerr := tmp.Close(); err == nil {
|
||||
err = cerr
|
||||
}
|
||||
if err != nil {
|
||||
fail(w, http.StatusBadRequest, fmt.Errorf("receiving the file: %w", err))
|
||||
return
|
||||
}
|
||||
if n == 0 {
|
||||
fail(w, http.StatusBadRequest, errors.New("the file is empty"))
|
||||
return
|
||||
}
|
||||
source := strings.TrimSpace(r.Header.Get("X-Llmgw-Source"))
|
||||
if source == "" {
|
||||
source = "uploaded"
|
||||
}
|
||||
added, err := Import(r.Context(), a.m.root, tmp.Name(), e, "", source, nil)
|
||||
switch {
|
||||
case errors.Is(err, ErrExists):
|
||||
fail(w, http.StatusConflict, err)
|
||||
return
|
||||
case err != nil:
|
||||
fail(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusCreated, FetchResponse{
|
||||
Status: Status{Model: e.Model, Version: e.Version, File: e.File, State: Present, BytesDone: added.Entry.Bytes, BytesTotal: added.Entry.Bytes},
|
||||
URL: e.URL(a.base),
|
||||
})
|
||||
}
|
||||
@@ -100,6 +100,36 @@ func (c *Client) Fetch(ctx context.Context, req FetchRequest) (FetchResponse, er
|
||||
return out, err
|
||||
}
|
||||
|
||||
// Upload sends a file for the mirror to keep at model/version/file, with
|
||||
// where it came from; it answers the URL nodes fetch it from.
|
||||
func (c *Client) Upload(ctx context.Context, model, version, file string, body []byte, source string) (FetchResponse, error) {
|
||||
var out FetchResponse
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodPut,
|
||||
c.base+"/v1/models/"+url.PathEscape(model)+"/"+url.PathEscape(version)+"/"+url.PathEscape(file), bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
req.Header.Set("Content-Type", "application/octet-stream")
|
||||
req.Header.Set("X-Llmgw-Source", source)
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return out, fmt.Errorf("reaching the mirror at %s: %w", c.base, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
|
||||
if err != nil {
|
||||
return out, err
|
||||
}
|
||||
if resp.StatusCode >= 300 {
|
||||
var e struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
_ = json.Unmarshal(raw, &e)
|
||||
return out, &APIError{Status: resp.StatusCode, Msg: e.Error}
|
||||
}
|
||||
return out, json.Unmarshal(raw, &out)
|
||||
}
|
||||
|
||||
// Status is where one model version stands.
|
||||
func (c *Client) Status(ctx context.Context, model, version string) (FetchResponse, error) {
|
||||
var out FetchResponse
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
// The mirror's machine keeps llmgw's firewall as the cluster's settings
|
||||
// say, as every other machine does. The mirror runs unprivileged and cannot
|
||||
// change the ruleset: the control plane tells it what the settings say, it
|
||||
// writes that beside its models, and a root path unit on the machine
|
||||
// applies it with `llmgw firewall sync`.
|
||||
|
||||
// FirewallState is what the cluster's settings say of a machine's firewall.
|
||||
type FirewallState struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
// Networks are let in besides those the machine reaches others on.
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
// Toward are the cluster's machines, by address: each one's network is
|
||||
// let in, as the machine's route to it says.
|
||||
Toward []string `json:"toward,omitempty"`
|
||||
}
|
||||
|
||||
// FirewallFile is where, in the mirror's directory, the state is written.
|
||||
// Hidden, so it is never taken for a model.
|
||||
const FirewallFile = ".firewall.json"
|
||||
|
||||
func (a *API) putFirewall(w http.ResponseWriter, r *http.Request) {
|
||||
var st FirewallState
|
||||
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20)).Decode(&st); err != nil {
|
||||
fail(w, http.StatusBadRequest, err)
|
||||
return
|
||||
}
|
||||
b, _ := json.Marshal(st)
|
||||
p := filepath.Join(a.m.root, FirewallFile)
|
||||
changed := false
|
||||
if old, err := os.ReadFile(p); err != nil || !bytes.Equal(old, b) {
|
||||
// Renamed into place: the path unit sees one change, of a whole file.
|
||||
tmp := p + ".new"
|
||||
if err := os.WriteFile(tmp, b, 0o644); err != nil {
|
||||
fail(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
if err := os.Rename(tmp, p); err != nil {
|
||||
fail(w, http.StatusInternalServerError, err)
|
||||
return
|
||||
}
|
||||
changed = true
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]bool{"changed": changed})
|
||||
}
|
||||
|
||||
// SetFirewall tells the mirror what the cluster's settings say of its
|
||||
// machine's firewall; it answers whether that changed anything.
|
||||
func (c *Client) SetFirewall(ctx context.Context, st FirewallState) (bool, error) {
|
||||
var out struct {
|
||||
Changed bool `json:"changed"`
|
||||
}
|
||||
err := c.do(ctx, http.MethodPut, "/v1/firewall", st, &out)
|
||||
return out.Changed, err
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestTheMirrorKeepsWhatTheSettingsSayOfItsFirewall(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
srv := httptest.NewServer(NewAPI(NewManager(root, discardLog()), "http://m:8081").Handler())
|
||||
defer srv.Close()
|
||||
c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
st := FirewallState{Enabled: true, Networks: []string{"192.168.5.0/24"}, Toward: []string{"10.0.0.147"}}
|
||||
if changed, err := c.SetFirewall(ctx, st); err != nil || !changed {
|
||||
t.Fatalf("first: %v %v", changed, err)
|
||||
}
|
||||
if changed, err := c.SetFirewall(ctx, st); err != nil || changed {
|
||||
t.Errorf("the same again: %v %v", changed, err)
|
||||
}
|
||||
var back FirewallState
|
||||
b, _ := os.ReadFile(filepath.Join(root, FirewallFile))
|
||||
if err := json.Unmarshal(b, &back); err != nil || !back.Enabled || back.Toward[0] != "10.0.0.147" {
|
||||
t.Errorf("kept %s", b)
|
||||
}
|
||||
if all, err := c.List(ctx); err != nil || len(all.Models) != 0 {
|
||||
t.Errorf("the file is taken for a model: %+v %v", all, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package mirror
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
|
||||
func TestAFileSentToTheMirrorIsKeptWithItsChecksumAndNeverReplaced(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
srv := httptest.NewServer(NewAPI(NewManager(root, nil), "http://mirror:8081").Handler())
|
||||
defer srv.Close()
|
||||
c, err := NewClient(srv.URL, tlsconf.Config{}, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
got, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("GGUF adapter"), "trained by job train-1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.URL != "http://mirror:8081/politika-tonu/train-1/politika-tonu.gguf" || got.Status.BytesTotal != 12 {
|
||||
t.Errorf("answered %+v", got)
|
||||
}
|
||||
e := Entry{Model: "politika-tonu", Version: "train-1", File: "politika-tonu.gguf"}
|
||||
if b, _ := os.ReadFile(e.Path(root)); string(b) != "GGUF adapter" {
|
||||
t.Errorf("kept %q", b)
|
||||
}
|
||||
if b, _ := os.ReadFile(sourcePath(e.Path(root))); !strings.Contains(string(b), "train-1") {
|
||||
t.Errorf("source %q", b)
|
||||
}
|
||||
if _, err := os.Stat(checksumPath(e.Path(root))); err != nil {
|
||||
t.Errorf("no checksum: %v", err)
|
||||
}
|
||||
if _, err := c.Upload(ctx, "politika-tonu", "train-1", "politika-tonu.gguf", []byte("other"), "x"); err == nil {
|
||||
t.Error("an existing version was replaced")
|
||||
}
|
||||
if _, err := c.Upload(ctx, "../etc", "v", "f", []byte("x"), "x"); err == nil {
|
||||
t.Error("a path outside the mirror was taken")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,171 @@
|
||||
package pgha
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PostgreSQL's own defaults — 128MB of cache, 4MB to sort in — are for a
|
||||
// machine it barely uses. A knowledge base's vector index is searched well
|
||||
// only from memory: one that does not fit is read from disk on every search.
|
||||
// So each member is sized to the memory it has, read when setup runs, and
|
||||
// again on every later run, so a machine given more memory gets the use of it.
|
||||
|
||||
// MemorySettings are PostgreSQL's memory parameters for a machine with total
|
||||
// bytes of memory. shared is a machine that also runs a control plane or an
|
||||
// agent: the database is given less of it, since a gateway swapped out to
|
||||
// give the database cache is worse than a smaller cache.
|
||||
func MemorySettings(total int64, shared bool) map[string]string {
|
||||
const mb = 1 << 20
|
||||
ram := total / mb
|
||||
// Shares of memory, in MB: the cache PostgreSQL keeps itself, what the
|
||||
// planner may count on the kernel caching besides, what building or
|
||||
// vacuuming an index may take, and what one sort may.
|
||||
buffers, cache, maint, sort := ram/4, ram*3/4, ram/16, ram/256
|
||||
if shared {
|
||||
buffers, cache, maint, sort = ram*3/20, ram/2, ram/32, ram/512
|
||||
}
|
||||
clamp := func(v, lo, hi int64) int64 { return min(max(v, lo), hi) }
|
||||
return map[string]string{
|
||||
"shared_buffers": strconv.FormatInt(clamp(buffers, 96, 32<<10), 10) + "MB",
|
||||
"effective_cache_size": strconv.FormatInt(clamp(cache, 256, 1<<20), 10) + "MB",
|
||||
"maintenance_work_mem": strconv.FormatInt(clamp(maint, 64, 2<<10), 10) + "MB",
|
||||
"work_mem": strconv.FormatInt(clamp(sort, 4, 64), 10) + "MB",
|
||||
}
|
||||
}
|
||||
|
||||
// MemTotal reads the machine's memory, in bytes, from /proc/meminfo.
|
||||
func MemTotal(meminfo string) (int64, bool) {
|
||||
for _, line := range strings.Split(meminfo, "\n") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 2 && f[0] == "MemTotal:" {
|
||||
kb, err := strconv.ParseInt(f[1], 10, 64)
|
||||
return kb << 10, err == nil && kb > 0
|
||||
}
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// memory sizes each member to its machine, in its own configuration — the
|
||||
// machines need not be alike. What takes effect on reload is applied so;
|
||||
// shared_buffers needs a restart, given one member at a time, the leader
|
||||
// last, each waited on before the next.
|
||||
func (s *Setup) memory(ctx context.Context, sec Secrets) error {
|
||||
var restart []Node
|
||||
for _, d := range s.DB {
|
||||
out, err := d.Remote.Run(ctx, `cat /proc/meminfo
|
||||
if [ -f /etc/llmgw/server.yaml ] || [ -f /etc/llmgw/agent.yaml ]; then echo "llmgw-shared"; fi`, nil)
|
||||
total, ok := MemTotal(string(out))
|
||||
if err != nil || !ok {
|
||||
s.emit("memory", false, "%s: could not read its memory; PostgreSQL left as it is", d.Name)
|
||||
continue
|
||||
}
|
||||
shared := strings.Contains(string(out), "llmgw-shared")
|
||||
params := MemorySettings(total, shared)
|
||||
body, _ := json.Marshal(params)
|
||||
out, err = d.Remote.Run(ctx, `set -e
|
||||
test -f /etc/patroni/config.yml || { echo missing; exit 0; }
|
||||
want=$(cat)
|
||||
r=$(sudo env WANT="$want" python3 - <<'PY'
|
||||
import json, os, yaml
|
||||
p = "/etc/patroni/config.yml"
|
||||
want = json.loads(os.environ["WANT"])
|
||||
c = yaml.safe_load(open(p))
|
||||
pg = c.setdefault("postgresql", {})
|
||||
params = pg.get("parameters") or {}
|
||||
pg["parameters"] = params
|
||||
if all(str(params.get(k)) == v for k, v in want.items()):
|
||||
print("same")
|
||||
else:
|
||||
restart = str(params.get("shared_buffers")) != want["shared_buffers"]
|
||||
params.update(want)
|
||||
open(p + ".new", "w").write("# llmgw: the control planes' store, kept by Patroni, its leader lock in etcd.\n" +
|
||||
yaml.safe_dump(c, default_flow_style=False, sort_keys=False))
|
||||
print("restart" if restart else "changed")
|
||||
PY
|
||||
)
|
||||
if [ "$r" != same ]; then
|
||||
sudo chown root:postgres /etc/patroni/config.yml.new
|
||||
sudo chmod 640 /etc/patroni/config.yml.new
|
||||
sudo mv -f /etc/patroni/config.yml.new /etc/patroni/config.yml
|
||||
sudo systemctl reload patroni
|
||||
fi
|
||||
echo "$r"`, body)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", d.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
what := fmt.Sprintf("%s: %d MB of memory%s; shared_buffers %s, effective_cache_size %s, maintenance_work_mem %s, work_mem %s",
|
||||
d.Name, total>>20, map[bool]string{true: ", shared with llmgw"}[shared],
|
||||
params["shared_buffers"], params["effective_cache_size"], params["maintenance_work_mem"], params["work_mem"])
|
||||
switch r := strings.TrimSpace(lastLine(out)); r {
|
||||
case "same", "missing":
|
||||
s.emit("memory", false, "%s", what)
|
||||
case "restart":
|
||||
s.emit("memory", true, "%s", what)
|
||||
restart = append(restart, d)
|
||||
default:
|
||||
s.emit("memory", true, "%s", what)
|
||||
}
|
||||
}
|
||||
// Patroni takes a reload in its own time: restarted before it has written
|
||||
// the new configuration, PostgreSQL comes back on the old one. So each is
|
||||
// waited on until Patroni says a restart is pending — and a member found
|
||||
// pending, by a run that stopped before restarting it, is restarted too.
|
||||
pending := func(c Cluster, name string) bool {
|
||||
for _, m := range c.Members {
|
||||
if m.Name == name {
|
||||
return m.PendingRestart
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
var c Cluster
|
||||
err := s.waitFor(ctx, s.dbHosts(), func(got Cluster) bool {
|
||||
c = got
|
||||
for _, d := range restart {
|
||||
if !pending(got, d.Name) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
})
|
||||
if err != nil {
|
||||
return fmt.Errorf("waiting for Patroni to take the new configuration: %w", err)
|
||||
}
|
||||
for _, d := range s.DB {
|
||||
if pending(c, d.Name) && !slices.ContainsFunc(restart, func(n Node) bool { return n.Name == d.Name }) {
|
||||
restart = append(restart, d)
|
||||
}
|
||||
}
|
||||
if len(restart) == 0 {
|
||||
return nil
|
||||
}
|
||||
// The leader last: its restart is the only one the control planes see,
|
||||
// a few seconds without writes, and by then the rest are back.
|
||||
slices.SortStableFunc(restart, func(a, b Node) int {
|
||||
al, bl := a.Name == c.Leader(), b.Name == c.Leader()
|
||||
switch {
|
||||
case al && !bl:
|
||||
return 1
|
||||
case bl && !al:
|
||||
return -1
|
||||
}
|
||||
return 0
|
||||
})
|
||||
for _, d := range restart {
|
||||
if _, err := restJSON(ctx, d.IP, http.MethodPost, "/restart", []byte(`{"restart_pending": true}`), sec.RestAPI, 2*time.Minute); err != nil {
|
||||
return fmt.Errorf("restarting PostgreSQL on %s for its new shared_buffers: %w", d.Name, err)
|
||||
}
|
||||
if err := s.waitFor(ctx, s.dbHosts(), func(c Cluster) bool { return c.Complete(len(s.DB)) }); err != nil {
|
||||
return fmt.Errorf("after restarting %s: %w", d.Name, err)
|
||||
}
|
||||
s.emit("memory", true, "%s: PostgreSQL restarted with the new shared_buffers", d.Name)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -465,3 +465,97 @@ func TestAControlPlaneWithoutTheAuthorityConnectsByTLSOnceTheClusterHasIt(t *tes
|
||||
t.Errorf("TLS asked of a cluster without it: %s", dsn)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgreSQLIsGivenAShareOfTheMachinesMemory(t *testing.T) {
|
||||
total, ok := MemTotal("MemTotal: 4009848 kB\nMemFree: 976000 kB\n")
|
||||
if !ok || total != 4009848<<10 {
|
||||
t.Fatalf("MemTotal %d %v", total, ok)
|
||||
}
|
||||
own := MemorySettings(total, false)
|
||||
if own["shared_buffers"] != "978MB" || own["effective_cache_size"] != "2936MB" ||
|
||||
own["maintenance_work_mem"] != "244MB" || own["work_mem"] != "15MB" {
|
||||
t.Errorf("a machine of its own: %v", own)
|
||||
}
|
||||
// Beside a control plane, less of it.
|
||||
if sh := MemorySettings(total, true); sh["shared_buffers"] != "587MB" || sh["work_mem"] != "7MB" {
|
||||
t.Errorf("a shared machine: %v", sh)
|
||||
}
|
||||
// Never below what PostgreSQL needs, never past what it can use.
|
||||
if tiny := MemorySettings(512<<20, true); tiny["shared_buffers"] != "96MB" || tiny["work_mem"] != "4MB" {
|
||||
t.Errorf("a small machine: %v", tiny)
|
||||
}
|
||||
if big := MemorySettings(1<<40, false); big["shared_buffers"] != "32768MB" || big["maintenance_work_mem"] != "2048MB" {
|
||||
t.Errorf("a large machine: %v", big)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewSharedBuffersRestartTheReplicasBeforeTheLeader(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var restarted []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodPost && r.URL.Path == "/restart" {
|
||||
if _, pass, _ := r.BasicAuth(); pass != "pw" {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
host, _, _ := strings.Cut(r.Host, ":")
|
||||
mu.Lock()
|
||||
restarted = append(restarted, host)
|
||||
mu.Unlock()
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
old := restPort
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "restart\n"}
|
||||
leader := &machine{name: "a", answers: answers}
|
||||
replica := &machine{name: "b", answers: answers}
|
||||
up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running", PendingRestart: true},
|
||||
{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
|
||||
s := Setup{
|
||||
DB: []Node{{Member{"a", "127.0.0.1"}, leader}, {Member{"b", "localhost"}, replica}},
|
||||
Status: func(context.Context, []string) (Cluster, error) { return up, nil },
|
||||
Wait: time.Second,
|
||||
}
|
||||
if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Join(restarted, " ") != "localhost 127.0.0.1" {
|
||||
t.Errorf("restarted %v; the replica first, then the leader", restarted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMemberLeftPendingARestartIsRestartedOnTheNextRun(t *testing.T) {
|
||||
// A run that stopped after writing the configuration, or restarted before
|
||||
// Patroni had taken it: nothing to change, but PostgreSQL still on the old.
|
||||
var restarted []string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(_ http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/restart" {
|
||||
host, _, _ := strings.Cut(r.Host, ":")
|
||||
restarted = append(restarted, host)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
u, _ := url.Parse(srv.URL)
|
||||
old := restPort
|
||||
restPort, _ = strconv.Atoi(u.Port())
|
||||
defer func() { restPort = old }()
|
||||
|
||||
answers := map[string]string{"cat /proc/meminfo": "MemTotal: 4009848 kB\n", "WANT": "same\n"}
|
||||
up := Cluster{Members: []MemberState{{Name: "a", Role: "leader", State: "running"},
|
||||
{Name: "b", Role: "replica", State: "streaming", PendingRestart: true}}}
|
||||
s := Setup{
|
||||
DB: []Node{{Member{"a", "127.0.0.1"}, &machine{answers: answers}}, {Member{"b", "localhost"}, &machine{answers: answers}}},
|
||||
Status: func(context.Context, []string) (Cluster, error) { return up, nil },
|
||||
Wait: time.Second,
|
||||
}
|
||||
if err := s.memory(context.Background(), Secrets{RestAPI: "pw"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Join(restarted, " ") != "localhost" {
|
||||
t.Errorf("restarted %v; only the member pending", restarted)
|
||||
}
|
||||
}
|
||||
@@ -128,6 +128,9 @@ func (s *Setup) Run(ctx context.Context) (string, error) {
|
||||
if err := s.ensurePatroniTLS(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("patroni api: %w", err)
|
||||
}
|
||||
if err := s.memory(ctx, sec); err != nil {
|
||||
return "", fmt.Errorf("memory: %w", err)
|
||||
}
|
||||
if err := s.database(ctx); err != nil {
|
||||
return "", fmt.Errorf("database: %w", err)
|
||||
}
|
||||
|
||||
@@ -19,6 +19,9 @@ type MemberState struct {
|
||||
State string `json:"state"`
|
||||
Host string `json:"host"`
|
||||
Timeline int `json:"timeline"`
|
||||
// PendingRestart is a parameter changed that PostgreSQL takes only on
|
||||
// restart.
|
||||
PendingRestart bool `json:"pending_restart"`
|
||||
// Lag is how far behind the leader a replica is, in bytes; -1 when
|
||||
// Patroni does not know, which is itself worth an alert.
|
||||
Lag int64 `json:"-"`
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
func TestAnOwnedAdapterIsSeenOnlyInItsPartOfTheTree(t *testing.T) {
|
||||
e := newFakeEngine(t, "qwen2.5-14b-instruct")
|
||||
rt, owner := pricedRouter(t, e.srv.URL, "tone")
|
||||
ctx := t.Context()
|
||||
key, _ := rt.store.KeyByID(ctx, "key-1")
|
||||
team, _ := rt.store.CreateTeam(ctx, mustOrgOf(t, rt), "other-team")
|
||||
other, _ := rt.store.CreateProject(ctx, team, "other")
|
||||
otherSecret := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(otherSecret))
|
||||
if err := rt.store.CreateKey(ctx, store.Key{ID: "key-2", Hash: sum[:], Prefix: "bbbb", Name: "o", ProjectID: other, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rt.storeModels([]catalog.Model{{Name: "qwen2.5-14b-instruct", Aliases: []string{"q14"},
|
||||
Adapters: []catalog.Adapter{{Name: "tone", Source: "/t.gguf", Owner: "project:" + key.ProjectID}}}})
|
||||
h := rt.Handler()
|
||||
ask := func(secret string) int {
|
||||
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions",
|
||||
strings.NewReader(`{"model":"tone","messages":[{"role":"user","content":"hi"}]}`))
|
||||
if secret != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec.Code
|
||||
}
|
||||
if code := ask(owner); code != http.StatusOK {
|
||||
t.Errorf("its own project: %d", code)
|
||||
}
|
||||
if code := ask(otherSecret); code != http.StatusNotFound {
|
||||
t.Errorf("another project: %d", code)
|
||||
}
|
||||
list := func(secret string) string {
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec.Body.String()
|
||||
}
|
||||
if !strings.Contains(list(owner), `"tone"`) || strings.Contains(list(otherSecret), `"tone"`) {
|
||||
t.Errorf("listed: own %s\nother %s", list(owner), list(otherSecret))
|
||||
}
|
||||
// No identity, no owned adapter.
|
||||
if rt.adapterHidden(auth.Principal{}, false, "tone") != true {
|
||||
t.Error("an owned adapter was shown to a caller with no identity")
|
||||
}
|
||||
}
|
||||
@@ -42,7 +42,7 @@ func (r *Router) budgetFallback(p auth.Principal, req *parsedRequest) (accountin
|
||||
// answer than saying the prompt is too large, and it would be done
|
||||
// silently, to a request that was already being quietly moved.
|
||||
canonical, cand := r.resolve(name)
|
||||
if why := checkContextFits(cand, req.PromptTokens, req.MaxTokens); why != "" {
|
||||
if why := checkContextFits(cand, req.contextTokens(), req.MaxTokens); why != "" {
|
||||
continue
|
||||
}
|
||||
res, err := r.acct.Check(p, canonical, req.estimatedTokens())
|
||||
|
||||
@@ -56,6 +56,7 @@ func (r *Router) registerChat(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
func(w http.ResponseWriter, req *http.Request, _ store.User) {
|
||||
web.ReplyRenderer(r.cfg.MaxBodyBytes)(w, req)
|
||||
}))
|
||||
r.registerImagesPage(mux, pages)
|
||||
}
|
||||
|
||||
// chatSend dispatches one turn on behalf of a signed-in person.
|
||||
@@ -256,6 +257,12 @@ func (r *Router) principalFor(req *http.Request, user store.User, asked string)
|
||||
// by name, "big-model" came first and every visitor who pressed Send without
|
||||
// touching the picker got placeholder text.
|
||||
func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
return r.modelsOf(req, user, false)
|
||||
}
|
||||
|
||||
// modelsOf is the models a person may use from a page: those that make
|
||||
// pictures, or those that write text.
|
||||
func (r *Router) modelsOf(req *http.Request, user store.User, pictures bool) []web.ChatModel {
|
||||
principal, err := r.principalFor(req, user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
return nil
|
||||
@@ -283,7 +290,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
if entry.MakesImages() == pictures && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{Name: name, Simulated: entry.Simulated(), Hardware: hardware[name]})
|
||||
}
|
||||
}
|
||||
@@ -299,7 +306,7 @@ func (r *Router) modelsFor(req *http.Request, user store.User) []web.ChatModel {
|
||||
continue
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
if r.permits(principal, name) {
|
||||
if entry.MakesImages() == pictures && r.permits(principal, name) {
|
||||
out = append(out, web.ChatModel{
|
||||
Name: name, Simulated: entry.Simulated(), OnDemand: true,
|
||||
})
|
||||
|
||||
@@ -48,12 +48,23 @@ type Info struct {
|
||||
Peer string
|
||||
PeerCluster string
|
||||
PeerTags bool
|
||||
// Workflow is the ComfyUI workflow sent for an endpoint whose engine is
|
||||
// EngineComfyUI: a built-in's name or the workflow's JSON.
|
||||
Workflow string
|
||||
}
|
||||
|
||||
// EngineExternal marks an endpoint that is a provider outside the cluster.
|
||||
// It publishes no metrics, so it is never scraped and never reads as stale.
|
||||
const EngineExternal = "external"
|
||||
|
||||
// EngineComfyUI marks a ComfyUI server. The router speaks ComfyUI's API to
|
||||
// it, not OpenAI's, and it publishes no metrics either.
|
||||
const EngineComfyUI = "comfyui"
|
||||
|
||||
// Scraped reports whether an engine publishes metrics the router reads. One
|
||||
// that does not is never read as stale for having none.
|
||||
func Scraped(engine string) bool { return engine != EngineExternal && engine != EngineComfyUI }
|
||||
|
||||
// Serves reports whether the replica serves the given model.
|
||||
func (i Info) Serves(model string) bool {
|
||||
return slices.Contains(i.Models, model)
|
||||
@@ -257,7 +268,7 @@ func (r *Registry) Snapshots(model string) []Snapshot {
|
||||
continue
|
||||
}
|
||||
var age time.Duration
|
||||
stale := l.info.Engine != EngineExternal
|
||||
stale := Scraped(l.info.Engine)
|
||||
if !l.scrapedAt.IsZero() {
|
||||
age = now.Sub(l.scrapedAt)
|
||||
stale = age > r.cfg.StaleAfter
|
||||
|
||||
@@ -102,6 +102,8 @@ func (r *Router) Handler() http.Handler {
|
||||
mux.HandleFunc("POST /v1/score", r.handleInference("/v1/score"))
|
||||
mux.HandleFunc("POST /v1/audio/transcriptions", r.handleInference("/v1/audio/transcriptions"))
|
||||
mux.HandleFunc("POST /v1/audio/translations", r.handleInference("/v1/audio/translations"))
|
||||
// Pictures, made by ComfyUI from a workflow the catalog names.
|
||||
mux.HandleFunc("POST "+imagesPath, r.handleInference(imagesPath))
|
||||
// Answered by the guardrails' judge model.
|
||||
mux.HandleFunc("POST /v1/moderations", r.handleModerations)
|
||||
// Codex, and whatever else speaks only the Responses API, translated to
|
||||
@@ -319,7 +321,7 @@ func (r *Router) Handler() http.Handler {
|
||||
}
|
||||
exempt := []string{"/health", "/ready", "/leader", "/", "/about", "/_static/", "/_lang/", "/claim/",
|
||||
"/signup", "/login", "/login/", "/logout", "/profile", "/profile/", "/scim/",
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/org", "/org/", "/setup/"}
|
||||
"/admin", "/admin/", "/chat", "/chat/", "/images", "/images/", "/org", "/org/", "/setup/"}
|
||||
// The docs are public unless the site said otherwise, and then guarded by
|
||||
// the session check in front of them — or, on a gateway with keys and no
|
||||
// accounts to sign in to, by the key like any other path.
|
||||
@@ -391,7 +393,11 @@ type parsedRequest struct {
|
||||
ProfileOverride string
|
||||
PromptPrefix string
|
||||
PromptTokens int
|
||||
MaxTokens int
|
||||
// ContextTokens is what has to fit in one context: the prompt, except
|
||||
// for embeddings, where each input is embedded on its own and the
|
||||
// longest one is what a context must hold.
|
||||
ContextTokens int
|
||||
MaxTokens int
|
||||
|
||||
// ThreadID is the conversation this request continues, empty for the
|
||||
// stateless path every request took before threads existed.
|
||||
@@ -412,9 +418,9 @@ type parsedRequest struct {
|
||||
guardIn guard.Verdict
|
||||
// knowledge is set when passages from knowledge bases were added.
|
||||
knowledge bool
|
||||
// kbOrg is the organization whose passages were added, for the prefix
|
||||
// kbSalt is the collections whose passages were added, for the prefix
|
||||
// cache's salt.
|
||||
kbOrg string
|
||||
kbSalt string
|
||||
// noStore is a policy that keeps nothing of this request.
|
||||
noStore bool
|
||||
// inCluster is a project whose requests may not leave the cluster.
|
||||
@@ -473,18 +479,27 @@ func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||
return
|
||||
}
|
||||
|
||||
// A picture asked of a model that writes text, or the other way
|
||||
// round, is refused before it waits for either.
|
||||
if why := r.imageRouteRefusal(path, parsed.Model, parsed.UpstreamBody); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
}
|
||||
|
||||
// Before authentication work and before admission: a request that
|
||||
// cannot fit should not consume a quota reservation or a queue slot on
|
||||
// its way to being refused.
|
||||
if _, entry := r.resolve(parsed.Model); entry.ContextLength > 0 {
|
||||
if why := checkContextFits(entry, parsed.PromptTokens, parsed.MaxTokens); why != "" {
|
||||
if why := checkContextFits(entry, parsed.contextTokens(), parsed.MaxTokens); why != "" {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error", why)
|
||||
r.m.observeRequest(r.metricModel(parsed.Model), "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if r.hiddenModel(parsed.Model) {
|
||||
p0, authed := auth.FromContext(req.Context())
|
||||
if r.hiddenModel(parsed.Model) || r.adapterHidden(p0, authed, parsed.Model) {
|
||||
// Said as a model that is not there: it is not, for clients.
|
||||
writeErrorCode(w, http.StatusNotFound, "invalid_request_error", "model_not_found",
|
||||
fmt.Sprintf("the model %q does not exist", parsed.Model))
|
||||
@@ -735,6 +750,7 @@ func (r *Router) account(req *parsedRequest, res dispatchResult, status int, too
|
||||
// the truth than charging nothing.
|
||||
u.PromptTokens = int64(req.PromptTokens)
|
||||
}
|
||||
u.Images = int64(res.outcome.Images)
|
||||
r.m.tenant.observe(req, status, u.PromptTokens, u.CompletionTokens, res.outcome.TTFT)
|
||||
r.acct.Record(req.reservation, u)
|
||||
}
|
||||
@@ -824,6 +840,12 @@ func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
|
||||
// plan §14). Measured over the whole prompt, not the routing prefix —
|
||||
// see promptBytes.
|
||||
p.PromptTokens = promptBytes(&b) / 4
|
||||
p.ContextTokens = p.PromptTokens
|
||||
if path == "/v1/embeddings" {
|
||||
if longest, ok := longestInput(b.Input); ok {
|
||||
p.ContextTokens = longest / 4
|
||||
}
|
||||
}
|
||||
|
||||
// Tags are the gateway's: an engine has no use for them, and its logs
|
||||
// are no place for what a client wrote there.
|
||||
@@ -896,6 +918,29 @@ func promptPrefix(b *inferenceBody) string {
|
||||
// bounded head of the conversation; a token estimate wants all of it, and using
|
||||
// one for the other meant TPM limits undercounted exactly the requests that
|
||||
// consume the most, and nothing could tell whether a prompt fit in a context.
|
||||
// longestInput is the length of an embeddings request's longest input, when
|
||||
// input is a list of strings: a batch of texts, each its own context.
|
||||
func longestInput(raw json.RawMessage) (int, bool) {
|
||||
var list []string
|
||||
if len(raw) == 0 || raw[0] != '[' || json.Unmarshal(raw, &list) != nil || len(list) == 0 {
|
||||
return 0, false
|
||||
}
|
||||
n := 0
|
||||
for _, s := range list {
|
||||
n = max(n, len(s))
|
||||
}
|
||||
return n, true
|
||||
}
|
||||
|
||||
// contextTokens is what has to fit in one context; requests built inside
|
||||
// the router set only PromptTokens.
|
||||
func (p *parsedRequest) contextTokens() int {
|
||||
if p.ContextTokens > 0 {
|
||||
return p.ContextTokens
|
||||
}
|
||||
return p.PromptTokens
|
||||
}
|
||||
|
||||
func promptBytes(b *inferenceBody) int {
|
||||
n := 0
|
||||
add := func(raw json.RawMessage) {
|
||||
@@ -1041,7 +1086,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
|
||||
principal, authed := auth.FromContext(req.Context())
|
||||
names := make([]string, 0, len(seen))
|
||||
for n := range seen {
|
||||
if authed && !r.permits(principal, n) || r.hiddenModel(n) {
|
||||
if authed && !r.permits(principal, n) || r.hiddenModel(n) || r.adapterHidden(principal, authed, n) {
|
||||
continue
|
||||
}
|
||||
names = append(names, n)
|
||||
@@ -1087,7 +1132,7 @@ func (r *Router) handleModels(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
entry["capabilities"] = map[string]bool{
|
||||
"tools": m.Capabilities.Tools, "vision": m.Capabilities.Vision,
|
||||
"embeddings": m.Capabilities.Embeddings,
|
||||
"embeddings": m.Capabilities.Embeddings, "images": m.MakesImages(),
|
||||
}
|
||||
data = append(data, entry)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/version"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/web"
|
||||
)
|
||||
|
||||
// /v1/images/generations, OpenAI's shape, made by ComfyUI: the request goes
|
||||
// through the same checks as any other — key, allowlist, limits, admission,
|
||||
// guardrails on its prompt — and only at the replica is it turned into a
|
||||
// workflow, queued and waited on, and its pictures sent back as base64.
|
||||
|
||||
const imagesPath = "/v1/images/generations"
|
||||
|
||||
// imageRequest is what a request for pictures asks. Beyond OpenAI's fields,
|
||||
// those a diffusion model has and DALL·E does not: seed, steps,
|
||||
// negative_prompt and guidance.
|
||||
type imageRequest struct {
|
||||
Prompt string `json:"prompt"`
|
||||
N int `json:"n"`
|
||||
Size string `json:"size"`
|
||||
ResponseFormat string `json:"response_format"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
NegativePrompt string `json:"negative_prompt"`
|
||||
Guidance *float64 `json:"guidance"`
|
||||
|
||||
width, height int
|
||||
}
|
||||
|
||||
// The bounds of a request. A FLUX latent is 1/16 of the picture, so a side
|
||||
// that is not a multiple of 16 is not one it can make; the rest keep one
|
||||
// request from holding a card for the length of a film.
|
||||
const (
|
||||
maxImages = 4
|
||||
minSide = 256
|
||||
maxSide = 2048
|
||||
maxSteps = 100
|
||||
defaultSide = 1024
|
||||
)
|
||||
|
||||
func parseImageRequest(body []byte) (imageRequest, error) {
|
||||
var in imageRequest
|
||||
if err := json.Unmarshal(body, &in); err != nil {
|
||||
return in, err
|
||||
}
|
||||
if strings.TrimSpace(in.Prompt) == "" {
|
||||
return in, errors.New("prompt is required")
|
||||
}
|
||||
switch {
|
||||
case in.N == 0:
|
||||
in.N = 1
|
||||
case in.N < 0 || in.N > maxImages:
|
||||
return in, fmt.Errorf("n is between 1 and %d", maxImages)
|
||||
}
|
||||
switch in.ResponseFormat {
|
||||
case "", "b64_json":
|
||||
case "url":
|
||||
// A URL would have to be served by somebody and kept for some time,
|
||||
// and a picture made from a prompt is the prompt's owner's: it is
|
||||
// not kept here.
|
||||
return in, errors.New(`response_format "url" is not offered: the picture comes back as b64_json and is kept nowhere`)
|
||||
default:
|
||||
return in, fmt.Errorf("response_format %q is not b64_json", in.ResponseFormat)
|
||||
}
|
||||
in.width, in.height = defaultSide, defaultSide
|
||||
if s := strings.TrimSpace(in.Size); s != "" && s != "auto" {
|
||||
w, h, ok := strings.Cut(strings.ToLower(s), "x")
|
||||
wi, err1 := strconv.Atoi(w)
|
||||
hi, err2 := strconv.Atoi(h)
|
||||
if !ok || err1 != nil || err2 != nil {
|
||||
return in, fmt.Errorf("size %q is not WIDTHxHEIGHT", s)
|
||||
}
|
||||
in.width, in.height = wi, hi
|
||||
}
|
||||
for _, side := range []int{in.width, in.height} {
|
||||
if side < minSide || side > maxSide || side%16 != 0 {
|
||||
return in, fmt.Errorf("size %dx%d: each side is a multiple of 16 from %d to %d", in.width, in.height, minSide, maxSide)
|
||||
}
|
||||
}
|
||||
if in.Steps < 0 || in.Steps > maxSteps {
|
||||
return in, fmt.Errorf("steps is between 1 and %d", maxSteps)
|
||||
}
|
||||
if in.Guidance != nil && (*in.Guidance <= 0 || *in.Guidance > 30) {
|
||||
return in, errors.New("guidance is above 0 and at most 30")
|
||||
}
|
||||
return in, nil
|
||||
}
|
||||
|
||||
// imageRouteRefusal says why a request cannot be answered at the path it was
|
||||
// sent to, before it takes a place in a queue: a picture asked of a model
|
||||
// that writes text, or text of one that makes pictures. Empty when it can,
|
||||
// and for a model the catalog does not know, which a hand-configured
|
||||
// endpoint may serve as it likes.
|
||||
func (r *Router) imageRouteRefusal(path, model string, body []byte) string {
|
||||
m, known := r.modelMap()[model]
|
||||
if path == imagesPath {
|
||||
if known && !m.MakesImages() {
|
||||
return fmt.Sprintf("%s does not make pictures", m.Name)
|
||||
}
|
||||
if _, err := parseImageRequest(body); err != nil {
|
||||
return err.Error()
|
||||
}
|
||||
return ""
|
||||
}
|
||||
if known && m.ServesComfyUI() {
|
||||
return fmt.Sprintf("%s makes pictures: ask for them at %s", m.Name, imagesPath)
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// generateImages answers a request at a ComfyUI replica. It returns what the
|
||||
// proxy would: a failure of this replica that another may not have is
|
||||
// retryable, and nothing is written until the pictures are all in hand.
|
||||
func (r *Router) generateImages(ctx context.Context, w http.ResponseWriter, req *parsedRequest, ep endpoint.Info, body []byte) (proxy.Outcome, error) {
|
||||
if req.Path != imagesPath {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s is a ComfyUI server: it answers %s only", ep.ID, imagesPath)
|
||||
}
|
||||
in, err := parseImageRequest(body)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
var upstream struct {
|
||||
Model string `json:"model"`
|
||||
}
|
||||
_ = json.Unmarshal(body, &upstream)
|
||||
tpl, err := comfy.Load(ep.Workflow)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
seed := comfy.RandomSeed()
|
||||
if in.Seed != nil {
|
||||
seed = *in.Seed
|
||||
}
|
||||
p := comfy.Params{
|
||||
Checkpoint: upstream.Model, Prompt: in.Prompt, NegativePrompt: in.NegativePrompt,
|
||||
Width: in.width, Height: in.height, N: in.N, Seed: seed, Steps: in.Steps,
|
||||
}
|
||||
if in.Guidance != nil {
|
||||
p.Guidance = *in.Guidance
|
||||
}
|
||||
workflow, err := tpl.Fill(p)
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, fmt.Errorf("%s: %w", ep.ID, err)
|
||||
}
|
||||
|
||||
if t := r.cfg.Timeouts.Total; t > 0 {
|
||||
var cancel context.CancelFunc
|
||||
ctx, cancel = context.WithTimeout(ctx, t)
|
||||
defer cancel()
|
||||
}
|
||||
start := time.Now()
|
||||
images, err := r.comfy.Generate(ctx, ep.URL, ep.APIKey, workflow)
|
||||
if err != nil {
|
||||
var ce *comfy.Error
|
||||
switch {
|
||||
case errors.As(err, &ce):
|
||||
return proxy.Outcome{Retryable: ce.Retryable}, err
|
||||
case errors.Is(err, context.Canceled) && ctxCauseIsClient(ctx):
|
||||
return proxy.Outcome{}, proxy.ErrClientGone
|
||||
case errors.Is(err, context.DeadlineExceeded):
|
||||
return proxy.Outcome{}, fmt.Errorf("%s made no picture within %s", ep.ID, r.cfg.Timeouts.Total)
|
||||
}
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
|
||||
data := make([]map[string]any, 0, len(images))
|
||||
for _, im := range images {
|
||||
data = append(data, map[string]any{"b64_json": base64.StdEncoding.EncodeToString(im.Data)})
|
||||
}
|
||||
out, err := json.Marshal(map[string]any{"created": time.Now().Unix(), "data": data})
|
||||
if err != nil {
|
||||
return proxy.Outcome{}, err
|
||||
}
|
||||
// The seed that made them, so a picture liked can be made again.
|
||||
w.Header().Set("x-llmgw-seed", strconv.FormatInt(seed, 10))
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusOK)
|
||||
_, _ = w.Write(out)
|
||||
took := time.Since(start)
|
||||
return proxy.Outcome{Wrote: true, Status: http.StatusOK, Bytes: int64(len(out)), TTFT: took, Decode: took,
|
||||
Images: len(images)}, nil
|
||||
}
|
||||
|
||||
// ctxCauseIsClient reports whether a context ended because the client went
|
||||
// away, rather than because a deadline set here passed.
|
||||
func ctxCauseIsClient(ctx context.Context) bool {
|
||||
return !errors.Is(context.Cause(ctx), context.DeadlineExceeded)
|
||||
}
|
||||
|
||||
// registerImagesPage mounts the page that makes pictures, beside the chat
|
||||
// and for the same reasons: a model tried without a client, with the
|
||||
// browser holding a session and never a key, and the request sent down the
|
||||
// one path every API request takes.
|
||||
func (r *Router) registerImagesPage(mux *http.ServeMux, pages *web.AccountPages) {
|
||||
pages.SetImages(func() bool {
|
||||
for _, m := range r.modelMap() {
|
||||
if m.MakesImages() {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
})
|
||||
mux.HandleFunc("GET /images", pages.Guard(func(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
contexts, _ := r.chatContexts(req.Context(), user)
|
||||
current, err := r.pickContext(req.Context(), user, req.URL.Query().Get("context"))
|
||||
if err != nil {
|
||||
current = ""
|
||||
}
|
||||
if current == "" && !r.personalOff {
|
||||
current = personalContext
|
||||
}
|
||||
web.Images(user, r.modelsOf(req, user, true), contexts, current, version.Version)(w, req)
|
||||
}))
|
||||
mux.HandleFunc("POST /images/generate", pages.Guard(r.imagesSend))
|
||||
}
|
||||
|
||||
// imagesSend makes pictures on behalf of a signed-in person.
|
||||
func (r *Router) imagesSend(w http.ResponseWriter, req *http.Request, user store.User) {
|
||||
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||
if err != nil {
|
||||
http.Error(w, "could not read the request", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
var in struct {
|
||||
Model string `json:"model"`
|
||||
Prompt string `json:"prompt"`
|
||||
Size string `json:"size"`
|
||||
N int `json:"n"`
|
||||
Seed *int64 `json:"seed"`
|
||||
Steps int `json:"steps"`
|
||||
NegativePrompt string `json:"negative_prompt"`
|
||||
// Context is what the pictures are charged to, checked as the
|
||||
// chat's is.
|
||||
Context string `json:"context"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &in); err != nil || in.Model == "" {
|
||||
http.Error(w, "a model and a prompt are required", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
principal, err := r.principalFor(req, user, in.Context)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), http.StatusForbidden)
|
||||
return
|
||||
}
|
||||
// Rebuilt from the fields the page has, so it can set nothing else.
|
||||
out := map[string]any{"model": in.Model, "prompt": in.Prompt, "size": in.Size, "n": in.N}
|
||||
if in.Seed != nil {
|
||||
out["seed"] = *in.Seed
|
||||
}
|
||||
if in.Steps > 0 {
|
||||
out["steps"] = in.Steps
|
||||
}
|
||||
if s := strings.TrimSpace(in.NegativePrompt); s != "" {
|
||||
out["negative_prompt"] = s
|
||||
}
|
||||
upstream, err := json.Marshal(out)
|
||||
if err != nil {
|
||||
http.Error(w, "could not build the request", http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
next := req.Clone(auth.WithPrincipal(req.Context(), principal))
|
||||
next.Body = io.NopCloser(bytes.NewReader(upstream))
|
||||
next.ContentLength = int64(len(upstream))
|
||||
next.Header.Set("Content-Type", "application/json")
|
||||
next.Header.Del(knowledgeHeader)
|
||||
r.handleInference(imagesPath)(w, next)
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// imageRouter has a ComfyUI model, flux, beside a text model, and returns
|
||||
// what the ComfyUI server was sent.
|
||||
func imageRouter(t *testing.T) (*Router, *map[string]any, *sync.Mutex) {
|
||||
t.Helper()
|
||||
return imageRouterWith(t, nil)
|
||||
}
|
||||
|
||||
func imageRouterWith(t *testing.T, edit func(*config.Server)) (*Router, *map[string]any, *sync.Mutex) {
|
||||
t.Helper()
|
||||
var mu sync.Mutex
|
||||
var queued map[string]any
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch r.URL.Path {
|
||||
case "/metrics":
|
||||
t.Error("ComfyUI was scraped for metrics")
|
||||
case "/prompt":
|
||||
var b struct {
|
||||
Prompt map[string]any `json:"prompt"`
|
||||
}
|
||||
_ = json.NewDecoder(r.Body).Decode(&b)
|
||||
mu.Lock()
|
||||
queued = b.Prompt
|
||||
mu.Unlock()
|
||||
fmt.Fprint(w, `{"prompt_id":"p1"}`)
|
||||
case "/history/p1":
|
||||
mu.Lock()
|
||||
n := 1
|
||||
if latent, ok := queued["4"].(map[string]any); ok {
|
||||
n = int(latent["inputs"].(map[string]any)["batch_size"].(float64))
|
||||
}
|
||||
mu.Unlock()
|
||||
var images []string
|
||||
for i := range n {
|
||||
images = append(images, fmt.Sprintf(`{"filename":"%d.png","type":"temp"}`, i))
|
||||
}
|
||||
fmt.Fprintf(w, `{"p1":{"status":{"status_str":"success","completed":true},"outputs":{"7":{"images":[%s]}}}}`,
|
||||
strings.Join(images, ","))
|
||||
case "/view":
|
||||
w.Header().Set("Content-Type", "image/png")
|
||||
fmt.Fprint(w, "png-bytes")
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
|
||||
cfg := baseConfig()
|
||||
cfg.Models = []catalog.Model{catModel("chat", nil), {Name: "flux", Source: "external",
|
||||
External: &catalog.External{URL: srv.URL, Model: "flux1-schnell-fp8.safetensors", API: catalog.APIComfyUI}}}
|
||||
if edit != nil {
|
||||
edit(&cfg)
|
||||
}
|
||||
rt := newTestRouter(t, cfg)
|
||||
rt.comfy = &comfy.Client{Poll: time.Millisecond}
|
||||
rt.SetEndpoints(t.Context(), []endpoint.Info{
|
||||
{ID: "external:flux", URL: srv.URL, Models: []string{"flux"}, Engine: endpoint.EngineComfyUI,
|
||||
Pool: "external", Weight: 1, UpstreamModel: "flux1-schnell-fp8.safetensors"},
|
||||
})
|
||||
return rt, &queued, &mu
|
||||
}
|
||||
|
||||
func postImages(t *testing.T, h http.Handler, path string, body any) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
b, _ := json.Marshal(body)
|
||||
req := httptest.NewRequest(http.MethodPost, path, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
func TestAPictureIsMadeByComfyUIAndComesBackAsBase64(t *testing.T) {
|
||||
rt, queued, mu := imageRouter(t)
|
||||
rec := postImages(t, rt.Handler(), imagesPath, map[string]any{
|
||||
"model": "flux", "prompt": "a lighthouse at dusk", "size": "768x512", "seed": 7})
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
var out struct {
|
||||
Data []struct {
|
||||
B64 string `json:"b64_json"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil || len(out.Data) != 1 {
|
||||
t.Fatalf("body %s: %v", rec.Body.String(), err)
|
||||
}
|
||||
if b, _ := base64.StdEncoding.DecodeString(out.Data[0].B64); string(b) != "png-bytes" {
|
||||
t.Errorf("picture = %q", b)
|
||||
}
|
||||
if rec.Header().Get("x-llmgw-seed") != "7" {
|
||||
t.Errorf("seed header = %q", rec.Header().Get("x-llmgw-seed"))
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
wf, _ := json.Marshal(*queued)
|
||||
for _, want := range []string{`"ckpt_name":"flux1-schnell-fp8.safetensors"`, `"text":"a lighthouse at dusk"`,
|
||||
`"width":768`, `"height":512`, `"seed":7`} {
|
||||
if !strings.Contains(string(wf), want) {
|
||||
t.Errorf("%s not in the workflow sent: %s", want, wf)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAndTextAreAskedForAtTheirOwnPaths(t *testing.T) {
|
||||
rt, _, _ := imageRouter(t)
|
||||
h := rt.Handler()
|
||||
for name, c := range map[string]struct {
|
||||
path string
|
||||
body map[string]any
|
||||
want string
|
||||
}{
|
||||
"text of flux": {"/v1/chat/completions", map[string]any{"model": "flux", "messages": []any{}}, "ask for them at " + imagesPath},
|
||||
"picture of chat": {imagesPath, map[string]any{"model": "chat", "prompt": "x"}, "does not make pictures"},
|
||||
"no prompt": {imagesPath, map[string]any{"model": "flux"}, "prompt is required"},
|
||||
"odd size": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "size": "1000x1000"}, "multiple of 16"},
|
||||
"url wanted": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "response_format": "url"}, "b64_json"},
|
||||
"too many at once": {imagesPath, map[string]any{"model": "flux", "prompt": "x", "n": 9}, "n is between"},
|
||||
} {
|
||||
rec := postImages(t, h, c.path, c.body)
|
||||
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), c.want) {
|
||||
t.Errorf("%s: %d %s", name, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelSaysItMakesPictures(t *testing.T) {
|
||||
rt, _, _ := imageRouter(t)
|
||||
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, req)
|
||||
if !strings.Contains(rec.Body.String(), `"images":true`) {
|
||||
t.Errorf("/v1/models: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAComfyUIModelIsCheckedInTheCatalog(t *testing.T) {
|
||||
ok := catalog.Model{Name: "f", Source: "external", External: &catalog.External{
|
||||
URL: "http://10.0.0.147:8188", Model: "flux1-dev.safetensors", API: catalog.APIComfyUI, Workflow: "flux-dev"}}
|
||||
if _, err := catalog.New([]catalog.Model{ok}); err != nil {
|
||||
t.Fatalf("refused: %v", err)
|
||||
}
|
||||
for name, e := range map[string]catalog.External{
|
||||
"unknown api": {URL: "http://a:1", Model: "m", API: "a1111"},
|
||||
"unknown workflow": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Workflow: "sdxl"},
|
||||
"workflow, openai": {URL: "http://a:1", Model: "m", Workflow: "flux-dev"},
|
||||
"comfyui from peer": {URL: "http://a:1", Model: "m", API: catalog.APIComfyUI, Peer: "other"},
|
||||
} {
|
||||
e := e
|
||||
if _, err := catalog.New([]catalog.Model{{Name: "f", Source: "external", External: &e}}); err == nil {
|
||||
t.Errorf("%s: accepted", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPicturesAreChargedByThePictureAndRecorded(t *testing.T) {
|
||||
rt, _, _ := imageRouterWith(t, func(cfg *config.Server) {
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
cfg.Auth.Pricing = map[string]config.Price{"flux": {PromptPer1k: 1000, PerImage: 50000}}
|
||||
cfg.Auth.FlushInterval = 20 * time.Millisecond
|
||||
cfg.Auth.BatchSize = 1
|
||||
})
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "design")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "posters")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
secret := store.KeyPrefix + strings.Repeat("b", 43)
|
||||
sum := sha256.Sum256([]byte(secret))
|
||||
if err := rt.store.CreateKey(t.Context(), store.Key{ID: "key-1", Hash: sum[:], Prefix: "bbbb", Name: "t",
|
||||
ProjectID: project, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
b, _ := json.Marshal(map[string]any{"model": "flux", "prompt": "three posters", "n": 3})
|
||||
req := httptest.NewRequest(http.MethodPost, imagesPath, bytes.NewReader(b))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, req)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
id := rec.Header().Get("X-Request-Id")
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
u, ok, err := rt.store.UsageByRequest(t.Context(), id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if ok {
|
||||
if u.Images != 3 {
|
||||
t.Errorf("recorded %d pictures, want 3", u.Images)
|
||||
}
|
||||
if want := (u.PromptTokens*1000+999)/1000 + 3*50000; u.CostMicros != want {
|
||||
t.Errorf("charged %d micros, want %d (%d prompt tokens and three pictures)", u.CostMicros, want, u.PromptTokens)
|
||||
}
|
||||
return
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("no usage was recorded")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestThePicturesPageOffersPictureModelsAndTheChatTheOthers(t *testing.T) {
|
||||
e := newFakeEngine(t, "chat")
|
||||
rt, _, _ := imageRouterWith(t, func(cfg *config.Server) {
|
||||
cfg.Auth.Enabled = true
|
||||
cfg.Auth.Store = filepath.Join(t.TempDir(), "store.db")
|
||||
})
|
||||
rt.SetEndpoints(t.Context(), append(rt.reg.Infos(), endpoint.Info{ID: "c", URL: e.srv.URL, Models: []string{"chat"}}))
|
||||
org, _ := rt.store.CreateOrg(t.Context(), "acme")
|
||||
team, _ := rt.store.CreateTeam(t.Context(), org, "design")
|
||||
project, err := rt.store.CreateProject(t.Context(), team, "posters")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
user := store.User{ID: "usr-1", ProjectID: project}
|
||||
names := func(pictures bool) []string {
|
||||
var out []string
|
||||
for _, m := range rt.modelsOf(httptest.NewRequest("GET", "/", nil), user, pictures) {
|
||||
out = append(out, m.Name)
|
||||
}
|
||||
return out
|
||||
}
|
||||
if got := names(false); len(got) != 1 || got[0] != "chat" {
|
||||
t.Errorf("the chat offers %v", got)
|
||||
}
|
||||
if got := names(true); len(got) != 1 || got[0] != "flux" {
|
||||
t.Errorf("the pictures page offers %v", got)
|
||||
}
|
||||
|
||||
// The page's request is rebuilt: a field it has no business sending
|
||||
// does not reach the model.
|
||||
body := `{"model":"flux","prompt":"a kite","size":"512x512","n":1,"seed":3,"response_format":"url"}`
|
||||
rec := httptest.NewRecorder()
|
||||
rt.imagesSend(rec, httptest.NewRequest("POST", "/images/generate", strings.NewReader(body)), user)
|
||||
if rec.Code != http.StatusOK || !strings.Contains(rec.Body.String(), "b64_json") {
|
||||
t.Fatalf("%d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if rec.Header().Get("x-llmgw-seed") != "3" {
|
||||
t.Errorf("seed %q", rec.Header().Get("x-llmgw-seed"))
|
||||
}
|
||||
}
|
||||
@@ -17,7 +17,7 @@ func TestSessionPagesAreNotAskedForAnAPIKey(t *testing.T) {
|
||||
h := rt.Handler()
|
||||
// The /admin pages are mounted only with a cluster view, which this
|
||||
// router has none of; these are the session pages it does mount.
|
||||
for _, path := range []string{"/profile", "/chat", "/org", "/org/org-x"} {
|
||||
for _, path := range []string{"/profile", "/chat", "/images", "/org", "/org/org-x"} {
|
||||
w := httptest.NewRecorder()
|
||||
h.ServeHTTP(w, httptest.NewRequest(http.MethodGet, path, nil))
|
||||
if strings.Contains(w.Body.String(), "api key") {
|
||||
|
||||
@@ -224,6 +224,8 @@ type Outcome struct {
|
||||
Decode time.Duration
|
||||
// Usage is the engine-reported token usage, if it was present.
|
||||
Usage *Usage
|
||||
// Images is how many pictures were made, by an endpoint that makes them.
|
||||
Images int
|
||||
// Bytes is how many bytes were relayed to the client.
|
||||
Bytes int64
|
||||
// Reply is the assistant's text, when CaptureReply asked for it. Truncated
|
||||
|
||||
+19
-5
@@ -10,6 +10,7 @@ import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -225,7 +226,14 @@ func (r *Router) retrieve(ctx context.Context, w http.ResponseWriter, req *parse
|
||||
req.UpstreamBody = body
|
||||
req.PromptTokens = len(body) / 4
|
||||
req.knowledge = true
|
||||
req.kbOrg = e.OrgID
|
||||
// The collections searched, which is who else could have had these
|
||||
// passages added: the prefix cache is shared among exactly them.
|
||||
ids := make([]string, len(colls))
|
||||
for i, c := range colls {
|
||||
ids[i] = c.ID
|
||||
}
|
||||
sort.Strings(ids)
|
||||
req.kbSalt = strings.Join(ids, ",")
|
||||
// Set now: the proxy writes the headers when the answer starts.
|
||||
w.Header().Set(sourcesHeader, strings.Join(sources, ","))
|
||||
result := "found"
|
||||
@@ -268,6 +276,10 @@ func splitList(v string) []string {
|
||||
// ordinary path, so the caller's limits, budget and guardrails apply to it
|
||||
// and it is counted to them.
|
||||
func (r *Router) embedFor(ctx context.Context, p auth.Principal, model, text string) ([]float32, error) {
|
||||
// A question, not a document: what the model wants before a query.
|
||||
if _, m := r.resolve(model); m.QueryPrefix != "" {
|
||||
text = m.QueryPrefix + text
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"model": model, "input": []string{text}})
|
||||
hreq, err := http.NewRequestWithContext(auth.WithPrincipal(ctx, p), http.MethodPost, "/v1/embeddings", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
@@ -429,14 +441,16 @@ func (r *Router) kbDefaults(ctx context.Context, p auth.Principal, project strin
|
||||
return names
|
||||
}
|
||||
|
||||
// withCacheSalt sets vLLM's cache_salt to one derived from an organization:
|
||||
// the same for its requests, so they still share a cache among themselves.
|
||||
func withCacheSalt(body []byte, org string) []byte {
|
||||
// withCacheSalt sets vLLM's cache_salt to one derived from key — the
|
||||
// collections a request's passages came from — so requests that can see the
|
||||
// same passages still share a cache, and nobody else's do: not another
|
||||
// organization's, nor another team's of the same one.
|
||||
func withCacheSalt(body []byte, key string) []byte {
|
||||
var top map[string]json.RawMessage
|
||||
if json.Unmarshal(body, &top) != nil {
|
||||
return body
|
||||
}
|
||||
sum := sha256.Sum256([]byte("llmgw-kb\x00" + org))
|
||||
sum := sha256.Sum256([]byte("llmgw-kb\x00" + key))
|
||||
top["cache_salt"], _ = json.Marshal(hex.EncodeToString(sum[:16]))
|
||||
out, err := json.Marshal(top)
|
||||
if err != nil {
|
||||
|
||||
@@ -9,7 +9,9 @@ import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/kb/kbsync"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
@@ -52,9 +54,9 @@ func TestPassagesJoinTheLeadingSystemMessage(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheCacheSaltIsAnOrganizations(t *testing.T) {
|
||||
func TestTheCacheSaltFollowsTheCollections(t *testing.T) {
|
||||
body := []byte(`{"model":"m","messages":[],"temperature":0}`)
|
||||
a1, a2, b := withCacheSalt(body, "org-a"), withCacheSalt(body, "org-a"), withCacheSalt(body, "org-b")
|
||||
a1, a2, b := withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a,kb_b"), withCacheSalt(body, "kb_a")
|
||||
var x, y struct {
|
||||
Salt string `json:"cache_salt"`
|
||||
T float64 `json:"temperature"`
|
||||
@@ -202,7 +204,7 @@ func mustID(t *testing.T, s *store.DB, name string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func TestAVLLMReplicaGetsTheOrganizationsSaltOnlyWithPassages(t *testing.T) {
|
||||
func TestAVLLMReplicaGetsASaltOnlyWithPassages(t *testing.T) {
|
||||
dsn := os.Getenv("LLMGW_TEST_POSTGRES")
|
||||
if dsn == "" {
|
||||
t.Skip("set LLMGW_TEST_POSTGRES to a pgvector database to run this")
|
||||
@@ -253,3 +255,25 @@ func TestAVLLMReplicaGetsTheOrganizationsSaltOnlyWithPassages(t *testing.T) {
|
||||
t.Errorf("a salt without passages:\n%s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAQuestionIsEmbeddedWithTheModelsQueryPrefix(t *testing.T) {
|
||||
// Qwen3-Embedding and its like are trained with an instruction before
|
||||
// the query, never before the documents: the documents come through
|
||||
// /v1/embeddings as they are; the question gets the prefix here.
|
||||
e := newFakeEngine(t, "emb")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
|
||||
m := catModel("emb", nil)
|
||||
m.Capabilities.Embeddings = true
|
||||
m.QueryPrefix = "Instruct: retrieve passages\nQuery: "
|
||||
cfg.Models = []catalog.Model{m}
|
||||
rt := newTestRouter(t, cfg)
|
||||
// The test engine answers with the input's length in bytes and words.
|
||||
v, err := rt.embedFor(t.Context(), auth.Principal{}, "emb", "kaç gün izin var")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := "Instruct: retrieve passages\nQuery: kaç gün izin var"
|
||||
if int(v[0]) != len(want) || int(v[1]) != len(strings.Fields(want)) {
|
||||
t.Errorf("embedded %v, not %q", v, want)
|
||||
}
|
||||
}
|
||||
+53
-12
@@ -29,6 +29,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/accounting"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/engines"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/auth"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/comfy"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
@@ -53,7 +54,9 @@ type Router struct {
|
||||
reg *endpoint.Registry
|
||||
poller *scrape.Poller
|
||||
px *proxy.Proxy
|
||||
m *metrics
|
||||
// comfy asks ComfyUI replicas for pictures.
|
||||
comfy *comfy.Client
|
||||
m *metrics
|
||||
// adm is nil when admission control is disabled.
|
||||
adm *admission.Controller
|
||||
|
||||
@@ -311,6 +314,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
kv: kvindex.New(),
|
||||
kvClient: newKVClient(),
|
||||
m: metrics,
|
||||
comfy: &comfy.Client{HTTP: &http.Client{Transport: &http.Transport{DialContext: (&net.Dialer{Timeout: cfg.Router.Timeouts.Dial}).DialContext}}},
|
||||
px: proxy.New(proxy.Config{
|
||||
DialTimeout: cfg.Router.Timeouts.Dial,
|
||||
ResponseHeaderTimeout: cfg.Router.Timeouts.ResponseHeader,
|
||||
@@ -370,7 +374,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
prices := make(map[string]accounting.Price, len(cfg.Auth.Pricing))
|
||||
for name, p := range cfg.Auth.Pricing {
|
||||
prices[name] = accounting.Price{
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k,
|
||||
PromptPer1k: p.PromptPer1k, CompletionPer1k: p.CompletionPer1k, PerImage: p.PerImage,
|
||||
}
|
||||
}
|
||||
r.recorder = accounting.NewRecorder(st, r.auth, accounting.RecorderConfig{
|
||||
@@ -386,6 +390,7 @@ func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||
accounting.NewPricing(prices, accounting.Price{
|
||||
PromptPer1k: cfg.Auth.DefaultPrice.PromptPer1k,
|
||||
CompletionPer1k: cfg.Auth.DefaultPrice.CompletionPer1k,
|
||||
PerImage: cfg.Auth.DefaultPrice.PerImage,
|
||||
}),
|
||||
r.recorder, log)
|
||||
r.threads = r.store
|
||||
@@ -487,6 +492,7 @@ func toInfos(eps []config.Endpoint) []endpoint.Info {
|
||||
Pool: e.Pool,
|
||||
Weight: e.Weight,
|
||||
Priority: e.Priority,
|
||||
Workflow: e.Workflow,
|
||||
})
|
||||
}
|
||||
return out
|
||||
@@ -713,7 +719,7 @@ func (r *Router) storeModels(models []catalog.Model) {
|
||||
// is applied to the request on the way there.
|
||||
for i, a := range entry.Adapters {
|
||||
m[a.Name] = entry
|
||||
ads[a.Name] = adapterRef{base: entry.Name, index: i, scale: a.EffectiveScale(),
|
||||
ads[a.Name] = adapterRef{base: entry.Name, owner: a.Owner, index: i, scale: a.EffectiveScale(),
|
||||
limit: entry.MaxLoadedAdapters, path: engines.RuntimeLoRAPath(a.Name, a.Source)}
|
||||
}
|
||||
}
|
||||
@@ -735,7 +741,9 @@ func (r *Router) storeModels(models []catalog.Model) {
|
||||
|
||||
// adapterRef is where an adapter runs and how it is applied.
|
||||
type adapterRef struct {
|
||||
base string
|
||||
base string
|
||||
// owner is who sees it (catalog.Adapter.Owner); empty is everyone.
|
||||
owner string
|
||||
index int
|
||||
scale float64
|
||||
// limit is the model's max_loaded_adapters: above zero, the adapter
|
||||
@@ -757,6 +765,33 @@ func (r *Router) adapter(name string) (adapterRef, bool) {
|
||||
return adapterRef{}, false
|
||||
}
|
||||
|
||||
// adapterHidden reports whether name is an adapter its owner keeps from this
|
||||
// caller: one outside the part of the tree it belongs to, or one with no
|
||||
// identity at all. Such a caller is told there is no such model.
|
||||
func (r *Router) adapterHidden(p auth.Principal, authed bool, name string) bool {
|
||||
ad, ok := r.adapter(name)
|
||||
if !ok || ad.owner == "" {
|
||||
return false
|
||||
}
|
||||
if !authed {
|
||||
return true
|
||||
}
|
||||
scope, id, _ := strings.Cut(ad.owner, ":")
|
||||
switch scope {
|
||||
case "org":
|
||||
return p.OrgID != id
|
||||
case "team":
|
||||
return p.TeamID != id
|
||||
case "project":
|
||||
return p.ProjectID != id
|
||||
case "user":
|
||||
return p.UserID != id
|
||||
case "key":
|
||||
return p.KeyID != id
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// LastUsed reports when each endpoint was last sent a request, satisfying
|
||||
// control.UsageSource so the control plane can tell a busy replica from one
|
||||
// holding its cards out of habit.
|
||||
@@ -1090,13 +1125,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
if isAdapter && !onDemand && dec.Endpoint.Engine != "vllm" {
|
||||
body = withAdapter(body, model, ad.index, ad.scale)
|
||||
}
|
||||
// Passages of one organization's documents never share cached
|
||||
// blocks with another's: a prefix found warm would say what another
|
||||
// tenant asked about, by how fast its answer started. vLLM keys its
|
||||
// prefix cache by this salt; other engines are not sent a field
|
||||
// they may refuse.
|
||||
if req.kbOrg != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
|
||||
body = withCacheSalt(body, req.kbOrg)
|
||||
// Passages share cached blocks only with requests that could have
|
||||
// been given the same ones: a prefix found warm would otherwise say
|
||||
// what another tenant asked about, by how fast its answer started.
|
||||
// vLLM keys its prefix cache by this salt; other engines are not
|
||||
// sent a field they may refuse.
|
||||
if req.kbSalt != "" && dec.Endpoint.Engine == "vllm" && req.ContentType == "" {
|
||||
body = withCacheSalt(body, req.kbSalt)
|
||||
}
|
||||
// What computes the answer, said before the proxy writes the headers.
|
||||
// A model can have replicas on cards and on CPUs at once, and the two
|
||||
@@ -1133,7 +1168,13 @@ func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *pa
|
||||
}
|
||||
// A filter per attempt, for the model this attempt is served by.
|
||||
filter := r.outputFilter(uctx, req, model, &preq)
|
||||
out, perr := r.px.Do(uctx, w, preq)
|
||||
var out proxy.Outcome
|
||||
var perr error
|
||||
if dec.Endpoint.Engine == endpoint.EngineComfyUI {
|
||||
out, perr = r.generateImages(uctx, w, req, dec.Endpoint.Info, body)
|
||||
} else {
|
||||
out, perr = r.px.Do(uctx, w, preq)
|
||||
}
|
||||
releaseAdapter()
|
||||
uspan.SetAttributes(attribute.Int("http.response.status_code", out.Status))
|
||||
endSpan(uspan, perr)
|
||||
|
||||
@@ -783,3 +783,28 @@ func TestAStoppedServiceAnswers503AndStillListsModels(t *testing.T) {
|
||||
t.Errorf("started again, still refused: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestABatchOfEmbeddingsIsCheckedTextByText(t *testing.T) {
|
||||
// llmgw-kb sends many passages in one request; each is embedded on its
|
||||
// own, so their sum over the context is no reason to refuse them. One
|
||||
// passage over the context still is.
|
||||
e := newFakeEngine(t, "emb")
|
||||
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"emb"}})
|
||||
m := catModel("emb", nil)
|
||||
m.ContextLength = 2048
|
||||
cfg.Models = []catalog.Model{m}
|
||||
rt := newTestRouter(t, cfg)
|
||||
send := func(input []string) *httptest.ResponseRecorder {
|
||||
b, _ := json.Marshal(map[string]any{"model": "emb", "input": input})
|
||||
rec := httptest.NewRecorder()
|
||||
rt.Handler().ServeHTTP(rec, httptest.NewRequest(http.MethodPost, "/v1/embeddings", bytes.NewReader(b)))
|
||||
return rec
|
||||
}
|
||||
passage := strings.Repeat("x", 6000) // about 1500 tokens
|
||||
if rec := send([]string{passage, passage, passage, passage}); rec.Code == http.StatusBadRequest {
|
||||
t.Errorf("four passages that each fit were refused: %s", rec.Body.String())
|
||||
}
|
||||
if rec := send([]string{"short", strings.Repeat("x", 40_000)}); rec.Code != http.StatusBadRequest {
|
||||
t.Errorf("a passage over the context was let through: %d", rec.Code)
|
||||
}
|
||||
}
|
||||
@@ -96,8 +96,8 @@ func (p *Poller) Sync(ctx context.Context) {
|
||||
|
||||
want := make(map[string]endpoint.Info, len(infos))
|
||||
for _, in := range infos {
|
||||
if in.Engine == endpoint.EngineExternal {
|
||||
continue // a provider elsewhere publishes no /metrics
|
||||
if !endpoint.Scraped(in.Engine) {
|
||||
continue // a provider elsewhere, or ComfyUI, publishes no /metrics
|
||||
}
|
||||
want[in.ID] = in
|
||||
}
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PutClusterState keeps a value under a key, for machines that read the
|
||||
// store rather than a control plane; it reports whether it changed.
|
||||
func (s *DB) PutClusterState(ctx context.Context, key, value string) (bool, error) {
|
||||
if old, err := s.ClusterState(ctx, key); err == nil && old == value {
|
||||
return false, nil
|
||||
}
|
||||
_, err := s.db.ExecContext(ctx, `INSERT INTO cluster_state (key, value, updated_at) VALUES (?, ?, ?)
|
||||
ON CONFLICT (key) DO UPDATE SET value = excluded.value, updated_at = excluded.updated_at`,
|
||||
key, value, millis(time.Now()))
|
||||
return err == nil, err
|
||||
}
|
||||
|
||||
// ClusterState reads a value; ErrNotFound when there is none.
|
||||
func (s *DB) ClusterState(ctx context.Context, key string) (string, error) {
|
||||
var v string
|
||||
err := s.db.QueryRowContext(ctx, `SELECT value FROM cluster_state WHERE key = ?`, key).Scan(&v)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return "", ErrNotFound
|
||||
}
|
||||
return v, err
|
||||
}
|
||||
@@ -73,6 +73,9 @@ var copyOrder = []string{
|
||||
"eval_runs",
|
||||
"eval_cases",
|
||||
"federation_peers",
|
||||
// Before their artifacts, which point at them.
|
||||
"train_jobs",
|
||||
"train_artifacts",
|
||||
}
|
||||
|
||||
// notCopied are tables deliberately left behind.
|
||||
@@ -83,7 +86,8 @@ var copyOrder = []string{
|
||||
// the exact moment somebody is watching a migration and wondering why nothing
|
||||
// is being placed.
|
||||
var notCopied = map[string]string{
|
||||
"leadership": "a lease on a running process, meaningless in a new store",
|
||||
"leadership": "a lease on a running process, meaningless in a new store",
|
||||
"cluster_state": "worked out again by the leader from the settings and the machines, within minutes",
|
||||
}
|
||||
|
||||
// CopyResult is how many rows each table received.
|
||||
|
||||
Loaded 100 of 136 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user