-
v0.14.7
Stable
gitea-actions
released this
2026-10-16 20:16:48 +02:00 | 158 commits to main since this releaseThe judge installed by
server initsized for what it does. On a machine
of fewer than eight cores v0.14.6 left every judgment past its budget, so
fail: closedrefused every request. A single-machine install of v0.14.7
on a 4-core VM passed 20 of 20, and on 8 cores 20 of 20 under load.Upgrading.
- No migration.
- Upgrading does not change a judge already in the catalog. v0.14.6
and earlier installed two judges, reserving no cores or one, and 2 or
4 GB. To give them what they need:
llmgw ctl model resources qwen3guard-0.6b --memory 2 --cores 2, then
llmgw ctl restart --model qwen3guard-0.6b. The two then need four cores
on a single machine; below that, a longerlatency_budgetor
fail: open. - On a node whose cores are all reserved,
ctl restart --modelcannot
start the new replica beside the old one and waits. Stop the old engine
process, and the replica comes back with the new figures.
Fixed
- Both judges fit an 8 GB machine again. v0.14.6 had
server initreserve 4 GB a judge replica. That figure came
from one replica at 3.2 GB after a day of traffic. The growth was llama.cpp's prompt cache, up to 8 GB by default,
which a judge never reuses: each text it reads is different. On a 4-core, 8 GB machine the second judge then fitted
nowhere ("has 7 GB of memory with 1 free, needs 4"). The judge now runs with--cache-ram 0, uses about 1.6 GB and
stays there, and reserves 2. - One machine runs one judge, on up to four cores. A second replica on the same machine shares its cores and is
lost with it, so it bought nothing but half the speed of each.server initon a single machine now serves one judge
on up to four cores (2.3 s a judgment where two on two cores took 4.2). Where there will be other nodes, it serves two
on two cores each, as in HA setups.--judge-replicasand the setup's Copies still choose. - A judge gets the two cores it needs. v0.14.6 gave each judge replica one core on machines under eight. A judgment
of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against the 5 s budget, so on a 4-core
machine every judgment timed out and every request was refused. Where there are two judges, each now reserves two
cores (one only below four CPUs), and the one judge of a single machine up to four; the starter has what is left, or
shares when nothing is. A judge on two cores stays close to its budget: give it more cores or a GPU, or raise
latency_budget, where requests must not wait.
Downloads
-
v0.14.6
Stable
gitea-actions
released this
2026-10-16 14:16:48 +02:00 | 164 commits to main since this releaseSetup that says what each part of an install is and lets every one be
chosen and set; three machines that each run the gateway, so any one can
be lost; a cluster that loses its store's primary without losing a
request; one ComfyUI for an environment's models; pictures, 3D and
workflows as MCP tools; alerts asked for at setup; extended thinking.Upgrading.
- No migration.
- Control planes: with
control.lease_ttlunset the lease is now 10 s
(was 15). Upgrade the standbys first and the leader last; a mix of
versions meanwhile is fine. - The store (PostgreSQL): each member is dialled for 3 s before the
next is tried. A store so far away that connecting takes longer needs
connect_timeout=10in its address. - Nodes notice a control plane that died without closing its
connection in 8 s (was 15) once they run this release. - Judges installed by an earlier
server initkeep reserving no cores
and 2 GB. To give them their own:
llmgw ctl model resources qwen3guard-0.6b --memory 4 --cores 2, then
llmgw ctl restart --model qwen3guard-0.6b. - Patroni replaces a lost primary within 20 s on a new cluster. On an
existing one, runserver init --patroni ...again from the machine with
the authority; it changes only that. - A floating address keeps VRRP router id 51 until
server add --vip
runs again, which moves it to one of its own; run it for every standby. - A request whose judge is not there yet waits up to the policy's
latency_budgetfor one beforefail: closedrefuses it.
Added
-
Setup lets every part be chosen, each with its settings, and says what each is.
sudo llmgwon a new machine lists
the parts of an install: the control plane, the store, the model mirror, this machine as a node, the starter model,
pictures and 3D, the guardrails' judge, the firewall, notifications, limits and keys, on-demand turns, an offline
bundle. Beside each is what it will be or what is missing without it. Space turns a part on or off, and Enter opens
its settings in a dialog that explains it at more length. The settings are: which starter model (six Qwen sizes, or
any GGUF on Hugging Face) and its context; the mirror's directory or another machine's mirror; the judge's
categories, copies, wait and what happens when it does not answer; the portal's language; the sign-up form; the
default limits; the longest a key lives; the on-demand waits. A plan page then lists what is installed and what is
left out, with the same install as a command. Every setting is aserver initflag too (--starter,
--judge-fail,--media,--max-key-ttl, ...). -
Three machines are high availability for the gateway, not only the store. With three machines, setup makes the two
database machines control planes too, standing by while the first leads, with an optional floating address
(keepalived) that follows whichever leads. Any one machine can now be lost. By command:server init --patroni ... --standbys --vip 10.0.0.160/24. The mirror's management API answers the three control planes, so a standby that
comes to lead can still fetch models. -
Pictures uploaded to work from:
/v1/filestakespurpose=visionor
user_databesidesbatch, kept as long as job results, so a picture is
sent once and named by itsfile-…id in the MCP tools and a workflow's
inputs. -
One ComfyUI for an environment's models: ComfyUI models in one
comfyui.environment(two or more) share one replica — the catalog
makes it a hidden entry,comfyui-env-<name>, on the first member's
profiles with the most memory any member needs, carrying every member's
weights and custom nodes. Asking for any member starts it; it answers for
each with that member's workflow and checkpoint, and the members get no
replicas of their own. A picture model and a 3D model on one card stop
handing it to each other with a cold start each way. -
Pictures, 3D models, vision and workflows as MCP tools at
/mcp,
beside the memory notes, each offered only where the gateway has a model
for it:generate_image(withconcept,sideandtexturestyles),
generate_mesh(from a picture or a prompt),review_images(a vision
model, to a JSON schema),run_workflow,job_statusand
gateway_status. Each makes its request under the caller's key, so
allowlists, limits, licences and budgets apply and it is charged like
any request; what it makes is kept as the caller's file and answered by
id and download URL. A client that takes a stream gets
notifications/progresswhile a mesh is made and waits up to ten
minutes; one that does not, 50 seconds, then the job id. -
Setup asks where alerts go:
llmgw server init --webhook URL
(repeatable) and a Notification webhooks field on both setup screens.
Slack's, Mattermost's and Rocket.Chat's addresses are sent Slack's text,
any other the event as JSON (slack:orjson:in front says which).
They are added to the cluster'snotify.webhooksbeside those already
there, and each is sent a test once the control plane runs on them. -
Extended thinking on
/v1/messages:thinking: {type: "enabled"}
tells a model that thinks by its chat template to (enable_thinking),
and its reasoning comes back asthinkingblocks before the text,
streamed asthinking_deltaevents;disabledtells it not to. Not
asked for, the reasoning is left out as before. -
How long on-demand models hold a card is a setting, in force on the
leader within seconds:on_demand.evict_after_idle(2m unless set) and
on_demand.min_lifetime(5m). On one card shared by a picture model and
a 3D model, 1m each brings the wait between a picture and its mesh to
about a minute.
Fixed
-
The judges and the starter model divide a machine's CPUs instead of each taking all of them.
server initgave the
judge and the starter no cores of their own, so every llama.cpp engine ran on every core: two judges and the starter
on a 4-core control plane competed, and under load 38 of 60 requests were refused for want of a judgment. Moved to a
node of their own with 2 cores each, the same test refused 2. When this machine is the node, init now gives each judge
replica 1 core (2 from 8 cores) and the starter what is left (at most 8); judges bound for other nodes get 2 each. A
judge also reserves 4 GB, not 2: it uses about 3.2. -
A lost store machine stops costing requests most of a minute. The store's connections try each member for 3 seconds,
not until the kernel gives up, and a connection to a machine that is gone is closed within about 10 seconds (TCP
keepalive, andTCP_USER_TIMEOUTon Linux). Patroni replaces a lost primary within 20 seconds (ttl20,loop_wait
5,retry_timeout5). Runningserver init --patroniagain applies this to an existing cluster. Measured on three
VMs with the primary's power cut: about 45 seconds of hanging requests before. -
Leadership stays put while the store's primary is replaced. Replacing a lost primary takes longer than the
leadership lease lives, so the lease had lapsed for everyone by the time the store answered again. Whoever asked first
took it, as often a standby as the leader. A standby that saw the store out of reach now leaves the lease alone for one
round after it returns, so the leader renews it, and a leader that is truly gone is replaced a round later. With that,
the default lease is 10 seconds (was 15). A node notices a control plane that died without closing its connection in
8 seconds (was 15), so it is on its way to the new leader as that one takes over. -
A judge replica that hangs no longer costs the request. All the tries of a judge request shared the policy's
budget, so a replica that hung used it all and the try on another replica had none left: underfail: closedthe
request was refused while a judge stood idle. When the first replica has not answered by half the budget, the same
question now goes to a replica on another machine too, and the first answer is taken. The first replica is not cut
short, which on a slow machine would refuse answers it was about to give. Replicas on one machine whose CPUs are taken
are slow together, so asking both only doubles the work there: measured that way, as many requests failed as before.
On such a machine the remedy is room, not a second question: judges on more than one node, more cores, or the judge
on a GPU. -
A new leader waits for the judge instead of refusing at once. The moment a control plane takes the lead it knows no
judge yet. When every judge fails at once, they are asked again every 250 ms while the policy'slatency_budgetlasts,
sofail: closedno longer answers every request in those seconds with 503 (about 17 seconds of them in the test). A
judge that is slow is not asked again, and an alarm is raised once per request, not once per try. -
Node discovery for Prometheus listed only the control plane asked and the
store, so a machine that is both was a control plane to one and a store
member to the other — scraped twice — and a standby asked alone lost every
node. Every control plane lists all of them alike; deploy/monitoring's
prometheus.yml says to scrape and ask each. -
A model whose weights and context pool did not fit its card was added
and then failed to make its context on every start, taking the card from
others each time (Apertus 70B with a 65536-token pool, 8B with 131072, on
24 GB).ctl model serveshrinks the pool to what the card holds, saying
so, and refuses weights that do not fit at all; the portal's serve form
refuses such a pool and says the largest that fits. -
Removing weights from the mirror that a catalog model still loads is
refused, naming the model: removed from the mirror alone, the model stayed
in the catalog and its replicas went on loading the node's copy. -
Jobs ran one at a time, so a 3D job waiting for the card held up another
model's job that could have run. They run side by side, one per model at
a time and eight in all, on the leader as before. -
llmgw ctl model serveserved an embedding model as a chat model: no
--embedding, four slots, a 512-token batch,embeddingsunset and a
"say hello" smoke test it could not pass. A model whose GGUF says it
pools (or--embeddings) is served for embedding: 32 slots of 1024
tokens unless sized otherwise, a batch that holds a whole passage, and
one vector asked for as its smoke test. -
llmgw ctl upgradechecks every node's version once the waves are done
and fails naming any not on the release, and says when the control plane
has no nodes registered;llmgw upgradeon a control plane says how many
nodes it left behind and the command that upgrades them. A site's nodes
stayed a release behind with nothing to say so. -
The cluster and node pages swapped their live part every few seconds,
forms and all, so filling one in raced the clock: what was typed, an
open choice and the focus were lost. The refresh now waits while a field
in it has the focus or a form in it has been typed into and not sent;
the mirror page's reload during a download waits the same way.
Downloads
-
v0.14.5
Stable
gitea-actions
released this
2026-10-14 01:34:00 +02:00 | 189 commits to main since this releasePicture and 3D models that add themselves where a card fits them, with a
card on the portal and a screen in the console to see and add them; a
mesh's colour brought to the picture it was made from; and the whole
licence written into every mesh.Upgrading.
- The store gains a table (migration 0047: preset_state); nothing to do.
presets.autois on: once upgraded, the leading control plane adds
flux-schnell and triposr, on demand, where a node's card fits one, the
cluster installs ComfyUI (engines.comfyui) and the catalog makes
nothing of that kind. Nothing is fetched until somebody asks for one.
Turn it off withllmgw ctl model presets auto off.- Nodes need this release for
albedo_match(it is in the save node
the gate puts in each replica); a ComfyUI replica takes it when it
starts again.
Added
- A mesh's base colour brought to its picture's:
LLMGWSaveMeshtakes
albedo_matchwith the picture and its mask, and scales the texture in
linear light to the object's brightness in the picture. Stable Fast 3D
takes the light out of what it sees and read a dark basalt (0.17) as a
light stone dimly lit (0.67); matched, it is 0.17. On sf3d it is the
albedo_matchinput (off unless asked). - Picture and 3D models by themselves: once a node has a card that fits
flux-schnell or triposr and the catalog makes nothing of that kind, the
leading control plane adds the preset on demand — so a new cluster's
portal has its Images and 3D pages without a definition written — where
the cluster installs ComfyUI.presets.autoturns it off; a preset
somebody removes is not added back.GET /api/v1/presetssays how each
stands and why one is not added;POST /api/v1/presets/{name}adds one
(and setsengines.comfyuiif the cluster has none);PUT /api/v1/presets/auto. Each automatic addition is in the audit log. - Presets on the portal and in the console: the cluster page's
Pictures and 3D models card lists each preset — what it makes, the
classes it fits here, whether and by whom it was added, why the leader
has not — with an Add button that says first when it would install
ComfyUI and restart the control planes, and the switch forpresets.auto.
The console's models screen shows the same; A adds one, P flips
the switch.ctl model add PRESETandctl model presetsgo through the
API (andctl model presets auto on|off), falling back to the entry as
v0.14.4 sent it on an older control plane.
Fixed
- A mesh carried only its licence's name, URL and notice; it carries the
whole licence as/v1/modelsgives it — id, commercial use, excluded
regions, revenue cap, attribution — for a pipeline that sorts assets by
licence. - A workflow's
{{name|false}}default filled a BOOLEAN input with the
word rather than a boolean.
Downloads
-
v0.14.4
Stable
gitea-actions
released this
2026-10-13 10:37:18 +02:00 | 194 commits to main since this releaseAny ComfyUI workflow as a job, with typed inputs; picture and 3D models
added by name; profiles that fit whatever card a site has, and ComfyUI on
a V100; licences a machine can match and a person can leave out; a job
waiting for the one card says what holds it. Many fixes to on-demand
models taking turns on one card.Upgrading.
- The store gains a table (migration 0046: user_excluded_licences);
nothing to do. - Nodes need this release for ComfyUI on Volta or Pascal cards (the
cu126 build), for a driver from NVIDIA's open kernel module to be read,
and for a node that loads its weights by repository name (sf3d); a
ComfyUI already installed is kept. - Control planes before nodes: an entry using the built-in
triposr
workflow, or agpu:need instead of a class, is refused by an older
control plane. - Models added with
llmgw ctl model add <preset>take the preset's name;
remove an entry of that name first.
Added
- The client API described: OpenAPI 3.1 at
/v1/openapi.json, no key
needed, generated from a route table a test holds to the router's own;
and/docs/api, Swagger UI (served from the gateway) over it and, for
staff, over the admin API's document. - A job waiting for the card says so:
progressis
{"state":"waiting_gpu","held_by":…,"eta_seconds":…}while another
on-demand model holds it; a direct request's 503 carries
X-Llmgw-Waiting-ForandX-Llmgw-Eta-Seconds. - Any ComfyUI workflow, as a job:
POST /v1/workflows/runswith
{model, inputs}. A model's catalog entry declares its inputs
(comfyui.inputs: text, int, float, bool, choice, image, mask, audio,
video, file, with bounds and defaults) and its output (comfyui.output:
pictures, meshes, video, audio or files); a request is checked against
them before a job is written, file inputs are base64, a data URL or a
/v1/filesid, and/v1/modelslists each model'sworkflowso a
client can build its form. A PNG result carries the licence in an
iTXtchunk, a GLB inasset.extrasas before. - Environments:
comfyui.environmentnames ComfyUI models that agree
on their custom nodes and may share one replica; the catalog refuses
one that cannot. - Profiles that fit the hardware: a profile may say
gpu: {min_compute_capability, min_memory_gb}instead ofgpu_class, and
stands for every class in the cluster that meets it, as
<profile>@<class>; a ComfyUI profile may carry its owncomfyui
workflow, checkpoint and nodes, and any profile its ownsource. One
catalog entry then runs fp8 where the card has it and fp16 on Turing. - Custom nodes that compile:
nodes[].buildnames directories of a
node that are C++/CUDA packages; the gate builds each as a wheel for the
card the replica runs on, keeps it on the node, and installs it beside
the replica.comfyui.huggingface_reposhows a model's weights to a node
that loads them by repository name. - A licence a machine can match:
license.id,commercial,
excluded_regions,revenue_cap_usd,attribution. - On-demand models in
/v1/models, withloadedsaying whether one is
up now; a picture is cut out and put on grey before TripoSR sees it
(LLMGWPrepareForMesh, rembg when it has no alpha). - Licences a person leaves out: on the profile page, by licence, or by
rule — every licence forbidding commercial use, every licence granting
nothing in a region — so a model under one is neither listed to nor
served for that person's keys and sessions (403licence_excluded). - Presets: a picture or 3D model added by name.
llmgw ctl model add flux-schnellortriposradds an entry that comes with llmgw — its
workflow (TripoSR's now built in), custom nodes, packages and licence,
and a profile by card need rather than class — and has the nodes install
ComfyUI when the cluster's settings name none.llmgw ctl model presets
lists them; the console's A on the models screen offers them beside
a definition file.
Fixed
- A model waiting for another on-demand model to give up the card failed at
once as "cannot be placed"; it now waits, and a job with it. - A model evicted for being idle was placed again on the next pass, ahead of
the one it made room for. - A job or batch leased by a standby control plane, whose router has no
endpoints, waited on "loading" for good; they are served only by the
leader. A job told its model fits nowhere asks again for five minutes
before failing, which an agent re-registering makes true for seconds. - An on-demand replica adopted after a restart was never idle enough to
evict, so a model waiting for its card waited out its whole idle timeout;
it counts as used at adoption. - A picture in a prompt was counted as its base64 by the context check, so
one 1280x720 PNG was "138 thousand tokens" and a vision model refused it;
a picture or a sound now counts as about a thousand. - An on-demand model of a higher class was evicted, idle, for a lower-class
one two minutes into a lull; it keeps the card through its idle timeout,
and a model waiting on it is told so. - A model removed from the catalog stayed wanted, so added again it loaded
unasked. - A node that loads its weights by repository name (sf3d) went to Hugging
Face for a gated repository and got a 401: the agent did not hand the
gatecomfyui.huggingface_repo, so the weights on the node were never
linked into its cache. - ComfyUI on a V100. torch's CUDA 12.8 and newer builds have no Volta
kernels, but its cu129 build was taken to run from compute capability
7.0, so on a V100 both ComfyUI builds were installed, about 10 GB each,
and failed with "no kernel image". Cards older than Turing now get
ComfyUI on PyTorch's CUDA 12.6 torch (buildcu126); a Turing or newer
node keeps the build it has. - The open NVIDIA driver read by its version. With the open kernel
module, the default since 560, the agent read the driver as "for" and so
as unknown, and tried every engine build whatever driver it needs.
Downloads
- The store gains a table (migration 0046: user_excluded_licences);
-
v0.14.3
Stable
gitea-actions
released this
2026-10-10 19:22:58 +02:00 | 221 commits to main since this release3D models from a picture, as jobs: made by ComfyUI on the cluster's cards,
with their progress, their licence written into the mesh, a price per mesh
and a page to make them. Also: a profile's own weights, a band for bulk
work, and fixes for control planes behind a balancer and agents that
restart.Upgrading.
- The store gains tables and columns (migrations 0043–0045: notify_sent,
jobs, usage meshes); nothing to do. - Nodes need this release for their engines' output to survive an
agent restart and for ComfyUI's custom nodes and progress; engines
started by an older agent keep its pipe until they are restarted. - A 3D model needs a model that makes them (see install.md, "A model that
makes them"); none is added.
Added
- Jobs: 3D models asked after, not waited for.
POST /v1/3d/generationsanswers 202 with a job id;GET /v1/3d/generations/{id}says its status, progress and, when done, the
file its result is in (/v1/files/{id}/content);POST .../cancelstops
it, in the engine's queue too. The request is replayed by whichever
gateway holds the job's lease through the same path a client's request
takes — key, allowlist, limits, admission, guardrails, accounting — so
it is charged like any request, taggedjob=<id>, and a model still
loading is waited out for as long as the job has (24 h) rather than
refused. A job survives the gateway that took it; results are kept for
router.batch.retain_days. Migration 0044. - ComfyUI models that make meshes (
comfyui.output: meshes). The
picture is uploaded to the replica's ComfyUI and the workflow's
{{image}}; the mesh the workflow saves is the job's file, an OBJ turned
into a GLB. TripoSR (MIT) is the tested model; on Ampere and newer cards
stronger ones can be added the same way. - Custom nodes (
comfyui.nodes): a repository at a full commit, put in
each replica's owncustom_nodesby the gate, with its packages
installed beside the replica, never into ComfyUI's environment. - A job's progress: the gate listens to ComfyUI's websocket and the job
says queued and how many ahead, then the node running, its step and
percent. - A model's
license(name, url, notice), in/v1/models, in the job,
on the portal and in the GLB'sasset.extras. per_meshinauth.pricing; usage rows, sums and exports count
meshes./3don the portal, beside/images: a picture chosen, the job
followed, the mesh in a viewer with a download and its licence; and
llmgw ctl 3d list|get|cancelfor every owner's jobs.- A profile's own
source: the weights it loads instead of the
model's, so an embedding model can run on vLLM from its repository on a
card and on llama.cpp from a GGUF on CPUs as one model. Placement prefers
nodes holding each profile's weights,ctl model addchecks every
source, andctl weightscollects none a profile loads. - A band's own
queue_ttl(router.admission.bands[].queue_ttl): how
long its requests may wait for room, instead of the admission's 30 s. A
band for bulk work — a knowledge-base import — waits behind people rather
than being turned away.
Fixed
- An engine's output no longer breaks when its agent restarts. Engines
wrote to a pipe of the agent's, which an agent that restarted and adopted
them left with no reader; an engine's next large write failed, and in a
Python engine took the request it was reporting on with it. Each process
now writes to its own file in the agent's work directory, and an adopted
engine's log keeps what it wrote before the restart. - Single sign-on behind a balancer. The cookie that carries a sign-in
across the identity provider was signed with a key of each process's own,
so with two control planes a callback that reached the other one was
refused. The key is now a secret in the store, the same on every control
plane. - Notifications sent once by two control planes. Each kept its own
hour of quiet per event, so a budget spent on a team both served was
announced twice. The control plane that claims an event in the store
(migration 0043) sends it; with the store unreachable it is sent anyway. - A request turned away for a busy cluster says when to come back. A
request that waited as long as it may (504) or found the queue full (503)
had noRetry-After, sollmgw-kbcounted the passage as failed instead
of sending it again; both now say 10 seconds. - The usage sums and exports count pictures (
images), as the request
records did; pictures made before are added to the sums.
Downloads
- The store gains tables and columns (migrations 0043–0045: notify_sent,
-
v0.14.2
Stable
gitea-actions
released this
2026-10-10 02:12:09 +02:00 | 239 commits to main since this releasePictures: OpenAI's
/v1/images/generations, made by ComfyUI on the
cluster's own cards. The agents install ComfyUI themselves, the gateway
prices pictures by the picture, and the portal has a page to make them.
Also: on-demand models that survive a failover, and knowledge-base
embeddings fixes.Upgrading.
- The store gains a column (
usage.images); nothing to do. - ComfyUI is installed only where asked:
llmgw ctl config set engines.comfyui 0.39.2has every node with an NVIDIA GPU build it (about
10 GB under/opt/llmgw/engines). Nodes need this release to install it
or run enginecomfyui.
Added
- Pictures at
/v1/images/generations, OpenAI's shape, through the same
key, allowlist, limit, admission and guardrail checks as any request. A
model with profiles on enginecomfyuiis placed on a GPU node like any
other. A ComfyUI server outside the cluster can be an external model with
api: comfyuiinstead. The workflow is a built-in (flux-schnell, the
default, orflux-dev) or a JSON file in ComfyUI's API format, with
{{prompt}},{{width}},{{seed}}and the like where the request's
values go. Requests may also sendseed,steps,negative_promptand
guidance. The seed used comes back inx-llmgw-seed. Pictures come back
asb64_json; ComfyUI's workflow is stripped from each PNG, and a client
that leaves takes its workflow off the queue./v1/modelssaysimages
for such a model. - Engine
comfyui. ComfyUI asks for no key, so the agent starts it
behindllmgw comfyui-gate. ComfyUI listens on the loopback only. The
gate listens on the replica's port, demands the replica's key, and lets
through only what the router asks. The model'scomfyui:block names
the workflow and, for a source that is a set of files, the checkpoint. - The agents install ComfyUI (
engines.comfyui). The release's source is
checked against a digest llmgw holds, since GitHub publishes none, and a
version without one is refused. Its requirements are resolved as PyPI
stood the day after the release, with PyPI's CUDA 13 torch or PyTorch's
CUDA 12.9 one for an older driver, and the result is checked on the card.
A ComfyUI installed by hand is named byengines.comfyui.dirin
agent.yaml. - A price per picture:
per_imageinauth.pricing, on top of the
prompt's tokens. A model priced only per picture is not free, so a spent
budget still stops it. Each usage row keeps how many pictures were made,
shown byllmgw ctl request. /imageson the portal, beside the chat, for the models that make
pictures. The chat no longer offers them. The page offers size, how many,
seed, steps and a negative prompt, and shows each picture with its seed
and a download link. Nothing is kept but the usage.- An embedding model's
query_prefix, put before the question a
knowledge-base search embeds and never before documents, for Qwen3-Embedding
and models trained like it.
Fixed
- An awake on-demand model survived no restart or change of leader of the
control plane. A control plane that had heard no requests read every
on-demand replica its nodes reported as wanted by nobody, and drained them
all. An adopted replica now gets a whole idle timeout. ctl modelssaidnot placedfor an on-demand model that had been woken
and was serving; it now saysready (on demand).- A batch of embeddings was refused as one over-long prompt when the
passages together exceeded the context, though each one fit. The longest
input is what has to fit. - The chat's kept conversation sat in the browser under one key for
everybody, so the next person to sign in on the same browser opened the
last one's conversation. It is kept per account, and signing out clears
what the pages stored.
Downloads
- The store gains a column (
-
v0.14.1
Stable
gitea-actions
released this
2026-10-09 10:11:23 +02:00 | 259 commits to main since this releaseAdapters trained on the nodes from a knowledge base and kept private to
their owners; the cluster's and the agents' settings edited in one place;
the tenancy tree and keys over the admin API and in Terraform; knowledge
bases that hold up with many documents; the firewall setting on every
machine.Upgrading.
- Nodes need this release to follow
agents.*in the cluster's
settings; until a node is upgraded its agent.yaml is what it runs on. - A separate mirror machine and store members that run nothing else
follow the firewall setting once what reads it is installed: runsudo llmgw mirror initon the mirror again with the flags it was set up with,
andllmgw server init --patroniagain for the store's members. A control
plane set up before v0.12.0 needssudo llmgw firewall sync --install. - The store's PostgreSQL memory applies after each member is restarted,
one at a time, the leader last; until then it runs as before. - The store gains tables and columns; nothing to do. Existing knowledge-base
collections get their full-text index when a control plane starts, and
keepvectorstorage; only new collections usehalfvec, and only on
pgvector 0.7 or later. - Cached knowledge-base passages in vLLM are made again once, under the new
cache_salt.
Added
- The cluster's settings on the portal (BACKLOG E4):
/admin/settings
for administrators. It edits the documentllmgw ctl config editopens:
checked as a control plane would load it, refused when made from a
version since replaced, with the history to view and roll back. Changes
are audited by size, never by contents. - The nodes' agent settings in the cluster's settings (BACKLOG E3):
agents.defaultandagents.nodes.<id>hold everything agent.yaml can
say but the agent's id, control planes, address and certificate. They are
sent to the agents as they change; an agent starts again on new settings,
or on an edited agent.yaml, with its engines serving throughout. Settings
that would not load are refused on save, and set aside on a node rather
than keep its agent from starting. Nodes need this release to follow
them. - LoRA adapters trained on the nodes (BACKLOG F4):
llmgw ctl kb dataset
writes question/answer pairs from a collection with a model of the
cluster.llmgw ctl train starttrains an adapter on a node with its
installed vLLM's torch and a small overlay, on a card with enough memory
free (beside an engine, if it leaves that much), converting to GGUF for
llama.cpp.llmgw ctl train publishkeeps it on the mirror (which now
accepts uploads) and makes it the model's adapter. - Adapters with owners. An adapter's
owner(org, team, project, user
or key) is the only part of the tree that sees it; to anyone else it is a
model that does not exist. An adapter trained on a knowledge base belongs
to a part of the tree that may read it, and is never made public. - The knowledge bases' vLLM
cache_saltfollows the collections searched,
not the organization, so two teams of one organization no longer share
cached passages. - Organizations, teams, projects, API keys and memberships in the admin
API:/api/v1/orgs(the tree),POST /api/v1/teamsandprojects,
PATCHto rename or move,DELETEfor what is empty;/api/v1/keysto
issue (the key is in the answer, once), list and revoke;/api/v1/users
and/api/v1/users/{id}/memberships. They were onlyllmgw org,keys
anduserson a control plane before. The Terraform provider manages them
asllmgw_org,llmgw_team,llmgw_project,llmgw_key(the key in a
sensitivesecret) andllmgw_membership. - The firewall setting reaches every machine. A separate mirror and the
store's members followfirewall.enabledandfirewall_networksas nodes
and control planes did: the leading control plane tells the mirror over
its API, and keeps the state in the store for members that run no llmgw,
which a timer on each reads. The mirror also lets in the network of every
node the leader knows.mirror initandserver init --patroniinstall
what follows it; run them again on a cluster set up before.
Changed
- The store's PostgreSQL is sized to its machine's memory. Before, it ran
on PostgreSQL's defaults (128MB cache) under Patroni, or on 96MB on one
machine. Now each member gets a share of its own RAM, a smaller one where
a control plane or agent shares the machine, and the share is read again
every timeserver init --patroniruns. The new cache is applied by
restarting one member at a time, the leader last. - Knowledge-base searches hold up with many documents. New collections
store their index ashalfvecwhere pgvector is 0.7 or later (half the
memory, and embeddings up to 4000 dimensions instead of 2000); older
collections, and clusters on pgvector 0.6, keepvector. Each collection
has its own full-text index, as it has its own vector index, made for
existing collections when the control plane starts. The text arm ranks by
the cheap measure first and scores only those candidates.
Fixed
- A knowledge-base search asking for more than 10 passages
(X-Llmgw-Knowledge-K) got at most 40 candidates per collection from the
vector index, whatever it asked for: pgvector'shnsw.ef_searchis now
raised to the number wanted, up to 1000. llmgw server backupandrestoreagainst a replicated store over TLS
failed looking for~/.postgresql/root.crt: pg_dump was given
sslmode=verify-fullwithout the cluster's authority.
Downloads
- Nodes need this release to follow
-
v0.14.0
Stable
gitea-actions
released this
2026-10-08 16:22:07 +02:00 | 276 commits to main since this releaseKnowledge bases: a company's documents, put in by people from their own
machines, kept apart per tenant, and answered from.Upgrading.
- pgvector on a cluster set up before this release. Knowledge bases
need pgvector, whichserver initand the high-availability database
setup now install, but a rerun on a running cluster does not:- Install
postgresql-<version>-pgvectoron every database machine. - As postgres, run
CREATE EXTENSION vectorin llmgw's database. - Restart the control planes.
Until then everything else works, andllmgw ctl kb listsays why
knowledge bases are off. On a SQLite store they stay off.
- Install
- An embedding model. A collection needs a catalog model with
capabilities.embeddings, for example bge-m3, whose weights are in the
mirror. - The client. People download
llmgw-kbfor their system from this
release.
Added
- Collections and who may read them. A collection is owned by an
organization, team, project or person. Grants add up down the tenancy
tree and never cross an organization. A collection a key may not read is
answered as one that does not exist. Chunks are kept in PostgreSQL with
pgvector, partitioned by organization, with an HNSW index per collection,
and row-level security on the table as a second line behind every
query's own filter. A project, team or organization that owns collections
is not deleted; a person's collections go with their account. llmgw-kb, its own download for Windows, macOS and Linux, run by
people with their own key. It reads documents where they are and sends
text and vectors, never the files, for what changed only; a SQLite ledger
on the machine remembers what it did.- It reads PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML,
MHTML, mail with attachments, notebooks, subtitles, ZIP archives, and
text and source code in any common encoding. - Legacy Office files are read with LibreOffice, and scanned pages with
Tesseract, when those are installed. --dry-runsays what would leave the machine;--deleteremoves only
what that machine sent.
- It reads PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML,
- Answers from documents. A chat request names collections in
X-Llmgw-Knowledge, or its part of the tree has a default (llmgw ctl kb default). The best passages, chosen by meaning plus a little for the
question's words, are added to the system message, numbered, and
x-llmgw-sourcessays which were used. This happens before the
guardrails judge the request, and the answer is never served from the
answer cache. On vLLM replicas the organization'scache_saltkeeps
tenants' prefix caches apart. It works through the Responses and Messages
APIs too. - Administration:
llmgw ctl kb(create, list, grant, revoke, default,
docs, rm, search, delete),/api/v1/kb, the console's Knowledge screen,
and a Documents picker on the portal's chat page. - Metrics
llmgw_kb_retrievals_totalandllmgw_kb_retrieval_seconds;
docs/knowledge.md.
Downloads
- pgvector on a cluster set up before this release. Knowledge bases
-
v0.13.0
Stable
gitea-actions
released this
2026-10-08 11:55:34 +02:00 | 295 commits to main since this releaseThe whole system stopped, started and restarted from one command, a
guardrails judge that no longer refuses ordinary coding requests, and the
rest of the field report from v0.10.6.Upgrading. A cluster keeps the judge its policies name: one installed
before this release goes on asking Llama Guard 3 1B untilllmgw ctl guard judge use qwen3guard-0.6b, which the leader then fetches and serves by
itself, and which replaces the policies' Llama Guard category codes.Added
- A judge on a GPU and one on CPUs together: a policy's
judge.models
is a list asked in turn, the first that answers giving the verdict; the
next is asked when one has no replica serving, fails, or misses its
latency_budget.llmgw ctl guard judge use qwen3guard-4b qwen3guard-0.6bsets the order for every policy with a judge, and
guard judge statussays which judge stands in for which. A judge on a
GPU is served as astandardmodel, so it takes a card only from what
may give one up. - A judge a policy names is fetched if it is missing: the leader
fetches the weights of a judge llmgw knows (qwen3guard-0.6b,-4b,
-8b,llama-guard-3-1b,-8b) into the mirror and serves it, when a
policy names it and the catalog does not have it — and again if it is
removed.guard judge enable --gpuserves Qwen3Guard-Gen 4B on the class
with the most free cards. - The whole system stops, starts and restarts from one command:
llmgw stoptakes every model down and answers every request for one with 503
(system_stopped), keeping the control planes, agents and mirror running
and no model's definition changed;llmgw startbrings every model back.
It is the cluster's settingserving.stopped, live on every control plane.
llmgw restartrestarts every node's agent and every control plane, the
leader last, with the engines serving throughout.llmgw statussays
whether the system serves. One machine's services arellmgw service status|start|stop|restart. - A replica says what it computes on:
ctl replicas(ON), the console
and the admin API showCUDAwith the GPUs its processes hold memory on
and how much, as nvidia-smi sees them, orCPU— for every engine, kept by
the agent across its own and the control plane's restarts. llama.cpp's
layer count is added when it says it, which it does only at-lv 4. ctl engine-check llamacppcomputes a token on the GPU: a small GGUF
(--model, or the smallest in the weights cache) is loaded with every layer
on the GPU at-lv 4and asked for one token; a build whose CUDA backend
does not load fails, by llama.cpp's own account of its layers. With no
--binit checks the build the agent installed.- Engine versions and compute capability on nodes:
ctl nodesshows
llamacpp b10991and6000-24g (7.5); the admin API gives
engine_versionsandcompute_capabilities. A profile'sengine_version
restricts it to nodes whose engine version starts with it, and
explain-placement says which version a node has, or which engines. ctl restart --max-surge Nand--max-unavailable N: that many of a
model's replicas restart at once, beside themselves or stopped first.- Multi-node readiness apart from the node's: doctor gives both, with
fabric findings in a section of their own;node addsays the multi-node
one;ctl nodesshows a declared fabric's adapters that are down. - A forgotten password is set through a link:
/forgotmails one when
the cluster'smailsettings say how (SMTP, sender, the public
base_urlit points at), saying the same whether or not the address has an
account;/resetsets the password once and closes every session. Without
mail,llmgw users reset-linkmakes the link for an operator to hand over.
Fixed
- A replica restarted in place keeps its room: the replica was stopped
and the pass went on to other models before placing its replacement, so a
model waiting for room took the freed cards. On the reference cluster a
GPU judge waiting for the only card took it from the model being
restarted there. The replacement is now placed in the same step. - The judge no longer refuses ordinary coding requests: Llama Guard 3
1B, the judgeserver initinstalled, could not tell "continue", "devam
et" or "how do I kill all child processes in bash" (0.51 to 0.92 unsafe)
from harmful requests (0.97 to 1.00), at any quantisation. The judge is
now Qwen3Guard-Gen 0.6B (formatqwen3guard), which gave the same
ordinary requests 0.15 at the most and the harmful ones 1.00, on CPUs,
in 0.8 GB. Its third verdict, controversial, counts as safe unless the
policy sayscontroversial: unsafe.guard judge use qwen3guard-0.6b
moves an installed cluster to it, and replaces Llama Guard's category
codes, which Qwen3Guard never names, with the baseline's. - A long message is judged rather than refused: a judge read the whole
last user message, and one with a file pasted into it overflowed the
judge's context, so it gave no verdict andfail: closedrefused the
request. It now reads the start and the end, about 6000 characters. - A replica no longer stays "starting" while it serves after a restart:
an agent that registered before its adopted engines had answered their
health check never said they had, so the control plane kept them starting
and routed nothing to them — on the reference cluster, afterllmgw restart, the only replica of a model. The agent now reports each shard
the registration called not ready once it is. - A failover no longer replaces healthy replicas: a standby that took
over placed at once, before the agents had reconnected and said what runs,
so it started a second copy of replicas it did not know of and then drained
the healthy ones as surplus — on the reference cluster the guardrails'
judge and a model's replica, within twenty seconds of a leader restarting.
Placement now waits after a takeover as it does at start.
Changed
- A GPU placement llama.cpp says it cannot compute on is refused when it
warns that it found no usable GPU, even where nvidia-smi cannot be asked;
where it can, holding no memory on its GPUs refuses it as before.
Downloads
- A judge on a GPU and one on CPUs together: a policy's
-
v0.12.2
Stable
gitea-actions
released this
2026-10-06 13:35:16 +02:00 | 316 commits to main since this releaseThe guardrails turned off and on as a whole, and a judge that refuses
ordinary requests explained: the hazard it saw is told to the client and
kept with the request, andllmgw ctl guard judge askshows what it says of
any text. Nothing to do to upgrade; the store gains a column.Added
- The guardrails turn off and on as a whole:
llmgw ctl guard off/
on, which isguard.enabledin the cluster's settings — in force within
seconds, the policies and their attachments kept.ctl status,ctl guard listand the guardrails page say when they are off. llmgw ctl guard judge ask TEXThas the judge read a text and shows
its answer as given and as read — for finding out why ordinary requests
are refused. Nothing of the text is kept, not even in the audit.
Changed
- A judge's refusal names the hazard: the client is told
(judge: S6,Specialized Advice)rather than(judge), and the verdict is kept with
the request (guard_rule):ctl request, the server's log and the
guardrails page show it. Llama Guard 3's codes come with their names,
S9 Indiscriminate Weapons, without a categories list.
Downloads
- The guardrails turn off and on as a whole: