notify.webhooks existed and setup never asked for one, so a new site
learnt of a parked model or a lost node from its users. `llmgw server
init --webhook URL` (repeatable) and a Notification webhooks field on
both setup screens take them now. Slack's address, Mattermost's and
Rocket.Chat's /hooks/ and Discord's …/slack are sent Slack's text, any
other the event as JSON; slack: or json: in front says which.
Once the control plane runs, they are added to the cluster's settings
beside the webhooks already there, none twice, and setup waits for the
control planes to start again on the change before sending each a test,
saying which answered. A webhook's path is its secret, so it is shown by
host only. A failure is said, not fatal: the cluster serves without its
alerts, and ctl notify test tries again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude Code and the Anthropic SDKs asking for extended thinking got the
answer alone: a reasoning model's reasoning_content was dropped. Asked for,
it comes back as a thinking block before the text — streamed as
thinking_delta events and closed with a signature before the text begins —
and the model is told to think through its chat template
(enable_thinking); disabled tells it not to. Not asked for, it is left out
as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One job at a time, a 3D job waiting for the card held up another model's
job that could have run (seen on the PoC, BACKLOG H7). The leader's worker
claims jobs while fewer than eight run, passing over the models that have
one running, and starts each in its own goroutine; one ending wakes it for
the next. Lease and leader-only rules are as they were.
BACKLOG: H5–H7 marked, H8 (presets) and H9 (what the PoC taught) written.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl model serve served one as a chat model: no --embedding, four slots, a
512-token batch, embeddings unset, and a chat smoke test it could not
pass. A GGUF that pools (<arch>.pooling_type), or --embeddings, gets
--embedding, 32 slots of 1024 tokens unless sized otherwise, a batch that
holds a whole passage, the embeddings capability, and one vector asked for
as its smoke test (canaries and updates too). An import of a hundred
thousand chunks had gone at the pace of four slots with the card a third
busy.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A site's nodes stayed a release behind after its control plane was
upgraded, with nothing printed to say so. ctl upgrade checks every node's
version once the waves are done, fails naming any not on the release, and
says when the control plane has no nodes registered. llmgw upgrade on a
control plane, which looks like the cluster's upgrade and is not, says how
many nodes it left behind and the command that upgrades them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On one card shared by flux and sf3d, a mesh asked for after a picture
waited out the picture model's five minutes of protection and two of idle.
on_demand.evict_after_idle and on_demand.min_lifetime set those for the
cluster, in force on the leader as they change; unset keeps 2m and 5m.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Five pages named /_static/pages.js outright, cached for a day, so a
browser kept the old script after an upgrade — the one holding refreshes
off while a form is filled in among them. They take it through asset, as
the rest do, whose address changes with its contents.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster and node pages swap their live part every few seconds, and the
forms inside it went with it: what was typed, an open choice, the focus.
Filling one in raced the clock. The swap now waits while a field in the
live part has the focus or a form in it has been typed into and not sent;
sending it, or leaving the page, ends that. The mirror page's reload
during a download goes through the same check instead of http-equiv.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
GLBs carried the licence's name, URL and notice in asset.extras.license.
A pipeline that sorts assets by licence needs what a machine matches on:
the id, whether commercial use is allowed, the regions it excludes, the
revenue cap and the attribution. The whole licence is written, as
/v1/models gives it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stable Fast 3D takes the light out of the picture it sees, and a dark
basalt (median 0.17) came back a light stone dimly lit (0.67). The save
node takes albedo_match, the picture and its mask, and scales the base
colour texture in linear light by the ratio of the object's medians, so
marks keep their contrast; texels nothing maps to are left alone. On the
game team's rock it is 0.18.
A workflow's {{name|false}} default now fills a BOOLEAN input with a
boolean rather than the word.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A site whose catalog had no flux-schnell or triposr showed no Images or 3D
page, and nothing said they were one click away. The cluster page now has
a Pictures and 3D models card under the models: each preset, what it
makes, the classes of card here it fits, whether and by whom it was
added, and why the leader has not added it by itself; an Add button, which
says first when it would install ComfyUI on the GPU nodes and restart the
control planes; and the switch for presets.auto. The console's models
screen shows the same under its table: A adds one, asking first in the
same case, and P turns automatic adding on or off.
`ctl model add PRESET` and `ctl model presets` go through the presets API,
so the control plane keeps who added one, and `ctl model presets auto
on|off` flips the setting; against a control plane from before, add sends
the entry as v0.14.4 did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A customer's portal had no Images or 3D pages: nobody had written a
definition for either, on a cluster whose V100 runs both. The leading
control plane now adds flux-schnell and triposr, on demand, once a node
has a card that fits one, the cluster installs ComfyUI, and the catalog
makes nothing of that kind. presets.auto (a live setting, on by default)
turns it off; a preset somebody removes is recorded as declined and not
added back until added again by hand. Each addition is in the audit log.
GET /api/v1/presets says how each preset stands, the ComfyUI the cluster
installs, and why one is not added; POST /api/v1/presets/{name} adds one
(setting engines.comfyui where there is none); PUT /api/v1/presets/auto
switches it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux-schnell and triposr come with llmgw as presets: their workflow,
custom nodes, packages and licence as they were found and tried on the
PoC, and a profile that says the card it needs rather than a class, so the
entry places on any site's cards. TripoSR's workflow is built in, with the
LLMGWPrepareForMesh node that keeps a picture's background out of the
mesh, which the copy in the document had left out.
`ctl model add PRESET` adds one, and has the nodes install ComfyUI when
the cluster's settings name none; `ctl model presets` lists them; the
console's add-model dialog offers them beside a definition file. A
ComfyUI build is now also checked to keep its order on a Turing card, so
a node's installed cu13 is not replaced by the cu126 build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyTorch's CUDA 12.8 and 12.9 builds start at Turing since torch 2.8, as
CUDA 13 does, so on a V100 both ComfyUI builds were installed, 10 GB each,
and failed on the card with "no kernel image". The cu129 build now needs
7.5, and a third, on PyTorch's CUDA 12.6 torch, carries Volta and Pascal.
The open kernel module, NVIDIA's default since 560, writes "Open Kernel
Module for x86_64 580.82.07" in /proc/driver/nvidia/version, and the word
after "Module" was taken as the driver: "for". With the driver unknown,
every build was tried whatever driver it needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VLM is to come first on the one card. Its class makes it take the card
from flux, triposr or sf3d when asked for, but idle eviction ignored class:
a 3D job two minutes into a lull would have taken the card from it. Idle
eviction offers only replicas of no higher class than the model wanting
room, and the wait such a model is told runs to the holder's idle timeout.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
sf3d's first jobs failed with a 401 from stabilityai/stable-fast-3d. The
weights were on the node, and the gate links them into the Hugging Face
cache when told the repository — but the agent passed on only the custom
nodes, so nothing was linked and the node went to the network for a gated
repository.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Removed and added again to give it a licence, qwen3-vl-8b came back loading
by itself: its name was still on the wanted list from before. A model that
leaves the catalog leaves the list too.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 1280x720 PNG sent to qwen3-vl-8b was estimated at 138 thousand prompt
tokens and refused against its 8k context before reaching the engine, which
makes about a thousand of it. Parts are counted by kind: text by its text,
a picture or a sound as a thousand tokens.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stable Fast 3D's texture_baker and uv_unwrapper are C++/CUDA packages no
index has; on the PoC they were built and installed into ComfyUI's
environment by hand. nodes[].build names such directories, and the gate
builds each as a wheel with ComfyUI's Python, for the compute capability
torch reports for the replica's own card, keeps it on the node by the
node's commit, the architecture and the directory, and installs it
beside the replica with the node's requirements. A copy in ComfyUI's
environment does not pin it. A node with no CUDA toolkit says so.
comfyui.huggingface_repo shows the replica's weights to a node that
loads them by repository name, in the engines' Hugging Face cache, so a
gated repository needs no token on the node.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A game sold in the EU cannot ship what a licence excluding the EU made. On
the profile page a person leaves out licences one by one, or by rule:
every licence forbidding commercial use, every licence granting nothing in
a region. The rules cover models added later. The choice reaches every
principal through the resolved limits, so permits() leaves such models out
of /v1/models, requests, jobs, fallbacks and the portal's pickers, and a
request for one is refused 403 licence_excluded saying which choice did.
Kept in user_excluded_licences (migration 0046), copied with the store and
gone with the person.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog entry meant for many sites cannot know their GPU class names:
the PoC has a Quadro RTX 6000, a customer an A100, another an L40S. A
profile may now say gpu: {min_compute_capability, min_memory_gb} in
place of gpu_class, and stands for every class in the cluster that meets
it, named <profile>@<class>, at its own priority; placement still sees
exact classes. ProfileByName and EngineProfile resolve such a name to
the profile on that class, so every caller that maps a replica back to
its profile works unchanged. A ComfyUI profile may carry its own
workflow, checkpoint and nodes, and with its own source an fp8 workflow
and weights go to cards with fp8 and fp16 ones to Turing.
A profile by need is left out of placement until the cluster's cards are
known: with no class it would have read as a CPU profile.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's catalog entry now says what its workflow takes and makes:
comfyui.inputs, each a placeholder of its name with a type (text, int,
float, bool, choice, image, mask, audio, video, file), bounds, choices
and a default; and comfyui.output, which gains video, audio and files
beside pictures and meshes. POST /v1/workflows/runs with {model, inputs}
checks a request against them before a job is written — a required input
left out, an unknown one, a number out of bounds, a song sent as a
picture, another owner's file id are each refused with what was wrong —
and the job worker replays it to a replica, where the files go into
ComfyUI's input directory, the workflow is filled and run, and the first
file of the model's output kind is the job's result. /v1/models lists
each ComfyUI model's workflow, so a client can build its form.
A PNG result carries the licence in an iTXt chunk, as a GLB does in
asset.extras. A file whose server named no type is typed by its name
before its bytes, so a WAV is audio/wav, not audio/wave.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A ComfyUI model may name an environment: the models of one environment
agree on every custom node they both name and on every package range,
so one replica can carry all of them and ComfyUI moves between their
weights itself rather than the card being handed over with a cold start
each time. The catalog refuses an environment that cannot share a
Python, naming both models; Catalog.Environments gives each one's
models, the union of their nodes and their weight sources, for
placement to build on.
A licence gains an id (SPDX where there is one), whether its output may
be used commercially, the regions it excludes, a revenue cap and the
attribution it asks for, so a person's or a site's choice of licences
can be matched without reading prose.
BACKLOG gains section H: ComfyUI as a platform.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
After a control plane restart flux, adopted and asked for by nobody, had no
last use and so was never idle enough to evict: a 3D job waited out its
whole fifteen minutes with no estimate. It counts as used at adoption, as it
is already counted as wanted then; a replica in use is reported by the
router within seconds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 3D job leased by the standby control plane, whose router has no endpoints
and which a job replayed in the process never leaves, was told "loading"
for seven minutes while the leader served the jobs behind it. Jobs and
batches are claimed only while leading.
Five jobs asked for while cuda01's agent restarted failed for good as
"cannot be placed": re-registering, the node had no engines for a few
seconds. A job asks again for five minutes before that is its end.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stable Fast 3D's own save node hands back the GLB as base64 in its UI
output, which the gate does not read. LLMGWSaveMesh writes each mesh to
output/ as a GLB and names it, as a picture's save node does. The
prepare node also gives the object's mask now, which SF3D's sampler
takes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 3D job waiting for the picture model to give up the one card had an empty
progress; its client could not tell waiting from stuck. The control plane
says which model holds the cards and when it lets go, plus the waiting
model's last start; the 503 carries it as headers, and a job keeps it as
progress {state: waiting_gpu, held_by, eta_seconds}.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux evicted to make room for triposr was placed again two seconds later:
still on the wanted list, and first in the catalog, it took back the card
it had just given up, and the 3D job went on waiting. Its last replica
evicted, it leaves the list until it is asked for again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 3D job asked for straight after a picture, on a cluster with one card,
failed at once with "cannot be placed": flux held the card, too freshly
started to evict. The cards coming back when an on-demand model goes idle
makes it a wait, so the request is told to retry and the job waits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LLMGWPrepareForMesh, llmgw's own ComfyUI node, written by the gate into
every replica's custom_nodes from the binary: TripoSR's own preparation
(tsr/utils.py) — the background taken away, by the picture's alpha where
it has one and by rembg (MIT; u2net, Apache 2.0) where it does not, the
object cropped to its bounds, set square at a ratio of the side (0.85),
on grey. A picture sent as it was had a white or black shell round the
object, whatever was behind it made into the mesh.
/v1/models listed only models with a replica up, so one that loads when
asked — triposr, flux-schnell — was invisible to the clients that would
have asked for it. Those are listed now, with loaded: false; every entry
says loaded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/v1/openapi.json is the API clients use — OpenAI's endpoints, Anthropic's,
and what llmgw adds — in OpenAPI 3.1, generated from internal/clientapi's
route table by the generator the admin API's document comes from, now
told each document's title, server, token and error. A test reads the
router's mux.HandleFunc calls and refuses a route described and not
mounted, or mounted and not described. Asked for before there is a key,
so it needs none.
/docs/api shows it in Swagger UI 5.33.1 (Apache 2.0, served from the
gateway as the page's policy requires): Authorize takes a key, Try it out
sends to this gateway. Staff are offered the admin API's document too,
from /docs/api/admin.json, which nobody else is given.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A finished job kept the last word heard from ComfyUI, the node running;
it is now said done before the job is marked completed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/3d beside /images, while some model makes meshes: a picture chosen, the
job asked for through the same path an API client's takes, its progress
shown (waiting for the card, n ahead; making it, the node and percent),
and the mesh in a viewer — model-viewer 4.4.0 (Apache 2.0, with three.js
and lit), served from the gateway since the page's policy loads scripts
from nowhere else — with a download and the model's licence. The mesh is
served to its owner from /3d/jobs/{id}/file; another project's person is
told there is no such job.
A job made from the portal acted for a key that does not exist: the
worker now rebuilds a session's principal from the account that made it,
while it is active and still in the project the job is charged to.
ctl 3d list|get|cancel, over /api/v1/client-jobs (/api/v1/jobs is the
control plane's own work, which ctl jobs shows).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TripoSR's checkpoint loads with transformers 4, and ComfyUI runs 5: of a
package a node names, its range decides, and of everything else what
ComfyUI runs.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
G2: the gate keeps a websocket open to its ComfyUI under a client id of its
own, puts that id on every workflow it lets through — ComfyUI tells only
the client that queued a workflow how it goes — and keeps the last word on
each: the node running, its step of how many. It answers at
/progress/{prompt_id}. While a job's request waits, the router reads the
queue (queued, n ahead) and the gate (running, node, step, percent) and
writes it into the job, which a client reads back. Only into a job of the
same key: a tag naming another's job writes nothing.
G3, the rest of the plumbing: comfyui.nodes names custom nodes by https
repository and full commit, with the PyPI packages they need. The gate puts
each in the replica's own custom_nodes at that commit and installs the
packages beside the replica, constrained to the versions ComfyUI's
environment already has, which it never changes. The catalog refuses a
branch for a commit and anything but package specs for requirements. An
OBJ a node saves leaves the gateway as a GLB, colours kept, so every mesh
can carry its licence.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
G3, the part that needs no model: a ComfyUI model's comfyui.output (or
external.output) says pictures or meshes. A request for a 3D model — a
job at /v1/3d/generations — carries the picture as base64; at the replica
it is uploaded to ComfyUI's input directory (the gate lets /upload/image
through), the workflow is filled in with its name, the seed and the
resolution, and the mesh the workflow saves comes back with its type,
which is the file the job keeps. comfy.Client reads every list of files an
output names, not "images" only, and types a file by its extension where
ComfyUI sends octet-stream.
A model's license (name, url, notice) is said in /v1/models and in the
job, and written into a GLB's asset.extras, since the file travels without
the API; a mesh it cannot be written into is refused. per_mesh in
auth.pricing, usage.meshes in the rows, the sums and the export
(migration 0045). A picture asked of a mesh model, or a mesh of a picture
model, is refused saying which it makes.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The catalog had one source a model, so an embedding model served by vLLM
from its repository on a card and by llama.cpp from a GGUF on CPUs had its
entry swapped by hand halfway through the metallurgy import. A profile's
source is what its engine loads and placement looks for; the model's is
the default, and what updates and canaries move. ctl model add checks
every source, the GGUF check reads each llama.cpp profile's, and ctl
weights collects none a profile loads. An empty source leaves a profile's
fingerprint as it was, so running replicas do not read as stale.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A knowledge-base import sends far more than the cluster can take at once,
and every band waited the same 30 s. A band now has its own queue_ttl, so
one for bulk work waits behind people instead of being turned away.
And a request turned away because the cluster stayed busy — the queue
full (503) or waited as long as its band may (504) — now carries
Retry-After: 10. Without it llmgw-kb, which retries only what says when
to, counted each such passage as failed; that is what turned parts of the
metallurgy import away.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
POST /v1/3d/generations answers 202 with a job; GET says its status,
progress and result file; cancel stops it in the engine's queue too. The
request is kept as sent and replayed by whichever gateway holds the lease
through the path a client's request takes, so it is charged and judged
like any request (tagged job=<id>), and a model still loading is waited
out for as long as the job has. Migration 0044. Nothing makes 3D models
yet; the door refuses every model until the ComfyUI adapter learns meshes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
When notifications were last sent is not worth carrying: at worst one is
sent again an hour early.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each kept its own hour of quiet per event, so a budget spent on a team
both control planes served was announced by each. An event is now claimed
in the store before it is sent (notify_sent, migration 0043; one statement,
so two asking at once cannot both win), and only the claimer sends it. A
store that cannot be asked sends it anyway.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cookie that carries a sign-in across the provider was signed with a
key each process made for itself, so with two control planes behind a
balancer a callback reaching the other one was refused as tampered — half
of all sign-ins. The key is now the store's oidc-state secret, sealed like
the others and the same on every control plane.
Changelog: the fixes since v0.14.2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The five-minute, hourly and daily sums gain images (migration 0042), and
the CSV and JSON exports a column for them: once request rows were past
their retention, an export said what pictures cost but not how many there
were. The pictures made before are added to the sums from their rows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines wrote to a pipe of the agent's. An agent that restarted and
adopted them left that pipe with no reader, and an engine's next write
past its buffer failed with EPIPE — in a Python engine, inside whatever it
was reporting on. Each process now writes to its own file in the agent's
work directory (0600), followed into the shard's log; an adopted engine's
file is read from the start, so its output from before the restart is in
its log too. A file past 64 MB is emptied in place; the files go with the
shard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ComfyUI wrote straight to the agent's pipe. An agent that restarts and
adopts the replica leaves that pipe with no reader, and ComfyUI's next
progress bar failed with a broken pipe, failing the workflow. It now
writes to a pipe of the gate's, which reads for as long as ComfyUI writes
and passes on what it can.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
