When notifications were last sent is not worth carrying: at worst one is
sent again an hour early.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each kept its own hour of quiet per event, so a budget spent on a team
both control planes served was announced by each. An event is now claimed
in the store before it is sent (notify_sent, migration 0043; one statement,
so two asking at once cannot both win), and only the claimer sends it. A
store that cannot be asked sends it anyway.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cookie that carries a sign-in across the provider was signed with a
key each process made for itself, so with two control planes behind a
balancer a callback reaching the other one was refused as tampered — half
of all sign-ins. The key is now the store's oidc-state secret, sealed like
the others and the same on every control plane.
Changelog: the fixes since v0.14.2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The five-minute, hourly and daily sums gain images (migration 0042), and
the CSV and JSON exports a column for them: once request rows were past
their retention, an export said what pictures cost but not how many there
were. The pictures made before are added to the sums from their rows.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines wrote to a pipe of the agent's. An agent that restarted and
adopted them left that pipe with no reader, and an engine's next write
past its buffer failed with EPIPE — in a Python engine, inside whatever it
was reporting on. Each process now writes to its own file in the agent's
work directory (0600), followed into the shard's log; an adopted engine's
file is read from the start, so its output from before the restart is in
its log too. A file past 64 MB is emptied in place; the files go with the
shard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ComfyUI wrote straight to the agent's pipe. An agent that restarts and
adopts the replica leaves that pipe with no reader, and ComfyUI's next
progress bar failed with a broken pipe, failing the workflow. It now
writes to a pipe of the gate's, which reads for as long as ComfyUI writes
and passes on what it can.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engines.comfyui in the cluster's settings has every NVIDIA node build
ComfyUI: the release's source archive from GitHub, checked against the
digest the recipe holds — GitHub publishes none for a tag's archive, so a
version without one is refused — unpacked beside a Python environment
whose dependencies are its requirements file resolved as PyPI stood the
day after the release. torch is PyPI's CUDA 13 build, or PyTorch's CUDA
12.9 one on an older driver, and the install is checked on the card.
A requirements file that names anything but PyPI packages is refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A control plane that has just started or become leader has heard no
requests, and read the on-demand replicas its nodes report as wanted by
nobody: it drained every one of them at every failover. An adopted replica
of an on-demand model now gets a whole idle timeout from its adoption.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl models said "not placed" for a model with min 0 woken by a request and
serving; it now says "ready (on demand)". And the docs: pictures on the
cluster's own cards, with engine comfyui behind its gate.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ComfyUI lists its base directory's custom_nodes before anything else and
exits when there is none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model whose profiles run engine comfyui is placed on a node's card like
any other, and makes pictures there. ComfyUI asks for no key, so the agent
starts it behind llmgw comfyui-gate: ComfyUI on the loopback, the gate on
the replica's port demanding the replica's key and letting through only
what the router asks — queueing, waiting, fetching, cancelling. The gate
answers the agent's /health and /v1/models, shows ComfyUI the fetched
weights by their own names, and gives each replica its own base directory.
ComfyUI runs as the gate's child, so its card memory is the replica's.
A node offers the engine when agent.yaml names its ComfyUI
(engines.comfyui.dir). The model's comfyui: block names the workflow and,
for a source that is a set of files, the checkpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/images offers the models that make pictures, and the chat no longer
offers them. Like the chat, the browser holds a session and never a key;
the page's request is rebuilt from its own fields and sent down the path
every /v1/images/generations request takes. Size, how many, seed, steps
and a negative prompt; each picture shown with its seed, a download link
and a way to make it again. Nothing is kept but the usage. The menu
offers the page while some model makes pictures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.pricing takes per_image, charged on top of a prompt's tokens; a model
priced only per picture is not free, so a spent budget still stops it.
Each usage row keeps how many pictures were made (migration 0041), and
ctl request and the admin API's request view show them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ComfyUI saves the whole workflow — prompt, checkpoint, nodes — as PNG text
chunks in every picture. They are taken out before the picture is sent.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An external model with api: comfyui is a ComfyUI server. A request for
pictures goes through the same key, allowlist, limit, admission and guard
checks as any other; at the replica it becomes a workflow — a built-in
(flux-schnell, flux-dev) or a JSON file in API format, with {{prompt}},
{{width}}, {{seed}} and the like filled in — queued, waited on, and its
pictures sent back as b64_json. A client that leaves takes its workflow
off the queue or interrupts it. ComfyUI is never scraped for metrics.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Qwen3-Embedding and models like it are trained with an instruction on the
query side only. The prefix goes before the question a knowledge-base
search embeds, at answer time and in the admin API's search; documents
reach /v1/embeddings from llmgw-kb as they are.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It sat in localStorage under one fixed key, so the next person to sign in
on the same browser opened the chat on the last one's conversation. It is
now kept under the account's id, anyone else's is dropped when the page
loads, and signing out sends Clear-Site-Data: "storage" so nothing the
pages kept outlives the session.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw-kb sends many passages in one /v1/embeddings request. Their sizes
were added together and compared with the context, so a batch of passages
that each fit was refused as one over-long prompt. The longest input is
what has to fit; the total still counts for usage and limits.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The leading control plane tells the mirror over its API (PUT /v1/firewall,
kept as .firewall.json beside the models and applied by the same path unit
control planes use), and keeps the state in the store's cluster_state for
members that run no llmgw, which a root timer reads from the member's own
copy. The mirror also lets in the network of every node and control plane
the leader knows. Tried on the PoC: off, on and a network added, on every
kind of machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What llmgw org, keys and users did against the store on a control plane
can now be done remotely and by automation: the tree read, added to,
renamed, moved and emptied; keys issued (the key in the answer, once),
listed and revoked, with the scopes' lifetime limits; people put in and
taken out of the tree, their keys in projects they no longer reach
revoked with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A search asking for more than 10 passages got at most 40 candidates from
HNSW, pgvector's default ef_search. The full-text arm scored every chunk
holding a word of the question before limiting, over the whole
organization's GIN index; it now takes candidates by ts_rank from the
collection's own partial index and scores those. New collections index
halfvec on pgvector 0.7 and later (up to 4000 dimensions).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each Patroni member gets a share of its own RAM in its own config.yml:
shared_buffers a quarter (15% beside a control plane or agent),
effective_cache_size, maintenance_work_mem and work_mem scaled to it, all
clamped. It is read again on every server init --patroni, so a machine
given more memory gets the use of it. A single-machine PostgreSQL init
installs is sized the same way, as a shared machine, instead of a fixed
96MB.
The step itself went in with 85025df by accident, in an early form that
restarted PostgreSQL before Patroni had taken the reload, leaving the old
shared_buffers with a restart pending. It now waits for Patroni to report
the restart pending, restarts with restart_pending (replicas first, the
leader last), and restarts a member it finds pending from an earlier run.
Applied to the PoC: pg01 978MB, ctl02 (control plane beside it) 587MB,
both from 128MB; streaming, lag 0, gateways unaffected.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapter now has an owner (org, team, project, user or key): only callers
there see it, and to anyone else it is a model that does not exist, absent
from /v1/models too. A training job's owner defaults to the one collection
its pairs come from, must be able to read every collection they come from,
and pairs from a knowledge base are never made public. ctl train publish
keeps the GGUF or PEFT files on the mirror, which accepts uploads now, and
sets the owned adapter after the usual architecture check.
The knowledge bases' cache_salt follows the collections searched rather
than the organization, so two teams of one organization no longer share
cached passages.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
pgEnv passed sslmode=verify-full but dropped sslrootcert, so libpq looked
for ~/.postgresql/root.crt and both backup and restore failed against the
Patroni store. sslrootcert, sslcert and sslkey now go along as PGSSL*.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A job names the catalog model the adapter is for, the safetensors weights to
train on, the node and the MiB of a card it needs, and question/answer pairs.
The control plane sends it when the node's agent is there and nothing else
trains on it; the agent trains with its installed vLLM's torch and a pinned
overlay of peft and accelerate (no second environment), as the engines'
user, on the card with that much free — beside an engine, if it leaves that
much — converts the adapter to GGUF with the installed llama.cpp's own
conversion code when the model is served by llama.cpp, and sends what it
made back in pieces, kept in the store to download. Progress, loss and peak
card memory are reported as it goes; a node that goes fails its job.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A chat model of the cluster reads each chunk and writes questions a person
might ask of it with the answers it holds, streamed as NDJSON a chunk at a
time, for people to correct before an adapter is trained on it. A chunk that
gave no pairs says so on its own line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
agents.default and agents.nodes.<id> in the cluster's settings hold
everything agent.yaml can say but the agent's id, control planes, address and
certificate, which stay in the file and are refused there. The control plane
sends each node its settings on register and, a second apart, when they
change; an agent whose settings changed checks they load beside its file,
keeps them and starts again on them, engines serving throughout. An edited
agent.yaml that loads restarts the agent too. History, audit, ctl config, the
console and the portal's settings page cover them as the rest of the
settings.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/admin/settings edits the document llmgw ctl config edit opens: checked as a
control plane would load it, refused when made from a version since replaced
(the edit kept), its history viewed and rolled back as a new version, every
change audited by size and never by contents, since it can hold webhook
secrets. Administrators only; in English and Turkish.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A request given passages carries a cache_salt derived from its organization
when the replica chosen is vLLM, so how fast an answer starts cannot say what
another tenant's documents held. Other engines are not sent the field.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A Documents picker on /chat offers the collections the person may read in
the chosen project, the default when there is one, or none; the sources an
answer was given are listed under it. The page's choice becomes the
X-Llmgw-Knowledge header; one the page sent itself is dropped. In English
and Turkish.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On an organization, team, project or key, the narrowest winning: llmgw ctl
kb default, POST /api/v1/kb/defaults, and the console's Knowledge screen. A
request's own X-Llmgw-Knowledge overrides it and none turns it off. A default
never refuses a request: a collection the caller cannot read is left out, and
a failed search answers without passages. Removing a collection takes it out
of every default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A request naming collections in X-Llmgw-Knowledge has its last question
embedded on the caller's account, the best passages found and added to the
system message, numbered, with an instruction to cite them and to say when
they do not hold the answer; x-llmgw-sources says which were given. Done
before the guardrails judge the request, never served from the answer cache,
and a collection the key may not read is answered as one that does not
exist. Metrics llmgw_kb_retrievals_total and llmgw_kb_retrieval_seconds.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The best text match took the whole text weight, however little of the
question it matched: one everyday word ("most") put the remote-work page
above the travel one for "the most I can spend on a hotel". Now a chunk
gains the weight in proportion to the share of the question's words it
holds, counted once whatever their case.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fused by rank, a full-text match on a question's everyday words outvoted a
far nearer meaning: on the PoC, "how much leave do fathers get" found the
annual-leave page before the paternity one (cosine distance 0.44 against
0.25). Now a chunk scores its similarity to the question plus up to 0.3 for
a full-text match, each search brings four times k candidates, the text
search drops Turkish and English stop words, and an operator's search says
when the embedding model did not answer rather than quietly searching text.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections listed with owner, model, documents and grants; A makes one; on a
row its documents, grant, revoke, a search to try, delete. docs/knowledge.md
says what administrators do and how people use llmgw-kb on Windows, macOS
and Linux: what it reads, what leaves their machine, where its ledger is.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections made for an organization, team, project or person, their
dimension asked of the embedding model; granted and revoked by path, never
outside their organization; their documents listed and removed; and a search
an operator can try. In the OpenAPI document.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Its own binary for Windows, macOS and Linux, carrying no server code. It reads
the files there, cuts them, has the gateway embed them with the person's key
and sends text and vectors for what changed only: a SQLite ledger on the
machine remembers each document's file, size, time, digest and what happened,
so a rerun reads nothing unchanged, a file skipped stays skipped until
--retry, and --delete removes only what this machine sent under the paths
given. --dry-run says what would leave the machine. Commands: login,
collections, sync, status, history, formats.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
What llmgw-kb, run by a person with their own key, reads and writes. A
collection the key may not read answers as one that does not exist; writing
needs ownership or a write grant on the key's chain; a personal key names its
project in X-Llmgw-Project and must be a member of it. Changes are audited by
document name.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML, MHTML, mail with
its attachments, notebooks, subtitles, zip archives and text and source files
in any common encoding, in pure Go; legacy Office formats through LibreOffice
and scanned pages through Tesseract when they are installed. Chunks of about
500 tokens under their headings, overlapping by a paragraph.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collections owned by an organization, team, project or person; grants that
add up down the tree and never cross an organization; chunks in PostgreSQL
with pgvector, partitioned by organization, an HNSW index per collection, and
row-level security as a second line behind each query's filter. Search fuses
nearest-neighbour and full-text ranks. Off on SQLite and without pgvector,
saying why. server init and the HA database setup install and enable it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
judge ask showed Qwen3Guard's controversial verdict as "safe (Violent)"; it
now says controversial, which a policy's judge.controversial counts. The
stray-judge note called a judge no policy asks any more "served as a model
clients use"; a model with role: guard is now said to be a judge no policy
asks.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The replica was stopped and the pass went on to the other models before
placing its replacement, so a model waiting for room took the freed cards.
On the PoC a GPU judge waiting for the only card took it from
qwen3.8-35b-a3b-distill, restarted there after its definition changed,
until the judge was stopped by hand. With the waiting model first in the
catalog it happened one pass later. The replacement is now placed in the
same step that stops the old replica.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every replica's engine demands its own key, and the check sent none, so it
failed with a 401 on every engine in the cluster. The key is asked of the
admin API for the replica at --endpoint, as bench does, or given with --key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
