slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The choices screen has a Slurm part, and --slurm writes slurmctld on this
machine's node and slurmd on every node, those that join later too, into
server.yaml. slurm.all_nodes (ctl slurm setup --all-nodes) takes the nodes
from the leader's inventory. No node is lent at install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
max_output on POST /api/v1/models/{name}/context, --max-output on ctl model
context, and a field in the console's Context length… form. Alone it changes
only the catalog's limit, which the router holds max_tokens to, and starts no
replica again; with a context it is set with it, never past it.
Strata's context is now changed like the others', through its setup's
--context, which writes the engine's start script at each start. Neither
needs a model removed and added again any more.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.
ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.
Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.
FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.
Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw ctl model gpus MODEL N [--profile P] [--in-place],
POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of
one node each replica uses. Tensor parallel for vLLM, SGLang and
TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one
rule for the planner and this); the layers split for llama.cpp, its old
--tensor-split dropped. Refused for a count no node has, for cards that
would not hold the weights and the context, and for engines on one card.
The replicas start again beside the old ones; with no room beside them the
command says so, and --in-place stops each first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context
and the console's Context length…: the catalog, the engine's arguments and
the KV memory change together, refused past the trained context or what the
cards hold, then the replicas restart one by one.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mirror add-peer, run on the mirror's machine, sets up a mirror on
another over ssh, kept in step with this one: llmgw there if it has
none, mirror init --peer, this mirror told of it and allowed to be read
by it, and mirror.peers in the cluster's settings. The setup screens
offer one on either database machine, as server init --second-mirror.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
From review. An adapter given by its id is downloaded by vLLM itself, and
a TensorRT-LLM model by its id by TensorRT-LLM, past the node's check and
not held to safetensors: a model with either is refused when it is added,
unless it says allow_pickle, with how to mirror its safetensors instead.
vLLM's own model by id stays allowed: the agent gives it --load-format
safetensors. A test holds every preset to its own pickle check.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At the user's asking: we know what a card's link could be, so a card
linked slower is reported. Each agent reads, apart from the GPU query so
that a driver not knowing a field costs nothing else, the fastest PCIe
generation card and slot allow together, the card's own, and the lanes in
use and the card's most (nvidia-smi; NVML in the nvml build). The current
generation is not compared: it drops while a card idles. A card on fewer
lanes, or a slot of an older generation, is a node warning with the GB/s
it carries against what it could — on the dashboard, the console and in
llmgw doctor, which points at lspci's LnkCap and LnkSta. MIG instances are
said of their card. GPU proto fields 23-26.
The PoC's Quadro RTX 6000 reads Gen3 of Gen3, x16 of x16, idle at Gen1:
nothing to say.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review. A mirror set already on the node is looked at again, not
trusted for having been fetched once; Pickled cuts a query or a fragment
from any source, hf: too; .pt2 is a pickle. The flag check reads a flag as
the engine does: underscores as dashes and any beginning of a refused flag
as that flag, since argparse takes --trust-remote for --trust-remote-code;
SGLang refuses --load-format, which would undo the agent's safetensors.
TripoSR's preset says allow_pickle: without it the preset could not be
added and the presets pass failed on it each time. A shared ComfyUI
environment's hidden entry takes its first member's allow_pickle, as it
takes its source.
The GenAI trace attributes made temperature and top_p typed fields of the
request's parse, so "0.7" as a string was a 400; they are read loosely,
as they are read only for the trace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tool is seen where every level above that names tools names it (store
Effective.AllowsTool). The console and /admin/mcp wrote a level's list as
given, so a team that saw its organization's hr.* through having no list
was given jira.* and then saw neither; and revoking a level's last server
left [] in the store, closed to every tool, shown nowhere.
store.GrantTools and RevokeTools now do both screens' work: a level's first
list starts from the nearest list above it; taking a server away leaves the
others, and a level left naming nothing goes back to no list, or stays
closed when a level above names tools, so it opens to nothing it was kept
from. GrantTools says which levels above still keep the granted tools out
of reach, and both screens say so. The page shows closed levels and member
scopes by name, and offers no revoke for a * grant it would not change. The
console calls the store instead of keys limits with a list built from what
it had loaded, which overwrote changes made since. Found in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With restore: true and the answer cache on, alice's "write to
alice@example.com" and bob's "write to bob@example.com" redact to the
same request, and bob was given the answer kept for alice, her address put
back in it. A request with restore values is now never looked up in the
cache nor kept in it.
The similar-question cache compared the last message's text and blanked
its parts: two pictures asked about in the same words were one question.
A last message with any part that is not text is now reused only when the
request is the same.
Found in review; both tests fail without the fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.
BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.
The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cached_per_1k in a price charges the prompt tokens the answer counts as
cached_tokens at their own rate; zero keeps charging them as prompt tokens.
The count reaches the cost and not a column: CostMicros holds what it
changed. Cached beyond the prompt is clamped, an engine's miscount rather
than a refund. BACKLOG L5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.
BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tool's answer is read by the caller's policies as the prompt it
becomes, so an instruction planted in a ticket is withheld and personal
data redacted; its arguments are read before they leave. ctl mcp add
--rate bounds each caller's calls a minute to a server (migration 0050),
and five failures in a row take a server out for a minute, its tools
off the list.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Its tools say what llmgw ctl does: cluster_status, list_nodes,
list_models, list_replicas, explain_placement, node_logs; none changes
anything. The admin API serves it at POST /api/v1/mcp, and the gateway
answers a server registered as llmgw:admin in its own process, so no
token is kept and each control plane serves it. ctl mcp add takes
--bearer-file for a token sent as Authorization: Bearer.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The setup screen asks for up to two judge machines after the store's
two, and server init takes --judge-node for each. Each is added as a
node labelled llmgw.role: judge, and the judge's profile selects that
label, with cores by init's rule over those machines. A judge beside a
busy control plane refused 38 of 60 requests under load; alone, 2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
People → Sign-in shows single sign-on, Active Directory and SCIM, and
the directory's last sync. E sets up Entra ID from the tenant and client
IDs; D sets up Active Directory, tried from the control plane first:
each server bound, people who may sign in, members of each mapped group.
The secrets go into the store as store:auth/..., sealed by the master
key, and the CA into the settings, so no control plane needs a file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found trying auth.ldap against a Samba DC on the PoC. A CA file a
control plane could not use made every control plane exit, and no
ctl config could then put the setting right: directory sign-in is now
turned off there, and said. `llmgw server check` ran a whole server
that, its ports taken, held the leader's lease for three hours: server
refuses words it does not know and takes its ports before anything else.
The guides now say auth is set with ctl config, not server.yaml.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.ldap checks a username and password against the domain controllers
over LDAPS or StartTLS: found with a service account, checked by binding
as the person. Groups, nested ones included, give staff roles and places
in the tree. Accounts are tied to the objectGUID; the leader reads the
directory every 15 minutes, suspends whom it disabled or removed and
reinstates them after, and refuses a sync that would suspend most.
Entra ID sends no groups for someone in more than 200, only where to
fetch them; that was read as none and took an administrator's role away.
It is now left as it was, and logged. An account made by SCIM or OIDC
can be reinstated again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An external model takes api_key_files, each a file and an optional weight,
in place of api_key_file. Each key is an endpoint of its own
(external:MODEL#1, #2), so the router's scoring, retries and breakers work
per key. A key answering 429 is passed over while its breaker is open. A
key answering 401 is passed over too, and external_key_refused is sent,
naming the endpoint and never the key; with a single key the 401 is still
the client's. A key whose file cannot be read is left out, and usage
records which key answered by its endpoint.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A second judge replica on the same machine shares its cores and is lost
with it, so it bought nothing but half the speed of each. On a 4-core,
8 GB single-machine install, one judge on four cores passed 20 of 20
requests at about 3.5 s each, judged in and out. Two judges on two cores
each had passed 19 of 20.
server init now serves one judge when this machine is the only node, and
two, on two cores each and spread over the nodes, where there will be
others (HA setups, or a control plane that is not a node).
--judge-replicas and the setup's Copies still choose. install.md gives
the measured times in place of "half a second".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Found by a three-machine install on fresh VMs. The second standby's run
asked for the floating address's interface with `ip route get <vip>`.
The machine already held the address, so that answered lo, and keepalived
would not start. The interface now comes from the route to the address's
network. Every cluster's VRRP router id was 51, so the PoC's two control
planes and the new three on one LAN each logged the other's adverts as an
invalid password; the id now comes from the address.
The Caddyfile, deploy/ha and install.md send clients to the control plane
whose /leader is 200, not /ready: /ready is 503 on a leader with no model
yet and would take the portal away with it. Setup no longer hides a key
file's path as if it were a password, and the starter's text no longer
says it runs elsewhere when this machine is a node.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The setup screens list the parts of an install: control plane, store, model
mirror, this machine as a node, starter model, pictures and 3D, the judge,
firewall, notifications, limits, on-demand turns and an offline bundle. Each
says what it is or what is missing without it. Space turns a part on or off;
Enter opens its settings in a dialog that explains it. A plan page lists what
is installed and what is left out, and the same install as a command. Every
setting is a server init flag too, written into server.yaml when init writes
it.
With three machines, the two database machines become control planes too,
standing by (server add's steps), with an optional floating address. The
mirror's management API answers all three control planes, so a standby that
comes to lead can still fetch models.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
notify.webhooks existed and setup never asked for one, so a new site
learnt of a parked model or a lost node from its users. `llmgw server
init --webhook URL` (repeatable) and a Notification webhooks field on
both setup screens take them now. Slack's address, Mattermost's and
Rocket.Chat's /hooks/ and Discord's …/slack are sent Slack's text, any
other the event as JSON; slack: or json: in front says which.
Once the control plane runs, they are added to the cluster's settings
beside the webhooks already there, none twice, and setup waits for the
control planes to start again on the change before sending each a test,
saying which answered. A webhook's path is its secret, so it is shown by
host only. A failure is said, not fatal: the cluster serves without its
alerts, and ctl notify test tries again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A site whose catalog had no flux-schnell or triposr showed no Images or 3D
page, and nothing said they were one click away. The cluster page now has
a Pictures and 3D models card under the models: each preset, what it
makes, the classes of card here it fits, whether and by whom it was
added, and why the leader has not added it by itself; an Add button, which
says first when it would install ComfyUI on the GPU nodes and restart the
control planes; and the switch for presets.auto. The console's models
screen shows the same under its table: A adds one, asking first in the
same case, and P turns automatic adding on or off.
`ctl model add PRESET` and `ctl model presets` go through the presets API,
so the control plane keeps who added one, and `ctl model presets auto
on|off` flips the setting; against a control plane from before, add sends
the entry as v0.14.4 did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
flux-schnell and triposr come with llmgw as presets: their workflow,
custom nodes, packages and licence as they were found and tried on the
PoC, and a profile that says the card it needs rather than a class, so the
entry places on any site's cards. TripoSR's workflow is built in, with the
LLMGWPrepareForMesh node that keeps a picture's background out of the
mesh, which the copy in the document had left out.
`ctl model add PRESET` adds one, and has the nodes install ComfyUI when
the cluster's settings name none; `ctl model presets` lists them; the
console's add-model dialog offers them beside a definition file. A
ComfyUI build is now also checked to keep its order on a Turing card, so
a node's installed cu13 is not replaced by the cu126 build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A 3D job leased by the standby control plane, whose router has no endpoints
and which a job replayed in the process never leaves, was told "loading"
for seven minutes while the leader served the jobs behind it. Jobs and
batches are claimed only while leading.
Five jobs asked for while cuda01's agent restarted failed for good as
"cannot be placed": re-registering, the node had no engines for a few
seconds. A job asks again for five minutes before that is its end.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A 3D job waiting for the picture model to give up the one card had an empty
progress; its client could not tell waiting from stuck. The control plane
says which model holds the cards and when it lets go, plus the waiting
model's last start; the 503 carries it as headers, and a job keeps it as
progress {state: waiting_gpu, held_by, eta_seconds}.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A finished job kept the last word heard from ComfyUI, the node running;
it is now said done before the job is marked completed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The catalog had one source a model, so an embedding model served by vLLM
from its repository on a card and by llama.cpp from a GGUF on CPUs had its
entry swapped by hand halfway through the metallurgy import. A profile's
source is what its engine loads and placement looks for; the model's is
the default, and what updates and canaries move. ctl model add checks
every source, the GGUF check reads each llama.cpp profile's, and ctl
weights collects none a profile loads. An empty source leaves a profile's
fingerprint as it was, so running replicas do not read as stale.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
POST /v1/3d/generations answers 202 with a job; GET says its status,
progress and result file; cancel stops it in the engine's queue too. The
request is kept as sent and replayed by whichever gateway holds the lease
through the path a client's request takes, so it is charged and judged
like any request (tagged job=<id>), and a model still loading is waited
out for as long as the job has. Migration 0044. Nothing makes 3D models
yet; the door refuses every model until the ComfyUI adapter learns meshes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
engines.comfyui in the cluster's settings has every NVIDIA node build
ComfyUI: the release's source archive from GitHub, checked against the
digest the recipe holds — GitHub publishes none for a tag's archive, so a
version without one is refused — unpacked beside a Python environment
whose dependencies are its requirements file resolved as PyPI stood the
day after the release. torch is PyPI's CUDA 13 build, or PyTorch's CUDA
12.9 one on an older driver, and the install is checked on the card.
A requirements file that names anything but PyPI packages is refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl models said "not placed" for a model with min 0 woken by a request and
serving; it now says "ready (on demand)". And the docs: pictures on the
cluster's own cards, with engine comfyui behind its gate.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.pricing takes per_image, charged on top of a prompt's tokens; a model
priced only per picture is not free, so a spent budget still stops it.
Each usage row keeps how many pictures were made (migration 0041), and
ctl request and the admin API's request view show them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The leading control plane tells the mirror over its API (PUT /v1/firewall,
kept as .firewall.json beside the models and applied by the same path unit
control planes use), and keeps the state in the store's cluster_state for
members that run no llmgw, which a root timer reads from the member's own
copy. The mirror also lets in the network of every node and control plane
the leader knows. Tried on the PoC: off, on and a network added, on every
kind of machine.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
