With slurm.users llmgw, each project is a Slurm account, its organization's
and team's above it, with the project's members and its team's and
organization's on it; slurm.conf holds a job to its user's accounts. The
controller's agent keeps them with sacctmgr, only what it made, and reports
the jobs that ended in the last day; the leader records each once, its
GPU-hours (core-hours without cards) priced at auth.gpu_hour_cost into its
project's usage as model slurm/PARTITION.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw's Slurm users log in to the controller freely and to another Slurm
machine only while a job of theirs runs there: the agent installs
pam_slurm_adopt and puts it in sshd's account stack for the users it made
alone, so root and the machine's own users are never locked out. Their
session is put in the job's extern step (PrologFlags=Contain), and
pam_systemd, which would move it out, is skipped for them by taking the
session file into sshd's stack with a guard in front of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- server init printed every engine's line again on each look while the
node installed (some 280 lines); each engine's line now once, and again
only when it changes, as node add does.
- A package install that is no engine's (Slurm's, MariaDB's) read
": installing installing mariadb-server, ..."; it is "system
packages: installing: mariadb-server, ..., with apt-get".
- The slow-judge advice says "ctl model resources JUDGE --cores 4",
which was refused for want of --memory: cores alone now keep the
memory as it is.
- server add said both control planes for three.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Since 23.11 Slurm runs badly without accounting storage: the main
scheduler passes every job by as InvalidAccount. slurmdbd takes MySQL or
MariaDB only, so the controller's agent installs MariaDB beside it,
answering that machine alone, makes the database and its user (the
password made from the munge key, never on a command line), runs
slurmdbd and adds the cluster, all before slurmctld. A MariaDB that was
on the machine before keeps its settings. slurm.conf accounts the jobs
there, with what each used from its cgroup and the cards, and the
priority is multifactor again. Port 6819 joins Slurm's in the firewall.
ctl slurm status and the console say whether slurmdbd runs; the install
screen, server init --slurm and the guide say what is put where.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slurm.conf sets SrunPortRange=60001-63000, so a site's firewall can let
srun's step connections in. A node given a part in Slurm adds 6817-6818
and that range to llmgw's nftables table, open to the cluster's networks
and closed to everyone else, as the engines' ports are.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC: InstallPackages reported 'installing munge, slurm-client,
slurmd' under no engine and never cleared it, so the node's page kept
saying so (infiniband-diags and perftest did the same). slurmd started
before the leader has written its node into slurm.conf is refused; the
drop-in now has systemd try it again every 15 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The InvalidAccount seen on the PoC was not the reconfigure: 23.11.4 without
accounting storage has its main scheduler mark a batch job so, and backfill
starts it on its next pass (job 4 waited 15 s). bf_interval=10 keeps that
short; the extra restart is taken out again. MailProg=/bin/true quiets the
warning on every start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC: Slurm 23.11.4 without accounting storage left batch jobs
pending InvalidAccount after scontrol reconfigure until slurmctld was
restarted. And jobacct_gather with nowhere to keep it logs an error per job.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC (Ubuntu 24.04, Slurm 23.11.4): the packaged unit is
Type=notify, so slurmd needs --systemd or systemd times it out; and slurmd
keeps a 'slurmstepd infinity' for its cgroup v2 scope, which was counted as
a job and kept the node from llmgw.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the PoC: scontrol looks for its files even for --version, and a
configless node has none before slurmd has fetched them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/control/slurmsvc tells each node what it is, writes slurm.conf
and gres.conf from the nodes' reports and the inventory, and moves each node
of slurm.nodes between llmgw and Slurm: lent, drained of replicas first, then
resumed in Slurm; taken back, drained in Slurm and cordoned in llmgw until
its last job step ends. Kept every tick from the settings and the agents'
reports, so a new leader carries on. A broken Slurm never keeps a node from
llmgw. GET /api/v1/slurm; ctl slurm status, setup, lend, reclaim, off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Slurm message says whether the node runs slurmctld on the leader's files,
slurmd configless, both or neither, and whether it is lent. The agent
installs munge and Slurm from the machine's sources (22.05 or newer, for
cgroup v2), writes the munge key, starts slurmd through a drop-in with
--conf-server, resumes a lent node that llmgw drained and drains one that is
not, never touching a drain a Slurm administrator made, and reports its
version, slurmd -C, state and jobs in SlurmStatus. Told it has no part, it
stops what it started. New nodes enter slurm.conf drained.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Registration carries what the agent's startup scan found, and a heartbeat
sends the list only when it changed since last sent; so a file deleted
after the agent started came back on a control plane that restarted. The
fingerprint is now forgotten with the provisioning one on every stream.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A set is fetched into a directory named from its address without the
trailing slash; Orphans named it from the catalog's address with it, so
every set in the catalog was offered for deletion: 330 GB of six models on
the PoC. One function, cacheSource, now gives both the name.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sweep counted every file the nodes were asked to delete as freed as soon
as it asked, and a node refused a set's directory as not empty, so tens of
gigabytes were reported freed and nothing was. The agent removes a set's
directory whole (a link only as a link), and the control plane waits for
each node to list its files again, saying what it no longer lists and what
it kept.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each agent measures one core's memory rate beside the whole machine's; the
estimate takes a replica's threads at about one core's rate each, up to the
machine's, calibrated by llama-bench on the PoC. The serve form shows it
beside a card's, warns when slow, and asks for the CPU on purpose when a
reply would take minutes; ctl model plan, the console and serve jobs say
it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Single files were shared between the shards and replicas asking for them
at the same time; sets were not, and four replicas on one node's four cards
wrote the same .part files over one another until all but one failed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.
ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.
Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.
FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.
Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found running it through the cluster:
- Strata's server starts its engine in a process group of its own, so the
agent saw a GPU replica on the CPU and its 22 GB as somebody else's
memory. A process the shard started, at any remove, is now the shard's.
- An adopted shard's kept compute was sent at once and never again, so a
wrong one read again on adoption never reached the control plane: what
is sent is sent again when it changes, and a GPU shard kept as the CPU
is read again on adoption.
- /metrics/prometheus is llama.cpp's names without the KV cache; vLLM's,
with it, are /metrics?format=prometheus. The router now sees running 1,
waiting 2 under load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's
experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it
serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client.
- engines/strata.go: the adapter. The agent starts llmgw strata-serve,
which runs Strata's setup the way its desktop app does
(STRATA_CONTROL_INSTALL: installs nothing, writes only under the
replica's and the node's directories) on the fetched GGUF files, points
the configuration it writes at the installed engine, and execs Strata's
server. The key goes in STRATA_API_KEY, never on a command line.
- A profile's args go to setup: --family, --size (setup's --model, which
is the source everywhere else), --context, --parallel, ... Flags that
move files or pick the engine or GPU are refused.
- provision: the source pinned by digest in a Python environment, and the
release's prebuilt engine unpacked beside it, checked against the digest
GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9,
driver 575). Python builds can now carry release assets.
- Metrics are vLLM's names under /metrics/prometheus.
- The catalog wants one card and a family, and a directory, not one .gguf.
Opt-in: not in DefaultEngines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From the second review.
- An output service reads the answer with the values restore put back
taken out again (guard.Subject.Restored), as an input service reads the
request redacted.
- With n above 1, the service reads the first choice once it is whole,
not at whichever choice finished first.
- Restored values go into an answer asked for as JSON (response_format)
escaped for it, as into tool-call arguments.
- The embedding failure is logged once a minute through an outage, not
once.
- A card's own lane count and fastest generation come from the kernel
(max_link_width, max_link_speed): the driver's width.max is what card
and slot allow together, so an x16 card in an x8 slot read x8 of x8 and
was never said.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Revoking a server's tools takes them from every level beneath too, and a
level left naming no tool stays closed ([]) while any level above or
beneath names tools: an organization's revoke left its teams the copies
their first grants took, and going back to no list opened the level to
what was above it. A closed level is shown on the page and in the
console with an action to open it (store.OpenTools). A key's owner's
list is among the levels above a key.
- The flag check matches short flags as written again (-m, -hf, -mu were
let through), takes a long flag's beginning for it only for the engines
that parse with argparse (vLLM, SGLang, ComfyUI: --li is --listen), so
llama.cpp's own --chat-template is not refused, and refuses llama.cpp's
short spellings (-hfr, -hff, -mm, -jf, …).
- A stored model with an adapter by Hugging Face id can be changed again:
what it already had is not asked about anew.
- The catalog reads a Hugging Face id as the agent does (engines.IsHubID).
- The page's form cannot move llmgw:admin to another address.
- server init skips the memory check only where the gateway already runs,
not wherever a half-finished first run left server.yaml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review.
- A service two policies name is asked the stricter way of the two, both
directions and block over flag, rather than as the first one says.
- A service answering null where it says unsafe gave no answer, not safe.
- With restore: true, the kept answer had the request's redacted values in
it beside the redacted request; they are taken out again before it is
kept.
- guardrails.md says what fail: open lets through, and that of n answers a
service reads the first.
- The similar cache's embedding failures are logged once a minute, not per
request; the page's test reads the gateway's local MCP handler under its
lock; org/name:Q4_K_M is not taken for a Hugging Face id to read as
safetensors.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At the user's asking: we know what a card's link could be, so a card
linked slower is reported. Each agent reads, apart from the GPU query so
that a driver not knowing a field costs nothing else, the fastest PCIe
generation card and slot allow together, the card's own, and the lanes in
use and the card's most (nvidia-smi; NVML in the nvml build). The current
generation is not compared: it drops while a card idles. A card on fewer
lanes, or a slot of an older generation, is a node warning with the GB/s
it carries against what it could — on the dashboard, the console and in
llmgw doctor, which points at lspci's LnkCap and LnkSta. MIG instances are
said of their card. GPU proto fields 23-26.
The PoC's Quadro RTX 6000 reads Gen3 of Gen3, x16 of x16, idle at Gen1:
nothing to say.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- server init run again on a machine already set up is not refused its
memory (the docs fix a certificate with --host so), and judges count
only when one will be installed (judgeComes: not without a mirror, nor
from a bundle without it).
- A server's sealed secrets that it no longer reads are removed when it is
put or deleted; replacing a credential replaces every header it was
sent, rather than adding one beside the old.
- llmgw.gatewayUrl is a machine setting and restricted in untrusted
workspaces: a cloned repository's settings cannot send a sign-in to
another gateway.
- The bandwidth buffer is at least eight times the CPU's last cache and at
most a quarter of free memory, else nothing is measured until there is
room; a registration carries the figure measured so far.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review. A mirror set already on the node is looked at again, not
trusted for having been fetched once; Pickled cuts a query or a fragment
from any source, hf: too; .pt2 is a pickle. The flag check reads a flag as
the engine does: underscores as dashes and any beginning of a refused flag
as that flag, since argparse takes --trust-remote for --trust-remote-code;
SGLang refuses --load-format, which would undo the agent's safetensors.
TripoSR's preset says allow_pickle: without it the preset could not be
added and the presets pass failed on it each time. A shared ComfyUI
environment's hidden entry takes its first member's allow_pickle, as it
takes its source.
The GenAI trace attributes made temperature and top_p typed fields of the
request's parse, so "0.7" as a string was a 400; they are read loosely,
as they are read only for the trace.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven VMs on one host restarted at once read a third less each than any
gets alone. An agent now waits 10 s to 2 min, at random, before measuring,
and measures again every hour, keeping the best.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measuring it in the startup scan held every agent half a second or more
before registering, which on a shared CI runner was past the failover
tests' wait. Now it runs once a process, in the background, the first time
it is asked for, and Heartbeat.mem_bandwidth_gbs carries it when known;
the control plane keeps the last figure and a heartbeat without one leaves
it. The agent's tests do not measure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).
The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.
BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sudo llmgw on a node showed only agent.yaml, the manual, wipe and leave.
It now opens on an Overview read on the node itself: the agent's state and
version, the control planes it dials and whether they answer, each GPU with
the models on it, the engines from the supervisor's state files, the free
space for weights and engines, and the agent's warnings of the last hour.
No admin token is needed, every command is bounded by two seconds, and a
missing tool is said in a line.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Measured again after the store fixes. Cutting the store primary's power no
longer hung requests, but leadership moved anyway: the lease had lapsed for
everyone while Patroni promoted the standby, and a standby asked first. The
new leader then spent about 25 s without its nodes, and so without its
judges.
A standby that saw the store out of reach now leaves the lease alone for one
renew round after the store returns. The default lease is 10 s, and a node
detects a control plane that died silently in 8 s (5 s idle, 3 s ping)
rather than 15.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux and sf3d on one card handed it to each other, each starting ComfyUI
and loading its weights from nothing (BACKLOG H3, placement side). An
environment of two or more models now gets a hidden catalog entry,
comfyui-env-<name>: on its first member's profiles with the most memory
any member needs, on demand for the longest idle timeout, carrying every
member's weights, repositories and the environment's custom nodes. The
members get no replicas of their own; asking for one wants the entry.
The agent fetches each member's weights; the gate links all of them,
refusing two weights files of one name, links each member's Hugging Face
repository, and answers /v1/models for every member. The control plane
offers the replica under the members' names with each one's workflow and
checkpoint, which the router sends by the model asked for. A member's
sibling never evicts the replica serving it, and the members' load counts
as the entry's.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
flux-schnell and triposr come with llmgw as presets: their workflow,
custom nodes, packages and licence as they were found and tried on the
PoC, and a profile that says the card it needs rather than a class, so the
entry places on any site's cards. TripoSR's workflow is built in, with the
LLMGWPrepareForMesh node that keeps a picture's background out of the
mesh, which the copy in the document had left out.
`ctl model add PRESET` adds one, and has the nodes install ComfyUI when
the cluster's settings name none; `ctl model presets` lists them; the
console's add-model dialog offers them beside a definition file. A
ComfyUI build is now also checked to keep its order on a Turing card, so
a node's installed cu13 is not replaced by the cu126 build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PyTorch's CUDA 12.8 and 12.9 builds start at Turing since torch 2.8, as
CUDA 13 does, so on a V100 both ComfyUI builds were installed, 10 GB each,
and failed on the card with "no kernel image". The cu129 build now needs
7.5, and a third, on PyTorch's CUDA 12.6 torch, carries Volta and Pascal.
The open kernel module, NVIDIA's default since 560, writes "Open Kernel
Module for x86_64 580.82.07" in /proc/driver/nvidia/version, and the word
after "Module" was taken as the driver: "for". With the driver unknown,
every build was tried whatever driver it needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sf3d's first jobs failed with a 401 from stabilityai/stable-fast-3d. The
weights were on the node, and the gate links them into the Hugging Face
cache when told the repository — but the agent passed on only the custom
nodes, so nothing was linked and the node went to the network for a gated
repository.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
G2: the gate keeps a websocket open to its ComfyUI under a client id of its
own, puts that id on every workflow it lets through — ComfyUI tells only
the client that queued a workflow how it goes — and keeps the last word on
each: the node running, its step of how many. It answers at
/progress/{prompt_id}. While a job's request waits, the router reads the
queue (queued, n ahead) and the gate (running, node, step, percent) and
writes it into the job, which a client reads back. Only into a job of the
same key: a tag naming another's job writes nothing.
G3, the rest of the plumbing: comfyui.nodes names custom nodes by https
repository and full commit, with the PyPI packages they need. The gate puts
each in the replica's own custom_nodes at that commit and installs the
packages beside the replica, constrained to the versions ComfyUI's
environment already has, which it never changes. The catalog refuses a
branch for a commit and anything but package specs for requirements. An
OBJ a node saves leaves the gateway as a GLB, colours kept, so every mesh
can carry its licence.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines wrote to a pipe of the agent's. An agent that restarted and
adopted them left that pipe with no reader, and an engine's next write
past its buffer failed with EPIPE — in a Python engine, inside whatever it
was reporting on. Each process now writes to its own file in the agent's
work directory (0600), followed into the shard's log; an adopted engine's
file is read from the start, so its output from before the restart is in
its log too. A file past 64 MB is emptied in place; the files go with the
shard.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engines.comfyui in the cluster's settings has every NVIDIA node build
ComfyUI: the release's source archive from GitHub, checked against the
digest the recipe holds — GitHub publishes none for a tag's archive, so a
version without one is refused — unpacked beside a Python environment
whose dependencies are its requirements file resolved as PyPI stood the
day after the release. torch is PyPI's CUDA 13 build, or PyTorch's CUDA
12.9 one on an older driver, and the install is checked on the card.
A requirements file that names anything but PyPI packages is refused.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
