slurm.containers (ctl slurm containers on) has every Slurm machine's agent
install enroot from NVIDIA's release and build pyxis against its own
Slurm, both pinned and checked against their digests, and on GPU machines
nvidia-container-cli from NVIDIA's repository. enroot keeps each user's
containers under /run/enroot and /var/lib/enroot, needing no login
session, with the shared path in every container. The leader has Slurm
load pyxis only once every machine reports it: plugstack.conf's
"required" fails every job on a machine without the plugin.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tried on the PoC: lending cuda01, the one GPU node, its model had nowhere
else to go and waited out the drain's five-minute grace, while ctl slurm
status said only "1 replica(s) still to move off". It now says the time
that replica is stopped at. Also recorded: Slurm on a GPU node, tried on
cuda01 (gres, ConstrainDevices, accounting of gres/gpu).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw's Slurm users log in to the controller freely and to another Slurm
machine only while a job of theirs runs there: the agent installs
pam_slurm_adopt and puts it in sshd's account stack for the users it made
alone, so root and the machine's own users are never locked out. Their
session is put in the job's extern step (PrologFlags=Contain), and
pam_systemd, which would move it out, is skipped for them by taking the
session file into sshd's stack with a guard in front of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Setup asks for webhooks, but one left empty was never mentioned again: a
lost node or a judge down was told nobody. With no notify.webhooks in the
cluster's settings, ctl status and the console's Overview say so, and how
to set one. BACKLOG: the judge's refusal after a placement change, tried
again on idle VMs, did not happen; a cold judge's first answer is 2-3x
slower but inside its budget.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slurm.shared mounts the site's NFS export, has an llmgw node export the path
over NFS to the Slurm machines alone (2049 in the firewall), or looks for a
file system the site mounts. slurm.users "llmgw" makes each active account a
Linux user on every Slurm machine: login from its e-mail, a UID from 200000
kept in slurm_users and never reused, its home on the shared path, its SSH
keys from the portal's Profile (or ctl slurm users keys) read by sshd from
/etc/llmgw/ssh-keys. server init --slurm chooses /shared from the first
machine and llmgw's users; --slurm-shared and --slurm-users change that.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
priority/basic did not end InvalidAccount: 23.11 dropped
accounting_storage/none, and without slurmdbd the main scheduler passes
jobs by whatever the priority plugin. Basic stays, fair share needing
slurmdbd anyway.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Without accounting storage, 23.11's default priority/multifactor has the
main scheduler pass every job by as InvalidAccount; backfill started it a
pass later. slurm.conf sets priority/basic.
Every node was written State=DRAIN, so each start of slurmctld on new
files drained a lent node again until its agent resumed it. A lent node
is written without it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tried on the PoC: five one-second jobs on an 8-CPU node started 10 s
apart. Each asked for no memory, so each was given the node's all, ran
alone, and the next waited for backfill's pass. Not accounting, as
thought. The partition's DefMemPerCPU is now the leanest node's memory
per CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The console's Slurm section shows the controller, and each node that runs
slurmd with whose it is now, its state in Slurm and its jobs. Enter on a
row lends the node (its replicas move off first) or takes it back, by the
same change to slurm.lent as ctl slurm lend and reclaim.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The choices screen has a Slurm part, and --slurm writes slurmctld on this
machine's node and slurmd on every node, those that join later too, into
server.yaml. slurm.all_nodes (ctl slurm setup --all-nodes) takes the nodes
from the leader's inventory. No node is lent at install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/control/slurmsvc tells each node what it is, writes slurm.conf
and gres.conf from the nodes' reports and the inventory, and moves each node
of slurm.nodes between llmgw and Slurm: lent, drained of replicas first, then
resumed in Slurm; taken back, drained in Slurm and cordoned in llmgw until
its last job step ends. Kept every tick from the settings and the agents'
reports, so a new leader carries on. A broken Slurm never keeps a node from
llmgw. GET /api/v1/slurm; ctl slurm status, setup, lend, reclaim, off.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slurm.{enabled,cluster_name,controller,nodes,lent,partition} in the cluster's
settings, checked; internal/slurm writes slurm.conf (configless), gres.conf
and cgroup.conf from the nodes' slurmd -C lines and GPUs, derives the munge
key from a cluster secret, and reads scontrol and version output.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C1. A membership of an organization may be admin as well as member and
manager. An admin sees /org as a manager does and changes the
organization from it: its teams and projects, who of the approved
accounts is in them, its members' roles, and the keys spent in its
projects. Every id a form names is checked against the organization;
limits, accounts, the organization's name and the cluster stay with the
cluster's staff, and an admin cannot take their own role away.
A group from OIDC, SCIM or a directory no longer rewrites a membership
it names as a plain member at every sign-in, which took a manager's role
away (EnsureMembership).
Pages draw a form only for someone who may send it: Nav.May asks the
staff rule staffMay keeps, and a page whose forms a person may not send
gets no form token. A viewer sees the people, limits and mirror pages
without forms; an operator the mirror's.
Forms are compact: labels close to their fields, fields as wide as what
goes in them, smaller buttons, and the short forms on one line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each router's per-counter request and token counts, this minute's and the
last, ride the standby's two-second exchange with the leader. The leader
answers each standby with everyone else's sum, its own included, and puts
the standbys' sum in force on its own router; each limiter adds the
others' counts to its sliding window.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A standby follows the leader's Ready replicas over the agent port, every
two seconds, and reports back the replicas its router used and the
on-demand models it was asked for. No engine keys travel: every control
plane derives them from the cluster secret. Budgets are raised to the
store's every 30 s. Unheard from the leader for a minute, a standby
passes /v1 on as before. Off by default.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
mirror add-peer, run on the mirror's machine, sets up a mirror on
another over ssh, kept in step with this one: llmgw there if it has
none, mirror init --peer, this mirror told of it and allowed to be read
by it, and mirror.peers in the cluster's settings. The setup screens
offer one on either database machine, as server init --second-mirror.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A repository with neither GGUF nor safetensors (by the registry's summary
or its tags) is hidden under "runs here", which is ticked by default, and
counted with the architectures llama.cpp cannot load; unticked it is
listed. An open repository's pickle files are not offered, only counted,
and a repository of pickles only says to look for another version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.
BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw keys cache --similar 0.95 also gives a chat question the answer to
one like it: the last user message embedded by router.cache.embedding_model
(asked on the gateway's account, as a judge is, within 2 s), compared by
cosine with the questions answered in the same conversation — system
prompt, earlier turns, tools and sampling the same. The exact cache is
asked first; an exact hit embeds nothing. Said as x-llmgw-cache: similar
with x-llmgw-cache-similarity, charged as a hit.
The vectors live with their answers in the gateway's memory, counted in
max_bytes and evicted with them, not in a pgvector table as BACKLOG I1
planned: the cache keeps answer text out of the store on purpose. A
question over 2,000 characters is reused only when the same. Migration
0051 holds the setting. BACKLOG I1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cached_per_1k in a price charges the prompt tokens the answer counts as
cached_tokens at their own rate; zero keeps charging them as prompt tokens.
The count reaches the cost and not a column: CostMicros holds what it
changed. Cached beyond the prompt is clamped, an engine's miscount rather
than a refund. BACKLOG L5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docs/kong.md: a plain route or ai-proxy with llmgw as the upstream, the
client's llmgw key passed on so per-team budgets hold, Kong as a provider
behind llmgw, and Kong 3.9's streamed answers counted without completion
tokens. Closes BACKLOG L9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A site may be required to use a particular guardrail: Azure Content Safety,
Lakera, NeMo, a classifier of its own. A policy's services asks one over
HTTP — a body template given the text, header values from files read each
call, dotted paths to where the answer says unsafe and why — on input,
output or both, to block or flag. On input it reads the latest user turn
before the judge; on output the whole answer once at its end. A service
that does not answer is what the policy's fail says. It is told the text
only, never who sent it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With restore: true on a guardrail policy's input, what redact rewrote as
placeholders comes back in the answer: the model reads "[EMAIL_1]", the
client reads the address it sent. A streamed placeholder split across
events is held until it is whole, and what is held is sent at the end.
The swap runs before the output filter, so the output rules judge the
answer as the client reads it and a placeholder they make of a value the
model invented is never taken for the request's own. The map lives in the
Verdict for the request only; nothing stores, logs or traces it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An inference request and each attempt at an engine carry gen_ai.*: the
operation, the requested and answering model, max_tokens, temperature and
top_p when the client set them, the session as the conversation id, the
input, output and cached tokens, and on an attempt the engine that answered.
A customer's Grafana, Jaeger or Langfuse then shows llmgw's calls as model
calls. No prompt or answer text is ever put in a span, and a test says so.
BACKLOG gains section L: what is worth taking from Kong's AI Gateway, of
which this is the first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The errors other packages give a page (store, catalog, mirror) are
shown in its language; a percentage is written as the language writes
it (%93 in Turkish); the footer's time in its date form; and a stale
form is refused with a sentence rather than "bad request".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A tool's answer is read by the caller's policies as the prompt it
becomes, so an instruction planted in a ticket is withheld and personal
data redacted; its arguments are read before they leave. ctl mcp add
--rate bounds each caller's calls a minute to a server (migration 0050),
and five failures in a row take a server out for a minute, its tools
off the list.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ctl guard judge check, and POST /api/v1/guard/judges/check, time three
judgments of a 280-token prompt, then a burst at once, against the
policies' wait: ok, tight or slow, about how many requests a second can
be judged, and what to do. server init runs it once the judge is up and
says a machine too slow before the next steps; the wait is not changed
without asking. BACKLOG reordered by what goes wrong where installed.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
llmgw ctl mcp add|list|remove|test and /api/v1/mcp/servers keep the
servers; a token given with --secret-file is kept in the store, sealed,
and named by reference. keys limits --tools, and the console's limits
form, grant tools as patterns. Documented in install.md and clients.md.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
