The busy figure is nvidia-smi's utilization.gpu: the share of time any
kernel ran, which a kernel on one SM fills. On cuda01 decode read 92% busy
while the SMs had work 69% of the time and issued on 18% of their cycles.
Busy is now said to be that, and beside it: each card's draw against its
power limit (efficiency table, fleet and node charts, the console's GPU
work chart), and where the profiling counters can be read, SMs active, SM
occupancy, tensor cores and DRAM (llmgw_gpu_*_active_percent), from NVML's
GPM on Hopper and later or a dcgm-exporter on the node
(discovery.dcgm_exporter). The GPU map colours by work by default and keeps
busy as a mode. A MIG instance is never published with its card's profile.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Revoking a server's tools takes them from every level beneath too, and a
level left naming no tool stays closed ([]) while any level above or
beneath names tools: an organization's revoke left its teams the copies
their first grants took, and going back to no list opened the level to
what was above it. A closed level is shown on the page and in the
console with an action to open it (store.OpenTools). A key's owner's
list is among the levels above a key.
- The flag check matches short flags as written again (-m, -hf, -mu were
let through), takes a long flag's beginning for it only for the engines
that parse with argparse (vLLM, SGLang, ComfyUI: --li is --listen), so
llama.cpp's own --chat-template is not refused, and refuses llama.cpp's
short spellings (-hfr, -hff, -mm, -jf, …).
- A stored model with an adapter by Hugging Face id can be changed again:
what it already had is not asked about anew.
- The catalog reads a Hugging Face id as the agent does (engines.IsHubID).
- The page's form cannot move llmgw:admin to another address.
- server init skips the memory check only where the gateway already runs,
not wherever a half-finished first run left server.yaml.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review. A server's description, timeout and rate can be emptied from
the console and the page (MCPServerRequest.Clear), where an empty field
used to leave them as they were. An address with a user and password in
it is refused (adminapi.CheckMCPAddress): the credential belongs in a
header, sealed, and the address is shown and audited as written. The
console accepts llmgw:admin and the page lets it be changed, address
fixed. A grant to a key that is not there is refused rather than said
done.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tool is seen where every level above that names tools names it (store
Effective.AllowsTool). The console and /admin/mcp wrote a level's list as
given, so a team that saw its organization's hr.* through having no list
was given jira.* and then saw neither; and revoking a level's last server
left [] in the store, closed to every tool, shown nowhere.
store.GrantTools and RevokeTools now do both screens' work: a level's first
list starts from the nearest list above it; taking a server away leaves the
others, and a level left naming nothing goes back to no list, or stays
closed when a level above names tools, so it opens to nothing it was kept
from. GrantTools says which levels above still keep the granted tools out
of reach, and both screens say so. The page shows closed levels and member
scopes by name, and offers no revoke for a * grant it would not change. The
console calls the store instead of keys limits with a list built from what
it had loaded, which overwrote changes made since. Found in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/admin/mcp, for administrators only, since a server's credential reaches
whatever it guards: each server's state, who has been granted its tools,
its rate, the headers it is sent by name with the sealed ones marked, and
its address; adding one with a bearer token or another header kept in the
store; testing it with its credential and listing its tools; granting them
to a team, project, org, person or key, written into that level's limits
beside its other grants and its rate and budget; revoking; changing;
replacing the credential; turning off and on; removing; and how a person
points an agent at the gateway's /mcp with a key of their own. English and
Turkish. The secret's field is named so the audit line redacts it.
The admin API's server merge and test are package functions now,
adminapi.MCPServerFrom and adminapi.TestMCPServer, shared by both.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For each layout modelplan.Options offers here — one card, tensor parallel
across a node's cards, llama.cpp's layer split, a pipeline across nodes on
a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes
a token reads over each card's bandwidth, by its model name (a table, SXM
and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on
NVLink, 25 us and up on PCIe), turns between cards for a layer split, and
a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a
repository's files show the fastest, with the layout; mixed classes are
judged class by class, FP8 on 8.9 cards only.
Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB,
62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB,
144 measured, which says a model of experts reads 1.6 times its active
share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics
are not measured here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).
The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Judged by the nodes as they are now, through modelplan.Options, the
planner that serving a model uses: GGUF at Q4_K_M from the parameter count,
safetensors at the size the registry's dtype breakdown gives (packed AWQ,
GPTQ and MXFP4 at four bits), FP8 only where a card has compute capability
8.9, MLX safetensors never. A picture model is judged against one card. A
cluster with cards asks the registry for safetensors too, a hundred at a
time; one without asks for GGUF only. The page says what it judged by: the
largest card, its capability, the most RAM on one node.
On a repository's page a file set no node holds is marked with what it
needs, against what the best node offers, and has no Fetch button.
Tried against the PoC's seven nodes and Hugging Face's trending hundred:
63 shown, the rest too big for one 24 GB Turing card or FP8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A repository with neither GGUF nor safetensors (by the registry's summary
or its tags) is hidden under "runs here", which is ticked by default, and
counted with the architectures llama.cpp cannot load; unticked it is
listed. An open repository's pickle files are not offered, only counted,
and a repository of pickles only says to look for another version.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
llmgw's own extension for VS Code, plain JavaScript with no dependencies,
packed into a .vsix by the binary at /setup/vscode/llmgw.vsix with the
serving gateway's address in it, so it needs no build, no Node and no
marketplace. It registers llmgw as a language-model provider: the chat's
model picker, agent mode and any extension that asks VS Code for a model
get every chat model the key may use, sized and with tool and vision
capabilities as the catalog says, streamed, tool calls both ways.
VS Code is a tool on the account page's Connect your editor: a key, the
install command, and an Open in VS Code link that hands the key to the
extension's secret storage. The extension's Sign In opens the page with a
state of its own; a link without it is confirmed, the gateway named, since
it decides where code is sent.
Tested in VS Code 1.138 against the PoC: 27 models listed, an answer from
qwen2.5-0.5b-instruct, and qwen3-coder-30b calling a tool and using its
result (internal/vscodeext/e2e).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/install takes a reader from three bare Ubuntu machines to a first answer:
what each machine needs and the commands that check it, the release fetched
and its checksum checked, the eight setup screens, the system's state and
the console, and a first request with a key from the account page. Every
screen is from an install of v0.14.9 on three fresh VMs, recorded with tmux
and kept as terminal frames: the page shows them as text, and plays a
session back when asked, fetching it compressed from /install/cast/. The
administrator's password and the API key were masked before the recording
was kept, and a test fails if either shape reaches the page.
/about gains what llmgw now does beyond routing, and its install steps are
today's: the setup screens and node add, not onboard-node.sh and a model
file, with a link into the guide.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The errors other packages give a page (store, catalog, mirror) are
shown in its language; a percentage is written as the language writes
it (%93 in Turkish); the footer's time in its date form; and a stale
form is refused with a sentence rather than "bad request".
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
auth.ldap checks a username and password against the domain controllers
over LDAPS or StartTLS: found with a service account, checked by binding
as the person. Groups, nested ones included, give staff roles and places
in the tree. Accounts are tied to the objectGUID; the leader reads the
directory every 15 minutes, suspends whom it disabled or removed and
reinstates them after, and refuses a sync that would suspend most.
Entra ID sends no groups for someone in more than 200, only where to
fetch them; that was read as none and took an administrator's role away.
It is now left as it was, and logged. An account made by SCIM or OIDC
can be reinstated again.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Five pages named /_static/pages.js outright, cached for a day, so a
browser kept the old script after an upgrade — the one holding refreshes
off while a form is filled in among them. They take it through asset, as
the rest do, whose address changes with its contents.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster and node pages swap their live part every few seconds, and the
forms inside it went with it: what was typed, an open choice, the focus.
Filling one in raced the clock. The swap now waits while a field in the
live part has the focus or a form in it has been typed into and not sent;
sending it, or leaving the page, ends that. The mirror page's reload
during a download goes through the same check instead of http-equiv.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A site whose catalog had no flux-schnell or triposr showed no Images or 3D
page, and nothing said they were one click away. The cluster page now has
a Pictures and 3D models card under the models: each preset, what it
makes, the classes of card here it fits, whether and by whom it was
added, and why the leader has not added it by itself; an Add button, which
says first when it would install ComfyUI on the GPU nodes and restart the
control planes; and the switch for presets.auto. The console's models
screen shows the same under its table: A adds one, asking first in the
same case, and P turns automatic adding on or off.
`ctl model add PRESET` and `ctl model presets` go through the presets API,
so the control plane keeps who added one, and `ctl model presets auto
on|off` flips the setting; against a control plane from before, add sends
the entry as v0.14.4 did.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A game sold in the EU cannot ship what a licence excluding the EU made. On
the profile page a person leaves out licences one by one, or by rule:
every licence forbidding commercial use, every licence granting nothing in
a region. The rules cover models added later. The choice reaches every
principal through the resolved limits, so permits() leaves such models out
of /v1/models, requests, jobs, fallbacks and the portal's pickers, and a
request for one is refused 403 licence_excluded saying which choice did.
Kept in user_excluded_licences (migration 0046), copied with the store and
gone with the person.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/v1/openapi.json is the API clients use — OpenAI's endpoints, Anthropic's,
and what llmgw adds — in OpenAPI 3.1, generated from internal/clientapi's
route table by the generator the admin API's document comes from, now
told each document's title, server, token and error. A test reads the
router's mux.HandleFunc calls and refuses a route described and not
mounted, or mounted and not described. Asked for before there is a key,
so it needs none.
/docs/api shows it in Swagger UI 5.33.1 (Apache 2.0, served from the
gateway as the page's policy requires): Authorize takes a key, Try it out
sends to this gateway. Staff are offered the admin API's document too,
from /docs/api/admin.json, which nobody else is given.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/3d beside /images, while some model makes meshes: a picture chosen, the
job asked for through the same path an API client's takes, its progress
shown (waiting for the card, n ahead; making it, the node and percent),
and the mesh in a viewer — model-viewer 4.4.0 (Apache 2.0, with three.js
and lit), served from the gateway since the page's policy loads scripts
from nowhere else — with a download and the model's licence. The mesh is
served to its owner from /3d/jobs/{id}/file; another project's person is
told there is no such job.
A job made from the portal acted for a key that does not exist: the
worker now rebuilds a session's principal from the account that made it,
while it is active and still in the project the job is charged to.
ctl 3d list|get|cancel, over /api/v1/client-jobs (/api/v1/jobs is the
control plane's own work, which ctl jobs shows).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/images offers the models that make pictures, and the chat no longer
offers them. Like the chat, the browser holds a session and never a key;
the page's request is rebuilt from its own fields and sent down the path
every /v1/images/generations request takes. Size, how many, seed, steps
and a negative prompt; each picture shown with its seed, a download link
and a way to make it again. Nothing is kept but the usage. The menu
offers the page while some model makes pictures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
It sat in localStorage under one fixed key, so the next person to sign in
on the same browser opened the chat on the last one's conversation. It is
now kept under the account's id, anyone else's is dropped when the page
loads, and signing out sends Clear-Site-Data: "storage" so nothing the
pages kept outlives the session.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/admin/settings edits the document llmgw ctl config edit opens: checked as a
control plane would load it, refused when made from a version since replaced
(the edit kept), its history viewed and rolled back as a new version, every
change audited by size and never by contents, since it can hold webhook
secrets. Administrators only; in English and Turkish.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A Documents picker on /chat offers the collections the person may read in
the chosen project, the default when there is one, or none; the sources an
answer was given are listed under it. The page's choice becomes the
X-Llmgw-Knowledge header; one the page sent itself is dropped. In English
and Turkish.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
/forgot mails a link to set a new password when the cluster's mail settings
say how - SMTP relay, sender, and the public base_url the link points at,
never the request's Host - and says the same whether or not the address has
an account. /reset sets it once, closing every session. Links are kept by
their hash in password_resets, good for an hour, a newer one replacing an
older. Without mail, /forgot says to ask an operator, and llmgw users
reset-link makes the link to hand over. Single sign-on accounts get none.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
guard.enabled (unset is on) makes every policy check nothing while false,
live: llmgw ctl guard off/on set it in the cluster's settings, and ctl
status, ctl guard list and the guardrails page say when they are off.
A refusal by a judge names the hazard it saw - judge: S6,Specialized
Advice - and the verdict is kept with the request in a new usage column,
guard_rule, shown by ctl request, the request log and the guardrails page.
llmgw ctl guard judge ask has a judge read a text, past the cache, and shows
its raw answer and how it is read; the endpoint is kept out of the audit so
no text is recorded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every template and handler message in internal/web goes through a JSON
catalogue under internal/web/locales/<code>/, with English as the fallback.
The language is the person's choice (a cookie set by /_lang/{code}), else
Accept-Language, else landing.language. Tests refuse a key some language
lacks, a translation with other arguments than English, and template text
left in English.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The manual the binary carries, rendered on the client port: a sidebar of
the documents in the console's reading order (now one list, llmgw.Manual,
for both), heading anchors, links between documents kept in the portal and
links to files outside the binary sent to the repository, no script.
Public by default; landing.docs_sign_in sends visitors to sign in first.
docs/clients.md shows the official OpenAI and Anthropic SDKs against
llmgw in Python, TypeScript, Go and curl, with llmgw's headers; the
portal's index, the landing page and the account page link to it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A wrong password from anywhere counted against the address, so ten locked
its owner out. Attempts are now counted per address and client, with a
looser bound per address alone. The sign-in form carries a token matched
by a cookie (login CSRF). POST /login and /signup are rationed per client
address, a hash waits at most ten seconds for a turn, and server init
writes signup_open: false.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Percentiles are read from the histograms kept with the usage sums and shown
per group on the usage tab and in the people page's panels. Requests refused
by a limit are recorded as 429s against their owner, free of charge, so a team
hitting its limits shows as such. Usage can be grouped by model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An organization's members, limits and use are a panel of the people page,
where they can also be changed, so the administrator's menu loses its
Organizations entry and /org redirects there. Managers keep /org.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A team's members, quota and spend were on three pages that linked to each
other. They are three tabs of one page now, each at its old address, and
choosing anything in the tree opens its limits, capacity and use beside it,
editable in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model with replicas on a card and on CPUs answered at either speed with no
way to tell which. The picker names where each model runs, each reply names
the hardware that served it, and every answer carries x-llmgw-hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page draws every card coloured by utilisation, memory or
temperature, and lists replicas whose every card is under 5% busy.
llmgw_gpu_assigned says which replica, model and owner hold each card.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each engine has a list of builds, best first, from what its project
publishes and how each was compiled: llama.cpp CUDA 13.3, CUDA 12.8
(driver 570 for the GPUs it ships as PTX), then Vulkan, never the CPU
build on an NVIDIA node; vLLM CUDA 13, then CUDA 12.9, and for GPUs
older than every build of the version asked for, the newest built for
them (V100: 0.19.1). The node takes the first its hardware meets and
that installs and computes on the GPU, moving on when one does not, and
reports the version it runs. GPU processes are read from nvidia-smi's
full report, so Vulkan engines count as on their GPU.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>