A new internal/clientip says who sent a request and how: the connection's
address, scheme and Host, or through a configured proxy what it forwarded.
The base URL in pages and setup scripts, isHTTPS for cookies, and the
per-address limits of auth and the sign-in forms use it. /setup scripts are
no-store.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A wrong password from anywhere counted against the address, so ten locked
its owner out. Attempts are now counted per address and client, with a
looser bound per address alone. The sign-in form carries a token matched
by a cookie (login CSRF). POST /login and /signup are rationed per client
address, a hash waits at most ten seconds for a turn, and server init
writes signup_open: false.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Percentiles are read from the histograms kept with the usage sums and shown
per group on the usage tab and in the people page's panels. Requests refused
by a limit are recorded as 429s against their owner, free of charge, so a team
hitting its limits shows as such. Usage can be grouped by model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An organization's members, limits and use are a panel of the people page,
where they can also be changed, so the administrator's menu loses its
Organizations entry and /org redirects there. Managers keep /org.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A team's members, quota and spend were on three pages that linked to each
other. They are three tabs of one page now, each at its old address, and
choosing anything in the tree opens its limits, capacity and use beside it,
editable in place.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model with replicas on a card and on CPUs answered at either speed with no
way to tell which. The picker names where each model runs, each reply names
the hardware that served it, and every answer carries x-llmgw-hardware.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every minute the agent reads nvidia-smi's full report, its NVLink error
counters and the kernel's XID lines. They are published per card,
shown on the node page and the GPU map, charted in Grafana, and five
alerts watch them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The cluster page draws every card coloured by utilisation, memory or
temperature, and lists replicas whose every card is under 5% busy.
llmgw_gpu_assigned says which replica, model and owner hold each card.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
They were read once, at agent start, and published unchanged since.
Each sample now carries utilisation, memory and bandwidth in use, the SM
clock and uncorrected ECC errors. The node page charts each card, the
cluster page the fleet by class, Grafana has a GPU row, and three GPU
alerts are added.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Each engine has a list of builds, best first, from what its project
publishes and how each was compiled: llama.cpp CUDA 13.3, CUDA 12.8
(driver 570 for the GPUs it ships as PTX), then Vulkan, never the CPU
build on an NVIDIA node; vLLM CUDA 13, then CUDA 12.9, and for GPUs
older than every build of the version asked for, the newest built for
them (V100: 0.19.1). The node takes the first its hardware meets and
that installs and computes on the GPU, moving on when one does not, and
reports the version it runs. GPU processes are read from nvidia-smi's
full report, so Vulkan engines count as on their GPU.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A node that cannot run the CUDA 13 build installs the same version's
CUDA 12.9 build from vLLM's GitHub release, with PyTorch's cu129 torch;
the install computes on the GPU before it counts. The build can be
chosen per node — auto, cu13, cu129, none — on the cluster page, with
ctl engine-build, or over the admin API; the control plane keeps the
choice and tells the node at once.
Refs #2.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The explorer fetches a safetensors set with its config and tokenizer;
the mirror gives out a version's manifest only once nothing is
downloading and every shard its index names is there, with each file's
sha256, which nodes check; mirror add takes several URLs; a repository
page is refused as a source.
Refs #2 (item 9).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A taken address on PostgreSQL fell through to a generic error, because
the store looked for SQLite's "UNIQUE". It checks code 23505 too. The
done page tells someone whose address had an account to sign in or ask
for a reset, without saying which case it was; other failures give a
logged reference.
Refs #2 (item 6).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
engine_overrides give nodes picked by GPU class, id or labels their own
engine versions; node add --disable-engine keeps one off a node; what
vLLM cannot do on a node is a warning rather than a refused node; the
CUDA requirement follows the version; a hand-named binary is no longer
replaced by an installed one.
Refs #2 (items 4 and 7).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The form offered no placement and named every GGUF profile cpu. It now
lists the fleet's GPU classes (the freest chosen) and the CPU, a GPU
count and parallel requests, names the profile from the placement, shows
what the model will run as, and refuses the CPU while GPUs are free
unless that is ticked.
Refs #2 (item 1).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An adapters view in the explorer, filtered by the model a held model was
made from; GGUF files told apart as adapter or model by their first bytes;
held adapter files marked; and a model's card attaches and detaches held
adapters of its architecture, refusing models and mismatched architectures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A catalog model with external {url, model, api_key_file} and no profiles
is published as an endpoint by the control plane, the key read from its
file each time; the router rewrites the request's model to the provider's
name and sends its key. It is never scraped nor stale, and nothing is
placed for it. Named in fallbacks, it serves when the cluster has no room.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The quotas page gains a capacity section over the same API methods the CLI
uses; a model's card sets its class, owner, memory and cores, applying only
what changed. Headroom counts the replicas a model could get by preempting
what its class outranks, and the card and ctl model target use it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every agent samples, at most every ten seconds: nvidia-smi's temperature,
HBM temperature, power draw, enforced limit and throttle reasons per GPU;
hwmon's CPU package and board inlet temperatures; and CPU package power
from RAPL's energy counters. Sent with heartbeats, published as metrics
and shown on the node page; unmeasured sensors are absent, never zero.
Observe-only, step 0 of docs/facility-aware-placement.md, which now
describes the agent source and what these readings can and cannot say.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A model's class (critical, standard, opportunistic; unset follows its
scaling) decides what may be stopped to make room for it: a lower class
for a higher, and within a class only what another grew into under load,
for a model below its minimum. Victims are tried on a copy of the
inventory and stopped only if they make the room. Preempted replicas are
drained with a short grace and their model is held back until the room is
used. ctl model resources and class move under ctl model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A profile's cpu_cores is reserved out of its node's CPUs by placement, as
memory is, and becomes the engine's thread count. Agents report their CPU
count in registration and heartbeats. New CPU models must declare memory;
model views show reserved against measured memory with warnings, and
ctl model resources sets both for a stored model.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Headers never wrap, free text is clipped with the whole on hover, full-width
rows are only notes and forms. The cluster page hides columns empty on every
row and shortens node names; People shows two memberships and a count.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The replica count, its nodes and the mode selector sit side by side instead
of stacking in a fixed-width cell, and node names drop the prefix every node
shares, with the full name on hover.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Ready of wanted, how many are starting, and the nodes, from the fleet data
the card already reads, so the list answers it without a click.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Held tab's model name opens a card: each replica with its node, state,
age, CPU and memory, the room for more found by dry-running the scheduler
on a copy of the inventory, and a form for the always-running and
under-load replica counts. An always-running count placement cannot meet
is refused with the reason. Headroom is also on the admin API.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One long page made every visit wait on Hugging Face for a list it might not
show. Each tab now has its own address and does only its own work; only
Held reloads while a download runs, and a refused fetch returns to the form
it came from.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Adding a model took four steps and a YAML file — plan it, fetch it into the
mirror, write its catalog entry with a memory figure somebody estimated,
scale it — and whether it would load was found out at the end.
`llmgw ctl model serve OWNER/REPO` does it as one job in the control plane,
where the registry, the mirror, the catalog and the fleet already are:
- the registry: the GGUF file set matching --quant (Q4_K_M by default) or
--file, leaving out vision projectors and importance matrices, and naming
what the repository does have when nothing matches; gated repositories are
refused;
- the mirror: fetched at the repository's revision, with progress, and each
file's sha256 compared with what the registry published — the mirror is
the one place a corrupted download is caught once rather than loaded on
every node;
- the check: whether the nodes' llama.cpp loads the architecture, and the
memory it needs from its own header;
- the catalog, then a wait for a replica that stops early, with the reason,
when the model is parked or fits on no node;
- a smoke test: one short request to the replica, with its engine key, and
the answer and tokens per second.
ctl starts the job and prints each step as it changes; a download of hours
does not depend on the terminal. --on-demand adds the model without starting
a replica. Jobs are kept in memory until the control plane restarts.
The catalog name a repository suggests moved into the registry package, so
the mirror page and this agree on it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The serve form asked for weights plus half again plus two gigabytes. That
ignores the one thing that varies most: the KV cache. At the same 16k pool,
the models on the live mirror need from 0.44 GB of cache (Qwen2.5-1.5B, two
KV heads) to 5.25 GB (Gemma-2-9B, 256-wide heads) — so the flat guess
overcounted the small models, which on 7 GB nodes means refusing placements
that fit, and undercounted the ones with wide or many KV heads, which means
an engine killed for memory.
The GGUF header already carries the shape. The reader now also takes each
architecture's block_count, head counts, embedding length and key/value
lengths, reading past general.* until the tokenizer — and a header whose
tokenizer comes first and runs past the fetched bytes still returns what it
has rather than failing. The estimate is weights, plus the f16 KV cache for
the context pool (llama.cpp's --ctx-size, which --parallel divides rather
than multiplies), plus a gigabyte and five percent for the engine. It reuses
the header the servability check already fetched, and the form says what it
is made of. Checked against all twelve GGUF models on the live mirror, whose
shapes read back as their published architectures.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Checked on the live cluster: from any node, the mirror's management API on
:8082 answered — list, fetch, delete — with no credentials. Unset
--api-allow meant anyone. It now means this machine only; the control plane
is let in with --api-allow, which `server init` already wrote for the
mirrors it sets up, or by mutual TLS.
A fetch went to whatever URL it was given, and the file server on :8081
then handed the result to anyone: "fetch http://169.254.169.254/..." read a
cloud's instance metadata through the mirror, and a URL on the LAN read
whatever answered there. Downloads started through the API now go to http
and https only, and are refused at dial time — after resolution, on every
redirect — when the address is loopback, private, link-local, CGNAT or
multicast. --allow-private-sources lets them through for a mirror that
really does copy from a server on its own network. `llmgw mirror add`, run
by someone on the mirror itself, is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
install.md explains every choice and is long for someone who wants the
commands. quick-install.md has only those, and what to check after each:
the binary, server init, onboarding GPU nodes (with doctor first and the
script fetched from the release tag, since it is not installed anywhere),
a model from the mirror's explorer, keys and editors, and Caddy. Every
button and path it names was checked against the pages. quickstart.md is
now said to be the from-source, one-workstation path it is; README and the
about page point at the new guide.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vLLM was installed by hand and named in each agent's configuration. The
agent now installs it: a Python 3.12 environment built by uv, which is
itself a GitHub release checked against its checksum, from PyPI resolved as
it stood a day after that vLLM was published and installed with
--require-hashes. The resolved list is kept in the environment. The
environment is built relocatable under a scratch name and renamed into place
once it runs, like an archive.
The default is 0.30.0, whose flags were checked against the ones llmgw
builds. It is built for CUDA 13, so it needs driver 580 and a Turing GPU or
newer, and 25 GB free while it installs; a node lacking one says which, and a
node with no GPU skips it without being shown as a fault. gcc comes with it
because Triton compiles its launcher at the first model load. SGLang
installs the same way when the control plane names a version. llmgw doctor
reports whether vLLM can be installed on the node.
Checked by installing 0.30.0 this way on a test node: 7.7 GB, torch 2.13 for
CUDA 13.0, importing cleanly; the node has no GPU, so vLLM was not started.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The website was a file built by hand outside the repository. It is now a
template in internal/web: every gateway serves it at /about, without a key,
linked from the landing page's footer, and the published site is rendered
from the same template.
The fonts are vendored (Latin subsets, OFL) so the page sends nobody's
browser to Google. The proxy configurations it shows are copies of
deploy/proxy, kept equal by a test. A development build says so instead of
linking to release files that do not exist.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The account page makes a key for Continue, Codex or Cline and shows
what the tool needs with the key, this gateway's address and a served
model in it: Continue's assistant file, written to its own place under
~/.continue/assistants so the person's config.yaml is not touched;
Codex's provider and profile, added to ~/.codex/config.toml without
changing its default model; and for Cline, whose settings live in VS
Code's secret storage, the four fields to paste. Continue and Codex get
a one-line command whose script, served without a session, holds no
secret - the key comes from the environment, never a URL - keeps a copy
of whatever it replaces, and leaves the rest of the file as it was.
POST /v1/responses translates the Responses API, which Codex speaks, to
chat completions and back: instructions, messages, function tools and
Codex's free-form custom tools, their calls and outputs, and usage, as a
response object or as its stream of typed events. It goes down the same
path as any request - quotas, admission, routing, usage. Server-side
tools and stored responses have no equivalent and are ignored.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
While a reply streams the Send button is Stop, which aborts the request
- and with it the one to the engine - keeping what had arrived. Under
each reply: its tokens, from the engine's own count (the page's turn now
asks for include_usage), tokens a second, time to the first token and in
all. A settings fold takes a system prompt, sent before every turn, and
a temperature, which the gateway holds to 0-2 whatever the page sends.
The conversation, the prompt and the temperature are kept in the
browser's storage, so a reload puts them back - the model's turns
rendered again by the server as when they came - and "New conversation"
clears them. The gateway still keeps no conversation.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a phone every admin page was wider than the screen - its tables set
the width - so the browser zoomed the whole page out until nothing on it
could be read. Under 760px a table now scrolls sideways in its own box,
charts stack, form rows fold, and the menus wrap; every page measured at
390px wide now fits.
The cluster page drew a Fabrics heading and its note on a cluster with
no fabric at all.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fetch on a repository opened in the explorer posted a form, and the
page that came back had closed the repository and lost the list's place.
The page's script now sends it asking for JSON: the outcome appears
under the button, and only the mirror's own table is fetched again,
every five seconds while anything in it is downloading. Without script
the form posts as before.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An account's "add to" picker and its button sit on one line; suspend
and remove, the two things done least and the two drawn in red on every
row, are in a menu behind "⋯"; and the box for a new project under
every team, and for a new team under every organization, is folded
behind "+ project" and "+ team" until wanted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
