-
v0.16.3
Stable
gitea-actions
released this
2026-11-03 09:56:29 +01:00 | 41 commits to main since this releaseReplicas counted per profile — seven on cards and four on CPUs under one name, the cards filled first — and every
setting of a served model from the mirror page; how fast a model answers on the CPU, said before it is served, by
the threads it is given rather than the whole machine's memory.Upgrading.
- No migrations.
llmgw ctl upgrade v0.16.3: the control planes carry the replicas per profile, the model card and
the speed in the serve form and inctl model plan; the agents carry the measurement of one core's memory rate,
without which the speed is reckoned the old way, from the whole machine's.
Added
- Replicas counted per profile: seven on cards and four on CPUs, under one
name.llmgw ctl model replicas <model> gpu=7 cpu=4, the console's
Replicas per profile… and the mirror page's model card set how many of a
model's replicas run on each of its profiles, rather than every replica
on the best profile that fits. For a GGUF model with no profile on CPUs
one is added (--cpu-memory), preferred below the cards, so the router
fills the cards first and sends to the CPUs once they are full. A change
starts the replicas short before stopping those over: the model never
runs fewer.ctl modelsshows the counts; a scaling of another total
ends them. - Every setting of a served model from the mirror page. A served
model's row has a Settings link, and its card now changes the context
and longest answer, the cards a replica takes, the replicas on each
profile (with the CPU's speed beside), stops and starts it, restarts its
replicas, and takes it out of the catalog with its files or without —
beside the replicas, class, owner, memory and adapters it held before.
Stopped on the row is now the model stopped and kept, started again as
it was, rather than its scaling set to nothing. - How fast a model answers on the CPU, before it is served. The serve
form says, as the cores are changed, about how many tokens a second one
reply gets on the CPU and on one card of the fleet's fastest class, and
warns when it is slow (under 15) or very slow (under 5, a reply of 500
tokens taking minutes). Very slow on the CPU is refused until "run on the
CPU anyway" is ticked, GPUs free or not.llmgw ctl model planand the
console's Hugging Face screen show the same figure for every way a model
can run, andctl model serve --cpusays it in its log. - The speed follows the threads a replica is given. Each agent now
measures what one core reads beside what all of them read: a replica of
four threads on a large server reads at about four cores' rate, not the
server's, which the estimate used to assume and overstated several times.
Calibrated on the PoC by llama-bench at 1, 2, 4 and 8 threads (within 15%),
and a model of experts reads its experts' share, from the GGUF header.
Agents need upgrading to report one core's rate; until then the old
estimate is used.
Downloads
- No migrations.
-
v0.16.2
Stable
gitea-actions
released this
2026-11-02 08:42:29 +01:00 | 45 commits to main since this releaseStandby control planes that answer clients themselves, with rate limits and budgets kept across them; FreeToken, an
engine for models larger than the card; a served model's longest answer changed in place; and fixes found putting
the router under load and fetching large models: a multi-file model fetched once per node however many replicas
start there, downloads that carry on where they were cut off, and a router that does twice the work on two cores.Upgrading.
- No migrations.
llmgw ctl upgrade v0.16.2: the control planes carry active-active routing (off until
llmgw ctl config set router.active_active true) and the router fixes; the agents carry the set fetched once
per node and the set's progress and checking; the mirror carries the resumed downloads.
Added
- Standbys that answer the client API themselves. With
router.active_active: truein the cluster's settings, a standby control
plane routes/v1to the leader's replicas instead of passing every
request to the leader, so more machines carry the traffic. It hears the
replicas from the leader every two seconds over the agent port (no engine
keys travel) and reports back what it used, so the leader does not evict
a replica busy behind a standby, and the on-demand models it was asked
for. Rate limits are the cluster's: each router's request and token
counts reach the others through the leader every two seconds, so a key
spread over several control planes gets its limit, not that many times
it. Budgets are brought up to the store's every 30 seconds. Off by
default; the portal,/adminand/mcpstill go to the leader. - A served model's longest answer, changed in place.
llmgw ctl model context <model> --max-output N(andmax_outputonPOST /api/v1/models/{name}/context, and the console's Context length… form)
sets the longest answer on its own: the catalog's limit, in force at once,
with no replica started again. With a context it is set with it. Strata's
context can now be changed too (its setup's--context), so neither
needs the model removed and added again. - FreeToken, an engine for models larger than the card. Mixture-of-experts
models several times a card's size, from one card, with the experts in RAM
and streamed over PCIe — read from Hugging Face safetensors checkpoints as
published (DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8, gpt-oss, Gemma 4,
MiniMax), with batching. Installed like vLLM on every node it runs on: a
card of compute capability 8.0 or newer, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with; older cards are passed over (tried on a
Turing card, where 0.2.1 has no kernels). Profiles are one card
(engine: freetoken); the key goes in its environment, and flags that read
files or fetch are refused. Readiness now also waits for an engine whose
/healthanswers 200 while sayingloadingin its body. FreeToken is
followed release by release: a weekly workflow fails when PyPI has a version
newer thanDefaultEnginesinstalls. - How much of each GPU works, beside whether anything does. The busy
figure is nvidia-smi'sutilization.gpu— the share of time any kernel
ran, which a kernel on one SM fills — and it is now said to be that: on
the dashboard, the node page, the metric's help and Grafana. Beside it,
each card's draw against its power limit (the efficiency table, the fleet
and node charts, the console's GPU work chart), and where the card's
profiling counters can be read, its SMs, SM occupancy, tensor cores and
memory interface:llmgw_gpu_sm_active_percent,
llmgw_gpu_sm_occupancy_percent,llmgw_gpu_tensor_active_percent,
llmgw_gpu_dram_active_percent. The agent reads them from NVML's GPM on
Hopper and later, and from a dcgm-exporter on the node elsewhere
(discovery.dcgm_exporter, defaulthttp://127.0.0.1:9400/metrics,off
for none). The GPU map colours by this "work" by default — SMs active
where read, power against the limit otherwise — and keeps busy as a mode.
On cuda01 decode read 92% busy with the SMs at work 69% of the time.
Fixed
- Several replicas of a multi-file model on one node fetched it once
each, over one another. Four replicas placed on a node's four cards
wanted the same safetensors set at the same moment, each wrote the same
.partfiles, and all but one failed to start ("rename … no such file")
until a retry found the set finished. A set is now fetched once per node
and every replica waiting on it gets it, as single files always were. - A model of several files showed one file's download as the whole. A node fetching a set reported the part it
was on — "59% of 46.1 GB" for a 188 GB model — instead of the set: it now reports the set's bytes against its
whole size. - A set's files checked with nothing said, one at a time. After the last byte arrived a node checked each file
against the mirror's checksum, 195 GB on one core for eight minutes, while the replica still read "fetching 100%".
It now checks four at once and says so:checking 40% of 195 GBinctl replicas, "Checking the weights" in the
chat page; and a check that takes long is not taken for a stuck download. - A mirror download the registry cut off started again from nothing, or not at all. Hugging Face's CDN ended a
60 GB transfer at 23 GB (stream error … CANCEL), the mirror deleted what had arrived, andctl model serve
failed. The mirror now tries again up to six times, carrying on from where it got to (Range), keeps what arrived
when it gives up, and carries on from it after a restart too. - A model from an outside provider took three or four requests at once. Admission read every request in flight to
an endpoint it does not measure as a queue, so an external model (OpenAI, another llmgw cluster) was held to what a
saturation of 0.85 allows of "in flight / 4"; the rest waited and timed out with 504. An outside provider queues for
itself and says 429 when full, which its breaker already hears: it no longer counts towards saturation. - A valid key told to slow down under load, every thirty seconds. When a busy key's cached entry expired, every
request in flight missed the cache together and each took one of the address's allowance for unknown keys: past a
hundred or so streams at once, a valid key got 429 "too many unrecognised api keys". Requests missing the same key
at once now share one read and one allowance. - A streamed answer was one write to the client a token. Each event was flushed alone, a system call each: a
third of a busy router's CPU. Events already received from the engine now go out in one write, with no token
waiting for another; on the PoC's two-core leader the ceiling rose from ~575 to ~700 streams' requests a second,
and the first token at 128 at once from 91 to 45 ms.
Downloads
- No migrations.
-
v0.16.1
Stable
gitea-actions
released this
2026-10-30 13:26:47 +01:00 | 63 commits to main since this releaseThe mirror page's Strata tab, Strata's ready-made packs served with Strata fetching them, and two fixes found
serving Strata's models: repositories that keep each size in a directory, and DeepSeek on cards its engine does not
run on.Upgrading.
- No migrations. The control planes carry the tab, the plan and the serving; the agents carry the refusal of a
Strata profile its card cannot run (llmgw ctl upgrade v0.16.1). Mirrors need nothing new.
Added
- The mirror page's Strata tab. Every family and size Strata runs, with its download, the RAM it keeps and where
it would run here; a GGUF size fetched into the mirror in one press (checked against Hugging Face's checksums),
and the packs Strata fetches itself — NVFP4, DeepSeek V4.1 — served at once, their repository the source.
ctl model planandctl model servetake those repositories too, and the explorer names astrata-ple
repository for what it is: part of a pack Strata knows, or one Strata 0.2.7 does not.
Fixed
- A repository that keeps each quantisation in a directory of its own (ISTA-DASLab's
IQ2_XS/,IQ3_S/) listed
no weights: its file tree is now read recursively, by the explorer, the plan andctl model serve. - DeepSeek V4.1 offered on a card its engine was not built for. Strata's DeepSeek engine runs on RTX 30 (compute
8.6) and newer; on a Turing card the tab and the plan offered it, and Strata's setup refused it once placed. Now
neither offers it there, the tab says why, and the agent refuses such a profile before starting it.
Downloads
- No migrations. The control planes carry the tab, the plan and the serving; the agents carry the refusal of a
-
v0.16.0
Stable
gitea-actions
released this
2026-10-30 11:30:03 +01:00 | 68 commits to main since this releaseWeights pinned to the commit they were added at and checked, by the nodes and the mirror, against the sha256
Hugging Face published; the documentsllmgw-kbsends embedded by the gateway; Strata offered by the plan,ctl model serve, the explorer and the serve forms; and cluster upgrades that tell the nodes together, say how far they
are, and do not fail a node that came back on the version.Upgrading.
- No migrations. Upgrade the control planes, then the nodes (
llmgw ctl upgrade v0.16.0), then each mirror's host
(sudo llmgw upgrade v0.16.0 --service llmgw-mirror). - A node checks what it fetches against the catalog's checksums only once it runs this version; until then it
fetches as before. - A model added from Hugging Face is now pinned when it is added, which needs the registry reachable from the
control plane; one it cannot be looked up in is refused, saying why. Models already in the catalog are left as
they are. - A Strata profile must say
--sizeas well as--family, and both must be ones Strata runs. llmgw-kbsends text alone to a gateway of this version; an olderllmgw-kbkeeps working, sending vectors.
Added
- Strata offered where models are chosen. For a repository of a family Strata runs,
ctl model planoffers "one
card with Strata, experts in RAM" with the RAM it takes,ctl model serve --engine strata(or the plan's choice)
builds its profile, the Hugging Face explorer lists it as running here, and the mirror's serve form serves such a
version on Strata. Strata's families and sizes are known: the catalog refuses one it does not run and reserves the
RAM a size keeps its experts in. - The gateway embeds the documents
llmgw-kbsends. A document may be sent as its passages' text alone; the
gateway embeds them through its own/v1/embeddingsas the caller, 32 at a time and four batches at once,
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
of a vector down and up again, some thirty times more.llmgw-kbdoes so with a gateway that says it can
(server_embedsin a collection's answer);--embed-herekeeps the old way. - Weights pinned to a commit, and checked against what Hugging Face published. A model added from Hugging Face
(hf:, a resolve URL on a branch, or a repository vLLM or SGLang downloads) is pinned to the commit it is at
then — the file's URL at that commit, or--revision— and each file's published sha256 goes in its origin. A node
refuses and deletes a copy that is not it, naming both checksums; the mirror does the same for whatctl model servefetches, andllmgw mirror addfor a Hugging Face URL (--sha256for others). Nodes need this version to
check; older ones fetch as before.
Fixed
-
A cluster upgrade that sat silent, or stopped on a node that had upgraded.
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
seconds; before, each was told in turn, up to five minutes each, with nothing on the screen. - The control plane takes the node registering again on the version as done: the agent's answer, sent a moment
before it exits, could be lost, and a node that had upgraded was failed as "older than v0.10.4". - The agent's download no longer ends with its connection (a control plane restarting, leadership moving), and the
control plane waits longer than the download may take; a second request while one runs is refused. - A cordoned or draining node upgraded is done; a node whose idle model stopped meanwhile is done once nothing on it
is starting; a node not connected is passed over and named at the end rather than stopping every node after it. - The console's node upgrade waits as long as the control plane, and offers no development version.
- systemd never gives up restarting the agent (
StartLimitIntervalSec=0), in units written from now on.
- The nodes of a wave are told together, each saying when it is done, and what is still awaited is said every 30
-
Strata. One named by hand in
agent.yamlwas never reported, so never placed on; its engine, which leaves the
server's process group, is now stopped with it; Pascal and Volta, which its CUDA 12 build runs, were refused. -
A name with a slash in it, asked through a standby control plane.
llmgw ctl kb delete Demo/hron a standby
was answered 405: the standby passed the request on to the leader withDemo%2Fhrunescaped, two segments instead
of one. The path now reaches the leader as it was sent.
Downloads
- No migrations. Upgrade the control planes, then the nodes (
-
v0.15.3
Stable
gitea-actions
released this
2026-10-29 21:07:44 +01:00 | 77 commits to main since this releaseA model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
Upgrading.
- No migrations. The control planes carry stop, start and purge; the nodes need nothing new.
- The mirrors carry the fixes: on each mirror's machine,
sudo llmgw upgrade v0.15.3 --service llmgw-mirror.
A mirror with a peer then works out the checksums its older files lack, once, which reads every such file.
Added
- A model stopped and kept, and a model removed with its files.
llmgw ctl model stop|start MODEL: none of its
replicas run, it stays in the catalog with its files, and start runs it again scaled as it was; requests are told an
administrator stopped it.llmgw ctl model remove MODEL --purge(DELETE /api/v1/models/{name}?purge=true) also
deletes every version of it on the mirror, a second mirror following, and the nodes' copies once its replicas have
stopped; what another model uses stays. Both are in the console's model menu.
Fixed
llmgw mirror add-peeron a mirror with a machine of its own. It set the second mirror up and then
failed for want of the control plane's server.yaml; now it prints thellmgw ctl config set mirror.peers
to run on a control plane.- Models fetched before checksums were kept reach the second mirror. A mirror kept in step copies only what it
can check, and those files had no sha256; a mirror with peers now works theirs out and records them, once. - A file removed with
llmgw mirror removeis gone from the mirror's answers too. The running mirror kept
reporting a file it had fetched as present for a week after it was removed, so a control plane could place a
model on a file that was not there; and the removal was not recorded, so a mirror kept in step copied it back.
Downloads
-
v0.15.2
Stable
gitea-actions
released this
2026-10-29 16:58:02 +01:00 | 83 commits to main since this releaseA served model moved to more cards, or fewer, while it is served.
Upgrading.
- No migrations, and nothing changes until the command is used. The control
planes carry it; the nodes need nothing new.
Added
- A served model moved to more cards, or fewer.
llmgw ctl model gpus MODEL N [--profile P] [--in-place],POST /api/v1/models/{name}/gpus, and
GPUs… in the console's model menu. How many cards of one node each replica
uses: tensor parallel for vLLM, SGLang and TensorRT-LLM, checked against
the model's KV heads, and the layers split for llama.cpp. Refused for a
count no node has, or whose cards would not hold the model and its
context, with the figures; its replicas are then started again on it, one
by one, beside the old — or, with--in-placewhen there is no room
beside them, each stopped first.
Downloads
- No migrations, and nothing changes until the command is used. The control
-
v0.15.1
Stable
gitea-actions
released this
2026-10-29 16:09:12 +01:00 | 85 commits to main since this releaseA 125B model on one card, a second mirror, and weights that run code
refused. Strata runs as an engine: a mixture-of-experts model with its
experts in RAM, served from one 24 GB card. A second model mirror keeps in
step with the first, so an HA install has no single machine left. Pickled
weights are refused unless a model says otherwise. A served model's context
can be changed in place. The explorer says how fast a model would answer
here before it is fetched, and a card in the wrong PCIe slot is said so.Upgrading.
- No migrations. Upgrade the control planes before the nodes, for the
pickle refusal below. - A model whose weights are in a pickle format (
.bin,.pt,.pth,
.ckpt,.pkl) stops being placed: its node refuses the source. Give it
its safetensors or GGUF, or add it again withallow_pickle: true
(llmgw ctl model add -f) if its files were checked. Look at the
models' sources before upgrading the nodes: one ending in
.ckptor.bin, or a mirror version holdingpytorch_model.bin. Upgrade the control planes first: a node on
this version refuses those weights until its control plane can say
allow_pickle. - Strata is opt-in: nothing installs it until
enginesor an
engine_overridesentry names a version (0.2.7).
Added
-
Strata as an engine: a 125B model of experts on one card. Strata
(github.com/Niko1221/Strata) keeps a model's experts in RAM and the ones
used most in VRAM. On an RTX 6000 24 GB with 251 GB of RAM it serves
Qwen3.8-Flash-Next 125B (IQ2_XS) at 70–87 tokens a second to one client;
more clients share about that, its slots buying fairness rather than
throughput on one card. It runs a few families only — Qwen3.8-Flash-Next
and its variants, Qwen3.6-35B-A3B, DeepSeek V4.1-Flash — each named in a
profile's args with--familyand--size
(--family qwen --size IQ2_XS --context 32768). The source is the
directory of the model's GGUF files, or the repository, which Strata then
downloads itself, pinned. The agent installs it on an NVIDIA node of
x86-64: its source pinned by digest, in a Python environment, and its
release's engine checked against the digest GitHub publishes — the CUDA 13
build for Turing and newer on driver 580, the CUDA 12.9 build for Pascal
to Ada, a V100 included, on driver 575. A replica prepares its model the
first time (minutes; its draft layer is a 5 GB download) and loads in
about a minute and a half after. One card of one node per replica, for
now; not in an offline bundle, its packages coming from PyPI. -
A served model's context, changed.
llmgw ctl model context MODEL TOKENS [--slots N],POST /api/v1/models/{name}/context, and Context
length… in the console's model menu. The catalog, the engine's arguments and
the memory change together, checked against what the model was trained for
and what its cards hold; its replicas are then started again one by one. The
models API showscontext_lengthandmax_output. -
A second model mirror, kept in step. In a three-machine install the setup screens put one on a database machine
(--second-mirror 1|2). An install with one mirror gets one withsudo llmgw mirror add-peer user@host.- In step: each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
removes what the other removed, even if it was down at the time (llmgw mirror serve --peer). - Control planes: they name both (
mirror.peers) and use whichever answers. - Nodes: while one mirror is down, a node is given a model's file on the other.
This takes away the last single point of an HA install: with the first machine off, models are still added and
fetched. - In step: each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
Fixed
-
An engine that computes in a process of its own group was not its
replica's. A GPU replica read as running on the CPU, and the memory it
held on its card as somebody else's, which placement then kept free of
other work. A process the replica started, at any remove, is now the
replica's. And what a replica computes on, once sent, was never sent
again: a wrong reading kept from before an agent's restart stayed wrong.
It is sent again when it changes, and read again when a GPU replica was
kept as the CPU. -
VS Code's sign-in lost its place through the sign-in page, so the
prompt to confirm came every time; a link to the same gateway by plain
http could take an https setting down to http; and a new build of the
extension was not installed over an old one (the command now says
--force). -
The install guide's recordings were not checked for secrets, only
the page; every frame is now. -
MCP server fields could not be emptied, an address could carry a
password, the cluster's own server could not be changed from the
console, and a grant to a key that does not exist was said done. -
A guardrail service two policies named was asked as the first said,
not the stricter; anullanswer read as safe; and withrestorethe
request log kept the answer with the redacted values back in it. -
What an engine downloads by a Hugging Face id was not held to the
pickle refusal — an adapter by its id, a TensorRT-LLM model's; a model
with either now needsallow_pickle. -
server initrun again on a machine already set up was refused its
memory, as on an 8 GB install fixing its certificate with--host, and
judges that would not be installed (no mirror) were counted. -
Removing an MCP server left its sealed credential in the store, and
replacing a credential under another header kept the old one sent. A
server's secrets now go when it no longer reads them. -
A cloned repository could point the VS Code extension at another
gateway.llmgw.gatewayUrlis a machine setting now, never taken from a
workspace. -
A node's memory bandwidth could be measured in its cache on a busy
node, and was 0 after a reconnect until the next heartbeat. The buffer is
at least eight times the processor's last cache. -
A project kept in the cluster could have its questions embedded outside
it. The similar-question cache now asks the embedding model as the
question's project would, external fallbacks refused, and skips a model
that is itself outside. Its comparing no longer holds the cache's lock,
which every project's lookups waited on. -
Restored values were left as placeholders in streamed tool calls, in
completions and in reasoning, and a value with a quote in a whole answer's
tool-call arguments broke their JSON. -
An outside guardrail service read what
redacthad taken out, and a
streamed answer was sent to it twice. The docs now say a service stops a
streamed answer only understream: buffer_all. -
A replica found slow once could stay unused. Its time to first token
is forgotten after five minutes without an answer, so it is tried again. -
Weights that run code got past the refusal in four ways. A mirror
version already on a node was used without being looked at again; a
?queryor#fragmenton anhf:source hid the file's name;.pt2was
not on the list; and a profile could write--load_format,
--trust_remote_codeor an abbreviation such as--trust-remote, which
vLLM and argparse read as the flag, or give SGLang--load-format. All
refused now. TripoSR's preset saysallow_pickle, without which it could
not be added, and a shared ComfyUI environment takes its first member's. -
A temperature written as a string was refused. Since the GenAI traces
read it,"temperature": "0.7"got a 400 before reaching the engine; it
is read loosely again, for the trace only. -
A reused answer could carry another caller's personal data. With
restore: trueand the answer cache on, two callers' requests that
redacted to the same text shared one answer — with the first caller's
address put back in it. Such requests are now never answered from the
cache nor kept in it. -
Granting a server's tools could take others away; revoking could close
a level to every tool. A level with no tool list sees what the levels
above grant; giving it its first list made it see only that list. And a
level left with an empty list was closed to every tool, which the screens
did not show. Now a first list starts from the nearest list above, taking
a server away leaves the rest or goes back to no list, the screens say
when a level above keeps granted tools out of reach and which levels are
closed, and the console reads and writes the level at once rather than
from what it loaded. -
Similar questions about different pictures got each other's answer.
A question with a picture, a sound or a file in it is reused only when it
is the same request.
Added
- A card in the wrong PCIe slot is said so. Each agent reads how every
card is linked — the fastest generation card and slot allow together, and
the lanes in use — against the card's own most, from the kernel (the
driver's figure is what card and slot allow together); a card on fewer lanes or
an older generation is a node warning, with what it carries and what it
could, on the dashboard, the console and inllmgw doctor. - And on a web page,
/admin/mcp. For administrators: the same list,
adding a server with its credential sealed, testing it and listing its
tools, granting them in the tree beside that level's other grants and
limits, revoking, changing, replacing the credential, turning it off,
removing; and the line that points an agent at this gateway's/mcp.
In English and Turkish; every change in the audit log, the secret not. - The company's MCP servers in the console. A screen of its own: each
server with its state, who has its tools, its rate, the names of the
credentials it is sent and its address; add one with a bearer token or
a header kept sealed, test it and list its tools, grant them to a team,
project, organization, person or key beside that level's other grants,
revoke, change, replace the credential, turn off, remove. - And on the cards, as this cluster would lay it out. The explorer's
GPU tok/s and a repository's files give one reply's speed for each
layout the planner offers here — one card, tensor parallel across a
node's cards, llama.cpp's layer split, a pipeline across nodes on a
fabric — the fastest of them, with the layout in the tooltip. From each
card's memory bandwidth by its model name, NVLink or PCIe between a
node's cards, and InfiniBand or RoCE between nodes; a cluster of mixed
classes is judged class by class. A model of experts reads 1.6 times its
active share, as Qwen3-Coder-30B-A3B did on cuda01's RTX 6000 (144
tok/s; qwen3-vl-8b Q8: 62). - How fast a model would answer on a CPU, before it is fetched. Each
agent measures how fast its node's memory reads when it starts (Host
getsmem_bandwidth_gbs, the node's viewmem_bandwidth_gbs); the
explorer's CPU tok/s column and a repository's GGUF files say about
how many tokens a second one reply gets on the fastest node that holds
the model — the bandwidth over the bytes a token reads, its active part
for a model of experts. Calibrated on the PoC against llama.cpp's own
timings: within 15% from 1.5B to 14B. - The explorer lists what this cluster can run. Judged by the nodes
as they are now — every card's memory and compute capability, every
node's RAM — through the same planner as serving a model, rather than a
size picked from a list. A model too big for any node, in FP8 without a
card of 8.9, in MLX, or in neither GGUF nor safetensors, is left out and
counted; the page says what it judged by. With cards, safetensors repositories are
listed beside GGUF. On a repository's page a file too big for the
cluster is shown with what it needs and no Fetch button. - Weights that run code are refused. Python pickles —
.bin,.pt,
.pth,.ckpt— run whatever they hold when loaded, which is how the
malicious models on Hugging Face work. The node refuses such a source,
or a mirror version with one in it, before fetching it; adding such a
model is refused; vLLM and SGLang downloading a model by its id read its
safetensors only; the mirror page marks the file.allow_pickle: true
on a model lets it through on purpose. The mirror's explorer no longer
lists a repository with nothing a node loads (no GGUF, no safetensors)
while "runs here" is ticked, and a repository's pickle files are not
offered, only counted. BACKLOG M lists what is next:
checksums against Hugging Face everywhere, its own scan, trusted
publishers, engines without the internet.
Downloads
- No migrations. Upgrade the control planes before the nodes, for the
-
v0.15.0
Stable
gitea-actions
released this
2026-10-25 18:17:57 +01:00 | 116 commits to main since this releaseThe company's tools and editors through the gateway, and guardrails that
can lean on others. Its own MCP servers sit behind llmgw's/mcpunder the
guardrails, a rate and a breaker. VS Code gets the gateway's models from an
extension the gateway serves. A guardrail policy can ask an outside
service, and can put redacted values back in the answer. Chat goes to the
replica answering soonest. A similar question can get the answer already
given. A cached prompt can be priced apart. Traces follow the OpenTelemetry
GenAI conventions, and Kong is written down from trying it. Also an install
guide with every screen, at/install.Upgrading.
- Migrations 0049 (
mcp_servers), 0050 (their rate and breaker) and 0051
(projects.cache_similar), applied when the control plane starts. llmgw server initnow refuses a machine that would serve models without
the memory for them: the starter model and the judges need 12 GB on one
machine, where 8 GB was said before.--no-model,--no-guard-modelor
--no-nodemake it fit. A running cluster is not checked again.- Traces carry the GenAI conventions'
gen_ai.*attributes beside the
llmgw.*ones, which stay; docs/install.md, "Tracing", lists them. - Nothing else changes until it is turned on:
servicesandrestorein a
guardrail policy,keys cache --similarwith
router.cache.embedding_model,cached_per_1kin a price.
Added
-
Answers to similar questions.
llmgw keys cache --similar 0.95also
gives a chat question the answer already given to one like it in the same
conversation, compared byrouter.cache.embedding_model, as
x-llmgw-cache: similarwith the similarity. Held in memory beside the
exact answers; nothing written to the store but the setting. -
A cached prompt costs the cached rate.
cached_per_1kin a price
charges the prompt tokens an engine or provider read from its cache (the
answer'scached_tokens) at their own rate; without it they cost what any
prompt token does, as before. -
Chat goes to the replica answering soonest. The router keeps each
replica's time to first token, smoothed over its last answers, and the
chatprofile scores by it beside the queue and the KV cache: of two
replicas of one model, the one on the slower card or the busier host gets
less, and a replica not yet measured is tried. -
llmgw and Kong, written down (docs/kong.md): Kong 3.9
in front of llmgw by a plain route or itsai-proxy, each team's llmgw
key passed on; Kong as a provider behind llmgw; and that Kong 3.9 counts
no completion tokens of a streamed answer, so bill from llmgw's usage. -
A guardrail that is someone else's service. A policy's
services
asks Azure Content Safety, Lakera, NeMo or a company's own classifier over
HTTP: a body template, keys from files, dotted paths to where the answer
says unsafe and why, on input, output or both, block or flag, and the
policy'sfailwhen it does not answer. -
Redacted values put back in the answer.
restore: trueon a guardrail
policy's input: the model reads[EMAIL_1], the client reads its own
address in the reply, streamed or not. Only the request's own values, held
in memory for its life; the output rules read the answer as the client
will. -
Traces in the OpenTelemetry GenAI conventions. An inference request
and each attempt at an engine carrygen_ai.operation.name, the requested
and the answering model,max_tokens,temperatureandtop_pwhen set,
the session asgen_ai.conversation.id, and the input, output and cached
tokens, and an attempt the engine that answered asgen_ai.provider.name.
Tools that read the conventions show llmgw's calls as model calls with no
dashboard of their own. No prompt or answer text is ever in a span. -
llmgw for VS Code. An extension of llmgw's own, served by every gateway
at/setup/vscode/llmgw.vsixwith that gateway's address in it, made from
the binary itself: plain JavaScript, no dependencies, no build, and so no
marketplace needed at a site with no internet. It offers the gateway's chat
models to VS Code's language-model API — the chat's model picker, agent
mode, and every extension that asks VS Code for a model — with the sizes
and the tool and vision capabilities the catalog says, streamed, with tool
calls and their results both ways. VS Code is a tool on the account
page's Connect your editor: a key, the one-line install, and an Open
in VS Code link that hands the key to the extension, which keeps it in
VS Code's secret storage. llmgw: Sign In opens that page with a state
of its own; a link without it is confirmed first, the gateway named. -
An install guide with every screen, at
/install. From three bare
Ubuntu machines to a first answer: what each machine needs and how to
check it, the release fetched and checked, the setup screens one by one,
the system's state, and a first request with a key from the account page.
The screens are a real three-machine install, recorded and kept as
terminal frames — sharp at any size and a few kilobytes — with each
session played back on request; the administrator's password and every
key were masked before it was kept./aboutsays more of what llmgw does
(documents, guardrails, sign-in, high availability, pictures and 3D), and
its install steps are the ones of today, leading to the guide. -
The company's MCP tools under the guardrails, a rate and a breaker.
- The answer: the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
ticket is withheld as a pasted one would be, and personal data is redacted. - The arguments: they are read before they leave, so a secret pasted into one does not reach the company system.
- Rate:
ctl mcp add --rate Nallows each caller N calls a minute to a server. - Breaker: five failures in a row take a server out for a minute, its tools off the list.
- The answer: the caller's policies read a tool's answer as the prompt it becomes. An instruction planted in a
-
How fast the judge is, measured.
llmgw ctl guard judge check(andPOST /api/v1/guard/judges/check) works
out whether the judge keeps up with requests. It times three judgments of a 280-token prompt, then a burst at once,
and sets both against the policies' wait. It says ok, tight or slow, about how many requests a second can be judged,
and what to do: more cores, more judges, a GPU judge, a longer wait, orfail: open.server initruns it once the
judge is up. If the machine is too slow it says so before the next steps, not after the first refused request. -
The company's own MCP servers through llmgw's
/mcp.llmgw ctl mcp add jira --url … --secret-file Authorization=FILEregisters one;/mcpthen lists its tools asjira.searchbeside llmgw's own. Only keys granted
them see them:llmgw keys limits --tools 'jira.*', patterns at any level of the tenancy tree. None granted anywhere
is none. The server's credentials stay on the control planes, kept in the store sealed or read by reference. The
server is told the calling key, person and project. Each call is bounded in time and size, and its progress passes
on to a streaming client. Every call is in the audit log with its arguments' size and digest, never their text. The
admin API has/api/v1/mcp/servers. -
The cluster as a read-only MCP server. Registered as
llmgw ctl mcp add llmgw --url llmgw:adminand granted
asllmgw.*, its tools say whatllmgw ctldoes:cluster_status,list_nodes,list_models,list_replicas,explain_placement,node_logs;- nothing that changes the cluster.
It is answered in the control plane's own process, so there is no token to keep and every control plane serves it.
The admin API serves the same atPOST /api/v1/mcp.ctl mcp add --bearer-filekeeps a token, sent as
Authorization: Bearer. -
Large models across cards and nodes. For a model of experts on more than one card, the plan now adds vLLM's
--enable-expert-parallel. It reads the expert count fromconfig.jsonunder any of its names, and sizes a model
that also reads pictures by itstext_config. Several nodes are offered in any number, not only powers of two. Each
way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This
shows inctl model plan, in the console's Serve a model, and in a serve job's first step. The serve form checks that
a longer context still fits.ctl model serve --env KEY=VALUEputs engine variables such asNCCL_DEBUG=INFOon
every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric. -
Machines of the judges' own in a three-machine install. The setup screen asks for up to two judge machines after
the store's two;server inittakes--judge-node user@hostfor each. Each is added as a node labelled
llmgw.role: judge, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core
control plane that was also a node, 38 of 60 requests were refused for want of a judgment. With the judges on a node
of their own, 2 were. A node with anllmgw.rolelabel takes only profiles whose node selector asks for that role,
so no other model is placed on the judges' machine.
Fixed
-
A machine too small for the models it would serve is said so first.
The control plane that is also a node, serving the starter model and two
judges, needs about 10.5 GB. On 8 GB the install finished every step and
then had no room for the starter. Now the setup's checks fail its memory
with what it would serve, andserver initrefuses before it installs
anything, naming--no-model,--no-guard-modeland--no-node. -
Turkish pages are Turkish to the end.
- Errors: those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
catalog", the settings changed meanwhile, and others) are now in the page's language. - Percent: a percentage is written the language's way:
%93in Turkish,93%in English. - Footer: the time at the foot of a page is in the language's date form.
- Stale forms: they are refused with a sentence in that language rather than "bad request".
- Errors: those from the store, catalog and mirror that a page shows ("not found", "not empty", "already in the
-
llmgw keys limitsno longer clears the limits it was not given. Setting one limit at a scope wrote the whole
row, so--tools 'jira.*'alone took away that scope's rate, budget and model list. Only what is given changes now.
--replacewrites every limit, as the console's limits form does. -
The plan refused tensor parallelism past a model's KV heads. Qwen3-235B, with four KV heads, had no way to run
on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now
offered, as is one that divides them. -
A big model on a slow mirror was failed while it downloaded. The startup timeout counted the download, so
hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing.
The timeout now runs from when a node's download last moved. -
The judge init installs is no longer replaced by the leader's own. Just after the control plane started, the
leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so
a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes
after it starts before it serves a judge itself. If someone adds one while it fetches, theirs stays. -
A judge the leader serves by itself is sized as init's is. When a guard policy names a judge the catalog lacks,
the leader serves it. That judge reserved 3 GB and no cores, and ran llama.cpp's prompt cache, which grew one to
3.2 GB. Without cores of its own, a judgment on a busy node missed its 5 s budget, andfail: closedrefused the
request. It now gets whatserver initgives its judges: 2 GB, two cores each (fewer on a small node), and no
prompt cache. Every judge served throughmodelserveruns with--cache-ram 0.
Downloads
- Migrations 0049 (
-
v0.14.9
Stable
gitea-actions
released this
2026-10-19 16:17:43 +02:00 | 144 commits to main since this releaseSign-in set up from the console, with Active Directory tried before it is
kept. A node's console now shows what the machine is doing, and a node's web
page shows what it runs. Also fixes found trying Active Directory against a
Samba domain controller on the test cluster. Sign-in through it, nested
groups and a disabled account suspended by the sync all worked there.Upgrading.
- No migration.
auth.ldapandauth.oidcare in the cluster's settings, as before. A
block written intoserver.yamlon a cluster with a store was never
read; the log said so. Put it in withllmgw ctl config edit, or set it up
again from the console's Sign-in screen.- A Samba DC's own autogenerated certificate is refused (a negative serial
and no SAN). Give it one from a CA.
Added
- Sign-in set up from the console. Under People → Sign-in,
sudo llmgwshows how people get in: single sign-on,
Active Directory and SCIM, each on or off. It also shows the directory's last sync. E sets up Entra ID from the
tenant and client IDs. D sets up Active Directory, and the directory is tried before it is kept: each server
reached and bound, how many people may sign in, how many are in each group. Nothing is kept if a step fails. The
password, client secret and SCIM token are kept in the store asstore:auth/…, sealed by the master key. The
domain's CA goes into the settings (auth.ldap.ca), so no file has to be put on each control plane. The admin API
has the same:GET /api/v1/auth,POST /api/v1/auth/ldap/testandPUT /api/v1/auth/secrets/{name}. - A node's page says what it runs. It now shows the agent's version, marked when the control plane runs another,
and the engines with their versions. Each GPU row now names the models loaded on that card, and each engine row its
GPUs. sudo llmgwon a node opens on what the machine is doing. A new Overview screen shows the agent's state and
version, each control plane it dials and whether it answers, every GPU with the models on it, the engines running
here, the free space for weights and engines, and the agent's warnings of the last hour. It is read on the node
itself, every three seconds, so it needs no admin token and works while the control plane is out of reach.
Fixed
- A broken
auth.ldapno longer stops every control plane. It is the cluster's setting. A CA file that one
machine could not use madellmgw serverexit on every control plane, and noctl configcould then put it right.
Directory sign-in is now turned off on that control plane, with an error in its log. Local accounts still sign in. - A second
llmgw serveron a machine stops before it competes to lead.llmgw server check(no such command)
ran a whole server. Its ports were taken, yet it went on to hold the leader's lease for three hours while the real
control planes stood by.servernow refuses words it does not know and takes its ports before anything else. - The single sign-on and Active Directory guides said
server.yaml. On a cluster with a store,authis in the
cluster's settings and the file's block is not read. They now sayllmgw ctl config edit, and what certificate a
domain controller needs.
Downloads
-
v0.14.8
Stable
gitea-actions
released this
2026-10-18 11:18:14 +02:00 | 149 commits to main since this releaseSign-in with an on-premises Active Directory, and Entra ID made safe for
people in many groups. Also several keys for one outside provider, and a
measure of what the gateway adds to a request.Upgrading.
- Migration 0048 adds the
user_directorytable, applied when the control
plane starts. - Nothing changes until
auth.ldapis set; docs/install.md, "Active
Directory", has an example. It has been tested against a fake directory,
not yet against a real domain controller. - A judge installed by an earlier
server initis refitted by the leader
by itself (below); the steps given for v0.14.7 are no longer needed.
Added
-
Sign-in with Active Directory (
auth.ldap). People sign in on the gateway's form with their AD username or mail
and password, checked against the domain controllers over LDAPS or StartTLS. Groups, nested ones included, give
staff roles and places in the tree, as with OIDC. Accounts are tied to theobjectGUIDand made at first sign-in
whencreate_accountsis on. Every 15 minutes the leader reads the directory. People disabled or deleted there are
suspended and reinstated when they come back; roles and memberships follow their groups. A sync that would suspend
more than half the accounts is refused. See docs/install.md, "Active Directory". -
Several keys for one outside provider. An external model takes
api_key_files, each with an optional weight, in
place ofapi_key_file. Each key is an endpoint of its own (external:MODEL#1,#2), so a key answering 429 is
passed over while its breaker is open and the request goes to another. A key answering 401 is passed over too, and
external_key_refusedis sent, naming the endpoint, never the key. Usage records which key answered by that name. -
llmgw ctl bench overhead: what the gateway adds to a request, measured against an engine that answers at
once. Gateway and engine run in one process on loopback, so no cluster is needed. The load is sent straight to the
engine and through the gateway: bare, with authentication and accounting, and with the guardrails' pattern checks.
It reports p50 and p99 latency, what the gateway added, and a rough CPU per thousand requests. On an Apple M5 the
gateway added 0.05–0.42 ms at p50 and at most 0.49 ms at p99 (README).
Fixed
- Entra ID's groups past 200 no longer take a role away. For someone in more than 200 groups Entra sends none in
the token, only a pointer to Graph. That was read as no groups, so an administrator lost their role at the next
sign-in. Their role and memberships are now left as they were, and an error is logged saying what to change in the app
registration. docs/install.md has an Entra ID section. - An account made by SCIM or OIDC could not be reinstated. Setting it active was refused as "never approved".
- An upgrade refits the judge an earlier
server initinstalled. Upgrading to v0.14.7 left such a judge as it was:
no cores or one, 2 or 4 GB, the prompt cache on. On fewer than eight cores every request was refused until it was
fixed by hand. Now the leader finds a judge whose entry is exactly what an earlier init wrote and gives it 2 GB, no
prompt cache and cores by today's rule (up to four for one judge, two each for more). It then restarts the judge's
replicas onto that. The number of replicas is left as it is. A judge whose entry was changed by hand is left alone,
and the log says so once. - A restart is no longer refused for room another restart is giving back. With every core of a node reserved, two
models restarted together, one to need more and one less (as afterctl model resourceson a judge and a starter),
had the first refused: "fits on no node even with this replica stopped", though the second was about to give back
its cores. A restart now waits for other restarts under way, up to the drain grace, before it is refused. A drained
replica also counts a replica being restarted that still answers as serving, so replicas restarted one after
another no longer wait out the five-minute grace each.
Downloads
- Migration 0048 adds the