-
v0.15.1
Stable
gitea-actions
released this
2026-10-29 16:09:12 +01:00 | 85 commits to main since this releaseA 125B model on one card, a second mirror, and weights that run code
refused. Strata runs as an engine: a mixture-of-experts model with its
experts in RAM, served from one 24 GB card. A second model mirror keeps in
step with the first, so an HA install has no single machine left. Pickled
weights are refused unless a model says otherwise. A served model's context
can be changed in place. The explorer says how fast a model would answer
here before it is fetched, and a card in the wrong PCIe slot is said so.Upgrading.
- No migrations. Upgrade the control planes before the nodes, for the
pickle refusal below. - A model whose weights are in a pickle format (
.bin,.pt,.pth,
.ckpt,.pkl) stops being placed: its node refuses the source. Give it
its safetensors or GGUF, or add it again withallow_pickle: true
(llmgw ctl model add -f) if its files were checked. Look at the
models' sources before upgrading the nodes: one ending in
.ckptor.bin, or a mirror version holdingpytorch_model.bin. Upgrade the control planes first: a node on
this version refuses those weights until its control plane can say
allow_pickle. - Strata is opt-in: nothing installs it until
enginesor an
engine_overridesentry names a version (0.2.7).
Added
-
Strata as an engine: a 125B model of experts on one card. Strata
(github.com/Niko1221/Strata) keeps a model's experts in RAM and the ones
used most in VRAM. On an RTX 6000 24 GB with 251 GB of RAM it serves
Qwen3.8-Flash-Next 125B (IQ2_XS) at 70–87 tokens a second to one client;
more clients share about that, its slots buying fairness rather than
throughput on one card. It runs a few families only — Qwen3.8-Flash-Next
and its variants, Qwen3.6-35B-A3B, DeepSeek V4.1-Flash — each named in a
profile's args with--familyand--size
(--family qwen --size IQ2_XS --context 32768). The source is the
directory of the model's GGUF files, or the repository, which Strata then
downloads itself, pinned. The agent installs it on an NVIDIA node of
x86-64: its source pinned by digest, in a Python environment, and its
release's engine checked against the digest GitHub publishes — the CUDA 13
build for Turing and newer on driver 580, the CUDA 12.9 build for Pascal
to Ada, a V100 included, on driver 575. A replica prepares its model the
first time (minutes; its draft layer is a 5 GB download) and loads in
about a minute and a half after. One card of one node per replica, for
now; not in an offline bundle, its packages coming from PyPI. -
A served model's context, changed.
llmgw ctl model context MODEL TOKENS [--slots N],POST /api/v1/models/{name}/context, and Context
length… in the console's model menu. The catalog, the engine's arguments and
the memory change together, checked against what the model was trained for
and what its cards hold; its replicas are then started again one by one. The
models API showscontext_lengthandmax_output. -
A second model mirror, kept in step. In a three-machine install the setup screens put one on a database machine
(--second-mirror 1|2). An install with one mirror gets one withsudo llmgw mirror add-peer user@host.- In step: each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
removes what the other removed, even if it was down at the time (llmgw mirror serve --peer). - Control planes: they name both (
mirror.peers) and use whichever answers. - Nodes: while one mirror is down, a node is given a model's file on the other.
This takes away the last single point of an HA install: with the first machine off, models are still added and
fetched. - In step: each mirror reads the other every thirty seconds. It copies what it lacks, checksum checked, and
Fixed
-
An engine that computes in a process of its own group was not its
replica's. A GPU replica read as running on the CPU, and the memory it
held on its card as somebody else's, which placement then kept free of
other work. A process the replica started, at any remove, is now the
replica's. And what a replica computes on, once sent, was never sent
again: a wrong reading kept from before an agent's restart stayed wrong.
It is sent again when it changes, and read again when a GPU replica was
kept as the CPU. -
VS Code's sign-in lost its place through the sign-in page, so the
prompt to confirm came every time; a link to the same gateway by plain
http could take an https setting down to http; and a new build of the
extension was not installed over an old one (the command now says
--force). -
The install guide's recordings were not checked for secrets, only
the page; every frame is now. -
MCP server fields could not be emptied, an address could carry a
password, the cluster's own server could not be changed from the
console, and a grant to a key that does not exist was said done. -
A guardrail service two policies named was asked as the first said,
not the stricter; anullanswer read as safe; and withrestorethe
request log kept the answer with the redacted values back in it. -
What an engine downloads by a Hugging Face id was not held to the
pickle refusal — an adapter by its id, a TensorRT-LLM model's; a model
with either now needsallow_pickle. -
server initrun again on a machine already set up was refused its
memory, as on an 8 GB install fixing its certificate with--host, and
judges that would not be installed (no mirror) were counted. -
Removing an MCP server left its sealed credential in the store, and
replacing a credential under another header kept the old one sent. A
server's secrets now go when it no longer reads them. -
A cloned repository could point the VS Code extension at another
gateway.llmgw.gatewayUrlis a machine setting now, never taken from a
workspace. -
A node's memory bandwidth could be measured in its cache on a busy
node, and was 0 after a reconnect until the next heartbeat. The buffer is
at least eight times the processor's last cache. -
A project kept in the cluster could have its questions embedded outside
it. The similar-question cache now asks the embedding model as the
question's project would, external fallbacks refused, and skips a model
that is itself outside. Its comparing no longer holds the cache's lock,
which every project's lookups waited on. -
Restored values were left as placeholders in streamed tool calls, in
completions and in reasoning, and a value with a quote in a whole answer's
tool-call arguments broke their JSON. -
An outside guardrail service read what
redacthad taken out, and a
streamed answer was sent to it twice. The docs now say a service stops a
streamed answer only understream: buffer_all. -
A replica found slow once could stay unused. Its time to first token
is forgotten after five minutes without an answer, so it is tried again. -
Weights that run code got past the refusal in four ways. A mirror
version already on a node was used without being looked at again; a
?queryor#fragmenton anhf:source hid the file's name;.pt2was
not on the list; and a profile could write--load_format,
--trust_remote_codeor an abbreviation such as--trust-remote, which
vLLM and argparse read as the flag, or give SGLang--load-format. All
refused now. TripoSR's preset saysallow_pickle, without which it could
not be added, and a shared ComfyUI environment takes its first member's. -
A temperature written as a string was refused. Since the GenAI traces
read it,"temperature": "0.7"got a 400 before reaching the engine; it
is read loosely again, for the trace only. -
A reused answer could carry another caller's personal data. With
restore: trueand the answer cache on, two callers' requests that
redacted to the same text shared one answer — with the first caller's
address put back in it. Such requests are now never answered from the
cache nor kept in it. -
Granting a server's tools could take others away; revoking could close
a level to every tool. A level with no tool list sees what the levels
above grant; giving it its first list made it see only that list. And a
level left with an empty list was closed to every tool, which the screens
did not show. Now a first list starts from the nearest list above, taking
a server away leaves the rest or goes back to no list, the screens say
when a level above keeps granted tools out of reach and which levels are
closed, and the console reads and writes the level at once rather than
from what it loaded. -
Similar questions about different pictures got each other's answer.
A question with a picture, a sound or a file in it is reused only when it
is the same request.
Added
- A card in the wrong PCIe slot is said so. Each agent reads how every
card is linked — the fastest generation card and slot allow together, and
the lanes in use — against the card's own most, from the kernel (the
driver's figure is what card and slot allow together); a card on fewer lanes or
an older generation is a node warning, with what it carries and what it
could, on the dashboard, the console and inllmgw doctor. - And on a web page,
/admin/mcp. For administrators: the same list,
adding a server with its credential sealed, testing it and listing its
tools, granting them in the tree beside that level's other grants and
limits, revoking, changing, replacing the credential, turning it off,
removing; and the line that points an agent at this gateway's/mcp.
In English and Turkish; every change in the audit log, the secret not. - The company's MCP servers in the console. A screen of its own: each
server with its state, who has its tools, its rate, the names of the
credentials it is sent and its address; add one with a bearer token or
a header kept sealed, test it and list its tools, grant them to a team,
project, organization, person or key beside that level's other grants,
revoke, change, replace the credential, turn off, remove. - And on the cards, as this cluster would lay it out. The explorer's
GPU tok/s and a repository's files give one reply's speed for each
layout the planner offers here — one card, tensor parallel across a
node's cards, llama.cpp's layer split, a pipeline across nodes on a
fabric — the fastest of them, with the layout in the tooltip. From each
card's memory bandwidth by its model name, NVLink or PCIe between a
node's cards, and InfiniBand or RoCE between nodes; a cluster of mixed
classes is judged class by class. A model of experts reads 1.6 times its
active share, as Qwen3-Coder-30B-A3B did on cuda01's RTX 6000 (144
tok/s; qwen3-vl-8b Q8: 62). - How fast a model would answer on a CPU, before it is fetched. Each
agent measures how fast its node's memory reads when it starts (Host
getsmem_bandwidth_gbs, the node's viewmem_bandwidth_gbs); the
explorer's CPU tok/s column and a repository's GGUF files say about
how many tokens a second one reply gets on the fastest node that holds
the model — the bandwidth over the bytes a token reads, its active part
for a model of experts. Calibrated on the PoC against llama.cpp's own
timings: within 15% from 1.5B to 14B. - The explorer lists what this cluster can run. Judged by the nodes
as they are now — every card's memory and compute capability, every
node's RAM — through the same planner as serving a model, rather than a
size picked from a list. A model too big for any node, in FP8 without a
card of 8.9, in MLX, or in neither GGUF nor safetensors, is left out and
counted; the page says what it judged by. With cards, safetensors repositories are
listed beside GGUF. On a repository's page a file too big for the
cluster is shown with what it needs and no Fetch button. - Weights that run code are refused. Python pickles —
.bin,.pt,
.pth,.ckpt— run whatever they hold when loaded, which is how the
malicious models on Hugging Face work. The node refuses such a source,
or a mirror version with one in it, before fetching it; adding such a
model is refused; vLLM and SGLang downloading a model by its id read its
safetensors only; the mirror page marks the file.allow_pickle: true
on a model lets it through on purpose. The mirror's explorer no longer
lists a repository with nothing a node loads (no GGUF, no safetensors)
while "runs here" is ticked, and a repository's pickle files are not
offered, only counted. BACKLOG M lists what is next:
checksums against Hugging Face everywhere, its own scan, trusted
publishers, engines without the internet.
Downloads
- No migrations. Upgrade the control planes before the nodes, for the