Commit Graph
10 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 f41c96d30f Strata's DeepSeek not offered on cards its engine was not built for, and refused there by the agent
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 13:00:21 +01:00
Ilker ManapandClaude Opus 5.5 845de8eef4 The mirror page's Strata tab; Strata's ready-made packs served with Strata fetching them, strata-ple named
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-30 12:06:46 +01:00
Ilker Manap d6f8f433f4 CHANGELOG: one list 2026-10-30 10:39:30 +01:00
Ilker ManapandClaude Opus 5.5 f2c6f12cfb How fast a model would answer on the cards, as this cluster would lay it out
For each layout modelplan.Options offers here — one card, tensor parallel
across a node's cards, llama.cpp's layer split, a pipeline across nodes on
a fabric — modelplan.GPUTokensPerSecond gives one reply's speed: the bytes
a token reads over each card's bandwidth, by its model name (a table, SXM
and PCIe apart), plus two all-reduces a layer for tensor parallel (8 us on
NVLink, 25 us and up on PCIe), turns between cards for a layer split, and
a hop per node over InfiniBand or RoCE for a pipeline. The explorer and a
repository's files show the fastest, with the layout; mixed classes are
judged class by class, FP8 on 8.9 cards only.

Calibrated on cuda01's Quadro RTX 6000 (672 GB/s): qwen3-vl-8b Q8, 8.7 GB,
62.1 tok/s measured, 59 estimated; Qwen3-Coder-30B-A3B Q4_K_M, 18.6 GB,
144 measured, which says a model of experts reads 1.6 times its active
share (ActiveShare now does, on the CPU too). Tensor parallel and fabrics
are not measured here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 00:19:36 +01:00
Ilker ManapandClaude Opus 5.5 84444e71c8 How fast a model would answer on a CPU, before it is fetched
Each agent measures its node's memory read rate when it starts: every CPU
sums a share of a buffer past any cache, a tenth of free memory at most,
the best of four passes, in a fraction of a second; reported in Host
(mem_bandwidth_gbs) and NodeView. A token on a CPU reads every weight it
uses once, so one reply's speed is about that rate over those bytes:
modelplan.CPUTokensPerSecond, 0.68 of the measured rate plus 5 ms a token,
calibrated on the PoC against llama.cpp's predicted_per_second (node01
24.9 GB/s, qwen2.5-1.5b: 14.5 measured, 14.7 estimated; node05 41.9 GB/s,
qwen2.5-14b: 3.3 and 3.2). A model of experts named …-A3B reads its active
part (ActiveShare).

The explorer shows it as CPU tok/s, at Q4_K_M on the fastest node that
holds the model; a repository's page gives it for each GGUF by its size.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 00:09:47 +01:00
Ilker ManapandClaude Opus 5.5 2a45068461 The explorer lists what this cluster's hardware holds
Judged by the nodes as they are now, through modelplan.Options, the
planner that serving a model uses: GGUF at Q4_K_M from the parameter count,
safetensors at the size the registry's dtype breakdown gives (packed AWQ,
GPTQ and MXFP4 at four bits), FP8 only where a card has compute capability
8.9, MLX safetensors never. A picture model is judged against one card. A
cluster with cards asks the registry for safetensors too, a hundred at a
time; one without asks for GGUF only. The page says what it judged by: the
largest card, its capability, the most RAM on one node.

On a repository's page a file set no node holds is marked with what it
needs, against what the best node offers, and has no Fetch button.

Tried against the PoC's seven nodes and Hugging Face's trending hundred:
63 shown, the rest too big for one 24 GB Turing card or FP8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 18:30:16 +01:00
Ilker ManapandClaude Opus 5.5 e84631ce3a The explorer lists only what a node loads
A repository with neither GGUF nor safetensors (by the registry's summary
or its tags) is hidden under "runs here", which is ticked by default, and
counted with the architectures llama.cpp cannot load; unticked it is
listed. An open repository's pickle files are not offered, only counted,
and a repository of pickles only says to look for another version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 13:00:28 +01:00
Ilker ManapandClaude Opus 5.5 4466d2d15a Weights that run code are refused
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.

BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 23:17:22 +01:00
Ilker ManapandClaude Opus 5.5 6289b01ed6 The cluster's settings live in the store, and every change takes effect
Everything in server.yaml that is not one machine's own is now a versioned
document in the store (cluster_settings), the same on every control plane.
The first start against a store imports the file's cluster keys, with the
baseline guard policy when there is none; after that those keys in a file
are ignored, with a warning.

Each control plane polls the document's version and its own server.yaml's
hash. A change is validated and applied: guard live (Gate.Replace), anything
else by a graceful restart that systemd brings back, standbys first and the
leader 30s later. A version that does not load on a machine is not applied;
at start the newest one that loads is used, so a bad edit cannot keep every
control plane down.

llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API
under /api/v1/settings, and the console's Settings screen, which layers the
store's document over the machine's file, write it. Commands beside the
control plane that read cluster keys (init, node add, server add, backup,
upgrade) read the layered configuration; init --bundle writes offline to
the store once it holds the settings.

A standby that becomes the leader re-reads models, engine choices and
facility holds from the store.

Retires appendGuardIfMissing: the import does it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 15:25:23 +02:00
Ilker ManapandClaude Opus 5.5 11231d8c2b The portal in English and Turkish: a catalogue per language, a switcher at the foot of every page, landing.language, Turkish numbers and dates
Every template and handler message in internal/web goes through a JSON
catalogue under internal/web/locales/<code>/, with English as the fallback.
The language is the person's choice (a cookie set by /_lang/{code}), else
Accept-Language, else landing.language. Tests refuse a key some language
lacks, a translation with other arguments than English, and template text
left in English.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:12:53 +02:00