Commit Graph
11 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 f4b34dbf11 FreeToken, an engine for models larger than the card
CI / test (push) Successful in 9m37s
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 22:10:05 +01:00
Ilker ManapandClaude Opus 5.5 c0d5194ea0 Strata runs as an engine: a 125B MoE on one card
Strata (github.com/Niko1221/Strata) keeps a mixture-of-experts model's
experts in RAM and the ones used most in VRAM; on cuda01's RTX 6000 it
serves Qwen3.8-Flash-Next 125B at 70-87 tokens a second to one client.

- engines/strata.go: the adapter. The agent starts llmgw strata-serve,
  which runs Strata's setup the way its desktop app does
  (STRATA_CONTROL_INSTALL: installs nothing, writes only under the
  replica's and the node's directories) on the fetched GGUF files, points
  the configuration it writes at the installed engine, and execs Strata's
  server. The key goes in STRATA_API_KEY, never on a command line.
- A profile's args go to setup: --family, --size (setup's --model, which
  is the source everywhere else), --context, --parallel, ... Flags that
  move files or pick the engine or GPU are refused.
- provision: the source pinned by digest in a Python environment, and the
  release's prebuilt engine unpacked beside it, checked against the digest
  GitHub publishes: CUDA 13 (7.5+, driver 580) or CUDA 12.9 (6.0-8.9,
  driver 575). Python builds can now carry release assets.
- Metrics are vLLM's names under /metrics/prometheus.
- The catalog wants one card and a family, and a directory, not one .gguf.

Opt-in: not in DefaultEngines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 15:09:13 +01:00
Ilker ManapandClaude Opus 5.5 1948a787ae Fixes from the second review: grants, flags, updates, setup
CI / test (push) Successful in 9m15s
- Revoking a server's tools takes them from every level beneath too, and a
  level left naming no tool stays closed ([]) while any level above or
  beneath names tools: an organization's revoke left its teams the copies
  their first grants took, and going back to no list opened the level to
  what was above it. A closed level is shown on the page and in the
  console with an action to open it (store.OpenTools). A key's owner's
  list is among the levels above a key.
- The flag check matches short flags as written again (-m, -hf, -mu were
  let through), takes a long flag's beginning for it only for the engines
  that parse with argparse (vLLM, SGLang, ComfyUI: --li is --listen), so
  llama.cpp's own --chat-template is not refused, and refuses llama.cpp's
  short spellings (-hfr, -hff, -mm, -jf, …).
- A stored model with an adapter by Hugging Face id can be changed again:
  what it already had is not asked about anew.
- The catalog reads a Hugging Face id as the agent does (engines.IsHubID).
- The page's form cannot move llmgw:admin to another address.
- server init skips the memory check only where the gateway already runs,
  not wherever a half-finished first run left server.yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 12:31:07 +01:00
Ilker ManapandClaude Opus 5.5 5eac62d556 Guardrail services merged the stricter way; the request log keeps no restored value
CI / test (push) Failing after 7m44s
From review.
- A service two policies name is asked the stricter way of the two, both
  directions and block over flag, rather than as the first one says.
- A service answering null where it says unsafe gave no answer, not safe.
- With restore: true, the kept answer had the request's redacted values in
  it beside the redacted request; they are taken out again before it is
  kept.
- guardrails.md says what fail: open lets through, and that of n answers a
  service reads the first.
- The similar cache's embedding failures are logged once a minute, not per
  request; the page's test reads the gateway's local MCP handler under its
  lock; org/name:Q4_K_M is not taken for a Hugging Face id to read as
  safetensors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 11:03:57 +01:00
Ilker ManapandClaude Opus 5.5 fd94265e08 Pickles refused however they come; a string temperature passes again
CI / test (push) Failing after 7m17s
From review. A mirror set already on the node is looked at again, not
trusted for having been fetched once; Pickled cuts a query or a fragment
from any source, hf: too; .pt2 is a pickle. The flag check reads a flag as
the engine does: underscores as dashes and any beginning of a refused flag
as that flag, since argparse takes --trust-remote for --trust-remote-code;
SGLang refuses --load-format, which would undo the agent's safetensors.

TripoSR's preset says allow_pickle: without it the preset could not be
added and the presets pass failed on it each time. A shared ComfyUI
environment's hidden entry takes its first member's allow_pickle, as it
takes its source.

The GenAI trace attributes made temperature and top_p typed fields of the
request's parse, so "0.7" as a string was a 400; they are read loosely,
as they are read only for the trace.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-28 22:19:06 +01:00
Ilker ManapandClaude Opus 5.5 4466d2d15a Weights that run code are refused
CI / test (push) Successful in 8m57s
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.

BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 23:17:22 +01:00
Ilker ManapandClaude Opus 5.5 fde1501e8a ComfyUI as an engine of the cluster's own
A model whose profiles run engine comfyui is placed on a node's card like
any other, and makes pictures there. ComfyUI asks for no key, so the agent
starts it behind llmgw comfyui-gate: ComfyUI on the loopback, the gate on
the replica's port demanding the replica's key and letting through only
what the router asks — queueing, waiting, fetching, cancelling. The gate
answers the agent's /health and /v1/models, shows ComfyUI the fetched
weights by their own names, and gives each replica its own base directory.
ComfyUI runs as the gate's child, so its card memory is the replica's.

A node offers the engine when agent.yaml names its ComfyUI
(engines.comfyui.dir). The model's comfyui: block names the workflow and,
for a source that is a set of files, the checkpoint.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-10 00:39:10 +02:00
Ilker ManapandClaude Opus 5.5 7940e8593c TensorRT-LLM as an engine, and models from Ollama into the mirror
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.

mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 00:02:01 +02:00
Ilker ManapandClaude Opus 5.5 f1b49d0251 Bound idle time, header size and request reading on every listener, and connections on the client port
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 10:21:15 +02:00
Ilker ManapandClaude Opus 5.5 c19abe95e1 Refuse profile env and args that would run code or touch files on the nodes
env is an allowlist of tuning families less path, plugin, library, address
and credential names (NCCL_*_PLUGIN, CUDA_INJECTION64_PATH and the like
got through the old denylist); llama.cpp's LLAMA_ARG_* are checked as
flags. args refuse --trust-remote-code, --log-file, --slot-save-path,
--tool-parser-plugin, --middleware, --allowed-local-media-path and others
that read, write or load, and what the agent sets itself. Checked in the
catalog and again on the node before an engine starts.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 09:54:45 +02:00
Ilker ManapandClaude Opus 5.5 d167686cf1 Stop a browser page, or a model definition, from running code on the nodes
The admin listener has no login. Reaching it is meant to mean holding the
control plane, so it binds to loopback and an operator reaches it over an
ssh tunnel — and so does every page open in that operator's browser. The
API decoded JSON whatever the Content-Type said, so a page could POST a
text/plain model definition, no preflight needed. A model's profile env
reached an engine the agent runs as root, unfiltered: LD_PRELOAD in a
definition was code execution on every node. A page that rebound its own
name to 127.0.0.1 could also read pprof's heap, which holds API keys.

The admin listener now answers only when Host is an address or localhost,
refuses requests a browser marks as cross-site (Origin, Sec-Fetch-Site),
and takes changes only as application/json, which no page can send across
origins without a preflight nobody grants. ctl sends that on every change,
bodiless DELETEs included; curl and ctl are otherwise unaffected.

Profile env that changes what runs rather than how — the dynamic loader's
LD_*, PATH, Python's PYTHON* start-up variables, BASH_ENV and their like —
is refused when a model is defined, and dropped again by the engine adapter
before the process starts, whatever sent the profile. None of the live
catalog's models sets any.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 00:29:52 +02:00