FreeToken, an engine for models larger than the card
CI / test (push) Successful in 9m37s

FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-31 22:10:05 +01:00
1 parent c866b3a8ee
commit f4b34dbf11
25 files changed
+808 -18

No files matched your search

+56
View File
@@ -703,6 +703,7 @@ Nothing to do: the agent installs them once the node joins, under
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
| FreeToken | nodes with a GPU of compute capability 8.0 or newer (RTX 30, A100 and later), driver 580+, x86-64 and the CUDA 13 toolkit | a Python environment built by uv from PyPI (`freetoken[accel]`); see [FreeToken](#freetoken-models-larger-than-the-card) |
| Strata | only when you ask for it (`engines: {strata: "0.2.7"}`) | its source, pinned by digest, in a Python environment, and its prebuilt engine for CUDA 13 (driver 580+) or CUDA 12.9 (Pascal to Ada, driver 575+); see [Strata](#strata-a-large-model-of-experts-on-one-card) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
@@ -785,6 +786,61 @@ in (48 GB and the engine's 4 for IQ2_XS) unless it says `min_memory_gb`. Its
first start prepares the model on the node, which takes minutes, and fetches a
draft layer of about 5 GB from the internet.
#### FreeToken: models larger than the card
[FreeToken](https://github.com/FlashML-org/FreeToken) serves mixture-of-experts
models several times a card's size from one card: the experts stay in the
node's RAM, the ones used most are cached in VRAM, and the rest stream over
PCIe. Unlike Strata it reads Hugging Face safetensors checkpoints as they are
published — DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8 (Flash-Next among them),
gpt-oss, Gemma 4, MiniMax M2.5/M3 — in FP8, NVFP4, MXFP4 or BF16, and batches
several requests (`--max-running-requests`, 4 by default).
Every node that can run it installs it, as with vLLM: an NVIDIA GPU of compute
capability 8.0 or newer, driver 580 or newer, x86-64. Older cards are passed
over without a word — on a Quadro RTX 6000 (7.5) version 0.2.1 had no kernel
image for the card's first prefill, and in fp16 its first request ended in an
illegal memory access. Its FP8 checkpoints need 8.9 (Ada, Hopper) and its NVFP4
ones 12.0 (RTX 50, RTX PRO 6000 Blackwell). It compiles some kernels the first
time a model loads, so the node also needs the **CUDA 13 toolkit** (`nvcc` in
`CUDA_HOME`, `/usr/local/cuda` or `/usr/local/cuda-13.*`); a node of new cards
without one says so on the cluster page instead of installing an engine that
fails on its first model.
A FreeToken profile runs on one card of one node (`llmgw ctl model add -f`):
```yaml
name: glm-5.2
source: /var/lib/llmgw-weights/GLM-5.2-NVFP4 # a safetensors checkpoint's directory
profiles:
- name: one-card
engine: freetoken
gpu_class: rtx-pro-6000-96g
min_memory_gb: 200 # the experts live in RAM: say how much
args: ["--max-running-requests", "8"]
```
A Hugging Face id as the source works too — FreeToken downloads the
repository itself — but it is not pinned to a commit or checked against
Hugging Face's checksums as vLLM's are, FreeToken having no `--revision`; a
directory the mirror fetched, or one put on the node, is.
The agent gives it the model, its address, its card (`--gpu`) and the key (in
`FREETOKEN_API_KEY`, not on the command line); `--mmproj`, media paths and
domains, `--hf-overrides` and `--moe-cache-split` are refused as they read
files or fetch. `ctl model context` sets `--max-seq-len-override`. Readiness
waits for its `/health` to stop saying `loading`, since it answers 200 from the
moment its HTTP server is up. It serves no Prometheus metrics, so the router
counts its requests itself.
llmgw follows FreeToken release by release: the version is
`DefaultEngines["freetoken"]`, and a weekly workflow
(`.gitea/workflows/engine-releases.yml`) fails when PyPI has a newer one, the
notice to check it and move to it. A version of your own is named under
`engines` like any other (`freetoken: "0.2.2"`), and `freetoken: ""` installs
none; one node is turned off with its build set to *none* (console, the node's
menu).
#### Choosing a node's vLLM build
Which build a GPU node installs is chosen on the cluster page, in the **vLLM
+22
View File
@@ -126,6 +126,7 @@ rather than repeated.
| `name` | `string` | [Federation](#federation) |
| `interval` | `time.Duration` | [Federation](#federation) |
| `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
| `binary` | `string` | [FreeTokenConfig](#freetokenconfig) |
| `min_compute_capability` | `float64` | [GPUNeed](#gpuneed) |
| `min_memory_gb` | `int` | [GPUNeed](#gpuneed) |
| `format` | `string` | [GuardModel](#guardmodel) |
@@ -482,6 +483,7 @@ rather than repeated.
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
| `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
| `freetoken` | `FreeTokenConfig` | [engines.Config](#enginesconfig) |
| `enabled` | `*bool` | [guard.Config](#guardconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -1317,6 +1319,16 @@ site whose own firewall does that job; the rules are kept, and true
puts them back. Changed in the cluster's settings, it is in force on
every node and control plane within seconds.
## FreeTokenConfig
### `binary`
`string`
Binary is its ft command, in the environment it was installed into.
The agent installs one itself on a node whose GPUs it runs on; this
is for one installed by hand.
## GPUNeed
### `min_compute_capability`
@@ -4042,6 +4054,16 @@ offered by a node whose Strata is named here, or that installed it.
Takes a [StrataConfig](#strataconfig) block.
### `freetoken`
`FreeTokenConfig`
FreeToken serves mixture-of-experts models larger than a card from
RAM and one card of 8.0 or newer; installed by the agent where the
cards allow it, or named here.
Takes a [FreeTokenConfig](#freetokenconfig) block.
## guard.Config
### `enabled`