FreeToken, an engine for models larger than the card

FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CHANGELOG.md

@@ -30,6 +30,19 @@ APIs change between minors without a deprecation period.
 
 ### Added
 
+- **FreeToken, an engine for models larger than the card.** Mixture-of-experts
+  models several times a card's size, from one card, with the experts in RAM
+  and streamed over PCIe — read from Hugging Face safetensors checkpoints as
+  published (DeepSeek V4 Flash, GLM-5.x, Qwen3.6/3.8, gpt-oss, Gemma 4,
+  MiniMax), with batching. Installed like vLLM on every node it runs on: a
+  card of compute capability 8.0 or newer, driver 580+, x86-64, and the CUDA 13
+  toolkit it compiles kernels with; older cards are passed over (tried on a
+  Turing card, where 0.2.1 has no kernels). Profiles are one card
+  (`engine: freetoken`); the key goes in its environment, and flags that read
+  files or fetch are refused. Readiness now also waits for an engine whose
+  `/health` answers 200 while saying `loading` in its body. FreeToken is
+  followed release by release: a weekly workflow fails when PyPI has a version
+  newer than `DefaultEngines` installs.
 - **How much of each GPU works, beside whether anything does.** The busy
   figure is nvidia-smi's `utilization.gpu` — the share of time any kernel
   ran, which a kernel on one SM fills — and it is now said to be that: on