FreeToken, an engine for models larger than the card

FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several
times a card's size from one card, experts in RAM, from Hugging Face
safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node
it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13
toolkit it compiles kernels with. Older cards are passed over: on cuda01's
Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and
in fp16 hit an illegal memory access on its first request.

ft serve gets the model, address, card and served name; the key goes in
FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are
FREETOKEN_ variables naming paths or the agent's own settings. ctl model
context sets --max-seq-len-override.

Readiness now also waits for an engine whose /health answers 200 while its
body says "loading", as FreeToken's does from the moment its HTTP server is up.

FreeToken is followed release by release: a weekly workflow fails when PyPI
has a version newer than DefaultEngines installs (scripts/engine_releases.py).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/reference.md

@@ -126,6 +126,7 @@ rather than repeated.
 | `name` | `string` | [Federation](#federation) |
 | `interval` | `time.Duration` | [Federation](#federation) |
 | `enabled` | `bool` | [FirewallSettings](#firewallsettings) |
+| `binary` | `string` | [FreeTokenConfig](#freetokenconfig) |
 | `min_compute_capability` | `float64` | [GPUNeed](#gpuneed) |
 | `min_memory_gb` | `int` | [GPUNeed](#gpuneed) |
 | `format` | `string` | [GuardModel](#guardmodel) |
@@ -482,6 +483,7 @@ rather than repeated.
 | `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
 | `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) |
 | `strata` | `StrataConfig` | [engines.Config](#enginesconfig) |
+| `freetoken` | `FreeTokenConfig` | [engines.Config](#enginesconfig) |
 | `enabled` | `*bool` | [guard.Config](#guardconfig) |
 | `default` | `[]string` | [guard.Config](#guardconfig) |
 | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -1317,6 +1319,16 @@ site whose own firewall does that job; the rules are kept, and true
 puts them back. Changed in the cluster's settings, it is in force on
 every node and control plane within seconds.
 
+## FreeTokenConfig
+
+### `binary`
+
+`string`
+
+Binary is its ft command, in the environment it was installed into.
+The agent installs one itself on a node whose GPUs it runs on; this
+is for one installed by hand.
+
 ## GPUNeed
 
 ### `min_compute_capability`
@@ -4042,6 +4054,16 @@ offered by a node whose Strata is named here, or that installed it.
 
 Takes a [StrataConfig](#strataconfig) block.
 
+### `freetoken`
+
+`FreeTokenConfig`
+
+FreeToken serves mixture-of-experts models larger than a card from
+RAM and one card of 8.0 or newer; installed by the agent where the
+cards allow it, or named here.
+
+Takes a [FreeTokenConfig](#freetokenconfig) block.
+
 ## guard.Config
 
 ### `enabled`