Ilker ManapandClaude Opus 5.5
f4b34dbf11
CI / test (push) Successful in 9m37s
FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several times a card's size from one card, experts in RAM, from Hugging Face safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13 toolkit it compiles kernels with. Older cards are passed over: on cuda01's Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and in fp16 hit an illegal memory access on its first request. ft serve gets the model, address, card and served name; the key goes in FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are FREETOKEN_ variables naming paths or the agent's own settings. ctl model context sets --max-seq-len-override. Readiness now also waits for an engine whose /health answers 200 while its body says "loading", as FreeToken's does from the moment its HTTP server is up. FreeToken is followed release by release: a weekly workflow fails when PyPI has a version newer than DefaultEngines installs (scripts/engine_releases.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
54 lines
1.7 KiB
Go
54 lines
1.7 KiB
Go
package provision
|
|
|
|
import (
|
|
"errors"
|
|
"slices"
|
|
"strings"
|
|
"testing"
|
|
|
|
"gitea.mynodes.xyz/ilker/llmgw/internal/agent/discovery"
|
|
)
|
|
|
|
func TestFreeTokenIsInstalledOnlyWhereItsKernelsRun(t *testing.T) {
|
|
old := freetokenToolkit
|
|
toolkit := "/usr/local/cuda-13.4"
|
|
freetokenToolkit = func() string { return toolkit }
|
|
t.Cleanup(func() { freetokenToolkit = old })
|
|
|
|
ampere := discovery.Host{Arch: "amd64", NVIDIADriver: "580.95.05", MinComputeCapability: 8.6}
|
|
got, err := freetokenCandidates("0.2.1", "", ampere)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
b := got[0]
|
|
if b.Binary != "ft" || !slices.Equal(b.Requirements, []string{"freetoken[accel]==0.2.1"}) || !b.GPUCheck || b.Python == "" {
|
|
t.Errorf("build %+v", b)
|
|
}
|
|
|
|
// Not meant for these, and not reported as a fault: no GPU, a Turing
|
|
// card (cuda01, where 0.2.1 had no kernel image for it), an Arm node.
|
|
for name, h := range map[string]discovery.Host{
|
|
"no gpu": {Arch: "amd64"},
|
|
"turing": {Arch: "amd64", NVIDIADriver: "615.71.09", MinComputeCapability: 7.5},
|
|
"aarch64": {Arch: "arm64", NVIDIADriver: "580.95.05", MinComputeCapability: 9.0},
|
|
} {
|
|
_, err := freetokenCandidates("0.2.1", "", h)
|
|
var nh NotHere
|
|
if !errors.As(err, &nh) {
|
|
t.Errorf("%s: %v, want NotHere", name, err)
|
|
}
|
|
}
|
|
|
|
// Meant for these, and said why not.
|
|
if _, err := freetokenCandidates("0.2.1", "", discovery.Host{Arch: "amd64", NVIDIADriver: "570.1", MinComputeCapability: 9.0}); err == nil ||
|
|
!strings.Contains(err.Error(), "driver 580") {
|
|
t.Errorf("old driver: %v", err)
|
|
}
|
|
toolkit = ""
|
|
_, err = freetokenCandidates("0.2.1", "", ampere)
|
|
var nh NotHere
|
|
if err == nil || errors.As(err, &nh) || !strings.Contains(err.Error(), "CUDA 13 toolkit") {
|
|
t.Errorf("no toolkit: %v", err)
|
|
}
|
|
}
|