4 Commits

Include renames

Author	SHA1	Message	Date
Ilker ManapandClaude Opus 5.5	31f440ab6e	A served model's longest answer changed in place, and Strata's context ... max_output on POST /api/v1/models/{name}/context, --max-output on ctl model context, and a field in the console's Context length… form. Alone it changes only the catalog's limit, which the router holds max_tokens to, and starts no replica again; with a context it is set with it, never past it. Strata's context is now changed like the others', through its setup's --context, which writes the engine's start script at each start. Neither needs a model removed and added again any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-01 00:15:38 +01:00
Ilker ManapandClaude Opus 5.5	f4b34dbf11	FreeToken, an engine for models larger than the card ... FreeToken (FlashML-org/FreeToken) serves mixture-of-experts models several times a card's size from one card, experts in RAM, from Hugging Face safetensors checkpoints. Installed like vLLM, from PyPI by uv, on every node it runs on: compute capability 8.0+, driver 580+, x86-64, and the CUDA 13 toolkit it compiles kernels with. Older cards are passed over: on cuda01's Quadro RTX 6000 (7.5) 0.2.1 had no kernel image for the first prefill, and in fp16 hit an illegal memory access on its first request. ft serve gets the model, address, card and served name; the key goes in FREETOKEN_API_KEY. Flags that read files or fetch are refused, as are FREETOKEN_ variables naming paths or the agent's own settings. ctl model context sets --max-seq-len-override. Readiness now also waits for an engine whose /health answers 200 while its body says "loading", as FreeToken's does from the moment its HTTP server is up. FreeToken is followed release by release: a weekly workflow fails when PyPI has a version newer than DefaultEngines installs (scripts/engine_releases.py). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-31 22:10:05 +01:00
Ilker ManapandClaude Opus 5.5	e1cff0d4e3	A served model can be moved to more cards, or fewer ... llmgw ctl model gpus MODEL N [--profile P] [--in-place], POST /api/v1/models/{name}/gpus and the console's GPUs…: how many cards of one node each replica uses. Tensor parallel for vLLM, SGLang and TensorRT-LLM, held to the model's KV heads (modelplan.SplitsHeads, now one rule for the planner and this); the layers split for llama.cpp, its old --tensor-split dropped. Refused for a count no node has, for cards that would not hold the weights and the context, and for engines on one card. The replicas start again beside the old ones; with no room beside them the command says so, and --in-place stops each first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-29 16:40:49 +01:00
Ilker ManapandClaude Opus 5.5	986981ac81	A served model's context can be changed, engine and memory with it ... ctl model context MODEL TOKENS [--slots N], POST /api/v1/models/{name}/context and the console's Context length…: the catalog, the engine's arguments and the KV memory change together, refused past the trained context or what the cards hold, then the replicas restart one by one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>	2026-10-29 15:50:04 +01:00