Plan a repository's files against the fleet, and serve the one that suits it the way it fits

The control plane reads the model's shape from one GGUF header or its
config.json and lists, for every file, the best way it runs here: one card,
several of a node's (by layer for llama.cpp, tensor-parallel for vLLM),
several nodes' over one fabric (vLLM, pipeline-parallel between them), or a
CPU node; what each card holds and whether it is free now. ctl model plan
prints it; model serve takes its file and way unless told, gives a GGUF on
GPUs a GPU profile, and with --fallbacks adds the next best ways. The
console shows it per file and asks for the way and the fallback.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 01:03:48 +02:00
1 parent 630d2788c0
commit 54db7aecb7
15 files changed
+1224 -158

No files matched your search

+14 -6
View File
@@ -188,13 +188,21 @@ to restart it. Another control plane keeps its own copy.
- **Serve a model** — Hugging Face's models: trending, most downloaded,
most liked or newest (**S** changes which), or **/** to search. **F**
switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and
original safetensors weights, which vLLM runs on GPUs. Enter on a GGUF
repository lists its files with their sizes; a vision model offers its
projector too, which is fetched beside it and loaded with it so the model
reads images. Enter on original weights offers them with their config and
tokenizer, a GPU class and how many GPUs a replica spans. What is chosen is
original safetensors weights, which vLLM runs on GPUs. Enter on a
repository plans it against the cluster as it is, before anything is
fetched: the model's shape is read from one file's header, or its
config.json, and every file is listed with the best way it runs here — on
one card, on several of a node's cards, across several nodes' cards over a
fabric (original weights only), or on a CPU node — what each card or node
would hold, and whether that hardware is free now. The file that suits the
cluster best is first, marked ★; one that fits nowhere says so. Choosing a
file offers every way it fits, best first, and what to do when that one is
busy: the next best way, written as a lower-priority profile so placement
falls back to it, or wait. A vision model offers its projector too, fetched
beside it and loaded with it so the model reads images. What is chosen is
fetched into the mirror, checked, added to the catalog and started, and a
replica is asked to say hello (`llmgw ctl model serve`).
replica is asked to say hello (`llmgw ctl model serve`; the plan alone is
`llmgw ctl model plan`).
`mirror.registry` points it at another registry, for a site without the
internet.
- **Upgrade the cluster** to a release: this machine, then every node in turn