Plan a repository's files against the fleet, and serve the one that suits it the way it fits
The control plane reads the model's shape from one GGUF header or its config.json and lists, for every file, the best way it runs here: one card, several of a node's (by layer for llama.cpp, tensor-parallel for vLLM), several nodes' over one fabric (vLLM, pipeline-parallel between them), or a CPU node; what each card holds and whether it is free now. ctl model plan prints it; model serve takes its file and way unless told, gives a GGUF on GPUs a GPU profile, and with --fallbacks adds the next best ways. The console shows it per file and asks for the way and the fallback. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
630d2788c0
commit
54db7aecb7
15 files changed
+1224
-158
No files matched your search
+14
-6
@@ -188,13 +188,21 @@ to restart it. Another control plane keeps its own copy.
|
||||
- **Serve a model** — Hugging Face's models: trending, most downloaded,
|
||||
most liked or newest (**S** changes which), or **/** to search. **F**
|
||||
switches between GGUF files, which llama.cpp runs on the CPU or a GPU, and
|
||||
original safetensors weights, which vLLM runs on GPUs. Enter on a GGUF
|
||||
repository lists its files with their sizes; a vision model offers its
|
||||
projector too, which is fetched beside it and loaded with it so the model
|
||||
reads images. Enter on original weights offers them with their config and
|
||||
tokenizer, a GPU class and how many GPUs a replica spans. What is chosen is
|
||||
original safetensors weights, which vLLM runs on GPUs. Enter on a
|
||||
repository plans it against the cluster as it is, before anything is
|
||||
fetched: the model's shape is read from one file's header, or its
|
||||
config.json, and every file is listed with the best way it runs here — on
|
||||
one card, on several of a node's cards, across several nodes' cards over a
|
||||
fabric (original weights only), or on a CPU node — what each card or node
|
||||
would hold, and whether that hardware is free now. The file that suits the
|
||||
cluster best is first, marked ★; one that fits nowhere says so. Choosing a
|
||||
file offers every way it fits, best first, and what to do when that one is
|
||||
busy: the next best way, written as a lower-priority profile so placement
|
||||
falls back to it, or wait. A vision model offers its projector too, fetched
|
||||
beside it and loaded with it so the model reads images. What is chosen is
|
||||
fetched into the mirror, checked, added to the catalog and started, and a
|
||||
replica is asked to say hello (`llmgw ctl model serve`).
|
||||
replica is asked to say hello (`llmgw ctl model serve`; the plan alone is
|
||||
`llmgw ctl model plan`).
|
||||
`mirror.registry` points it at another registry, for a site without the
|
||||
internet.
|
||||
- **Upgrade the cluster** to a release: this machine, then every node in turn
|
||||
|
||||
Reference in new issue
Block a user