A big model's download no longer runs out its startup timeout
The startup timeout runs from when a node's download last moved, so a model of hundreds of gigabytes on a slow mirror is not failed while it arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO) on every node, and a layout given by hand keeps expert parallelism for a model of experts. docs/large-models.md: how a model is split, what each way needs of a card, and a checklist for bringing one up on a fabric. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4fe9b46451
commit
8ffde2b80c
12 files changed
+259
-4
No files matched your search
@@ -69,6 +69,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command {
|
||||
f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)")
|
||||
f.IntVar(&req.TP, "tp", 0, "the older name of --gpus")
|
||||
_ = f.MarkHidden("tp")
|
||||
f.StringToStringVar(&req.Env, "env", nil,
|
||||
"KEY=VALUE in the engine's environment on every node, e.g. NCCL_DEBUG=INFO; repeatable")
|
||||
f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy")
|
||||
f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it")
|
||||
f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)")
|
||||
|
||||
Reference in new issue
Block a user