The startup timeout runs from when a node's download last moved, so a model of hundreds of gigabytes on a slow mirror is not failed while it arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO) on every node, and a layout given by hand keeps expert parallelism for a model of experts. docs/large-models.md: how a model is split, what each way needs of a card, and a checklist for bringing one up on a fabric. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
4fe9b46451
commit
8ffde2b80c
12 files changed
+259
-4
No files matched your search
@@ -10639,6 +10639,10 @@
|
||||
"format": "int64",
|
||||
"description": "KVBytesPerToken is one token of context, the whole model: what an option needs for another context is worked out from it. 0 is unknown."
|
||||
},
|
||||
"experts": {
|
||||
"type": "integer",
|
||||
"description": "Experts is a mixture-of-experts model's routed experts; 0 for dense."
|
||||
},
|
||||
"gated": {
|
||||
"type": "boolean"
|
||||
},
|
||||
@@ -10789,6 +10793,13 @@
|
||||
"type": "integer",
|
||||
"description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on."
|
||||
},
|
||||
"env": {
|
||||
"type": "object",
|
||||
"additionalProperties": {
|
||||
"type": "string"
|
||||
},
|
||||
"description": "Env is set in the engine's environment, on every node of a replica: NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them. Only the variables engines are tuned with; the catalog refuses others."
|
||||
},
|
||||
"memory_gb": {
|
||||
"type": "integer",
|
||||
"description": "MemoryGB is what a replica on CPUs reserves, in place of the figure worked out from its weights and context: a judge's, whose own sizing is known."
|
||||
|
||||
Reference in new issue
Block a user