A big model's download no longer runs out its startup timeout

The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-20 09:45:31 +02:00
1 parent 4fe9b46451
commit 8ffde2b80c
12 files changed
+259 -4

No files matched your search

+2
View File
@@ -3219,6 +3219,8 @@ requests. On a slow model, set `timeouts.generation` higher.
- `docs/choosing-models.md` — which models to run and how many, if the answer
to "we have GPUs, now what" is not already obvious
- `docs/large-models.md` — a model on several cards or several nodes: how it
is split, what each way needs of a card, and bringing one up on a fabric
- `docs/data.md` — what is stored and what leaves the machine, for the review
that happens before this is allowed near anyone's documents
- `docs/reference.md` — every configuration key, generated from the source, so