A big model's download no longer runs out its startup timeout
CI / test (push) Successful in 5m18s

The startup timeout runs from when a node's download last moved, so a
model of hundreds of gigabytes on a slow mirror is not failed while it
arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO)
on every node, and a layout given by hand keeps expert parallelism for a
model of experts. docs/large-models.md: how a model is split, what each
way needs of a card, and a checklist for bringing one up on a fabric.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-20 09:45:31 +02:00
1 parent 4fe9b46451
commit 8ffde2b80c
12 files changed
+259 -4

No files matched your search

+6
View File
@@ -47,6 +47,12 @@ What is left, most needed first. The detail is in the item each names.
licences in video and audio (H5); nginx in front of several control
planes (J7).
14. **K** — large models on several cards and nodes (2026-10-20): code
done (expert parallelism, any number of nodes, memory per card, the KV
head rule, the startup timeout from the last download), and
docs/large-models.md. Left for the operator: trying it on a fabric with
the checklist there. Then fixes from what it shows.
Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a
v0.14.5 dev build; no hurry); where Alertmanager sends (its receiver is
"none"). Settled 2026-10-16: Apertus 8B is on demand (min 0); the HA test