Ilker ManapandClaude Opus 5.5
8ffde2b80c
The startup timeout runs from when a node's download last moved, so a model of hundreds of gigabytes on a slow mirror is not failed while it arrives. ctl model serve --env sets engine variables (NCCL_DEBUG=INFO) on every node, and a layout given by hand keeps expert parallelism for a model of experts. docs/large-models.md: how a model is split, what each way needs of a card, and a checklist for bringing one up on a fabric. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
146 lines
6.5 KiB
Markdown
146 lines
6.5 KiB
Markdown
# Large models: several cards, several nodes
|
||
|
||
A model too big for one card runs on several. llmgw chooses how from the
|
||
model's own figures and the cluster as it is; this page is the rules it
|
||
follows, the commands, and what to look at when a first one is brought up
|
||
on new hardware.
|
||
|
||
## How a model is split
|
||
|
||
In order, the first that fits:
|
||
|
||
1. **One card.** Nothing to split.
|
||
2. **Several cards of one node, tensor parallel.** Every layer is cut
|
||
across the cards, so they talk for every token: they must be on one
|
||
NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica
|
||
the cards with the most NVLink between them
|
||
([install.md](install.md#nvlink-if-your-gpus-have-it)).
|
||
3. **Several nodes, pipeline parallel between them.** Each node holds a
|
||
run of layers on all its cards (tensor parallel inside it), and only a
|
||
layer's output crosses to the next node. The nodes must share one
|
||
InfiniBand or RoCE fabric: a replica never spans two, and never a node
|
||
with none. Any number of nodes, not only two or four.
|
||
|
||
The plan takes the fewest cards, then the fewest nodes, then the better
|
||
precision up to eight bits.
|
||
|
||
**Tensor width and the KV heads.** The cards split the model's KV heads
|
||
among them, or past their number each card holds a copy of one. So the
|
||
width divides the KV heads or is a multiple of them: a model with 8 KV
|
||
heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8.
|
||
|
||
**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3,
|
||
Llama 4, Mixtral) on more than one card is served with vLLM's
|
||
`--enable-expert-parallel`: each card holds whole experts rather than a
|
||
slice of every one, which is less traffic per token. It is read from
|
||
`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`).
|
||
|
||
**When it does not fit.** A smaller precision first: an FP8 or AWQ
|
||
repository of the same model, or a GGUF quantisation on llama.cpp if one
|
||
node's cards hold it. Then more nodes. `llmgw ctl model plan REPO
|
||
--precision bf16,fp8` shows each precision's fit from the model's config.
|
||
|
||
## What a way needs
|
||
|
||
Each way the plan offers says what a replica needs of each card, part by
|
||
part, and of all its cards together:
|
||
|
||
```
|
||
$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768
|
||
FILE SIZE BEST WAY A CARD OF ALL CARDS NOW
|
||
original 131 GB 4 × h100-80g 37.4 GB 72 GB 150 GB free
|
||
|
||
Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards.
|
||
The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another).
|
||
```
|
||
|
||
- **weights:** the files, divided over the cards.
|
||
- **context:** the KV cache for the context planned, divided likewise.
|
||
- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM.
|
||
- **usable:** 90% of a card; the rest is the driver's and fragmentation's.
|
||
|
||
The console's **Serve a model** shows the same for every way a file runs.
|
||
A context longer than planned is checked against it before anything is
|
||
fetched.
|
||
|
||
## Serving one
|
||
|
||
Without flags the plan's best way is taken:
|
||
|
||
```
|
||
llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768
|
||
```
|
||
|
||
Or a way given by hand, for instance two nodes of eight:
|
||
|
||
```
|
||
llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \
|
||
--gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO
|
||
```
|
||
|
||
`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline
|
||
is the nodes and the tensor width the cards of each.
|
||
|
||
`--env KEY=VALUE` goes into the engine's environment on every node:
|
||
`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it.
|
||
Only the variables engines are tuned with are taken. llmgw already sets
|
||
these:
|
||
- `NCCL_IB_HCA` to the fabric's adapters;
|
||
- `NCCL_IB_GID_INDEX` on RoCE;
|
||
- `NCCL_SOCKET_IFNAME` to the management network;
|
||
- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`.
|
||
|
||
The job's first step says what it needs of each card. The weights then
|
||
come to the mirror once and from it to each node. The startup timeout (20
|
||
minutes unless the profile's `startup_timeout` says otherwise) runs from
|
||
when a node's download last moved. So a model of hundreds of gigabytes is
|
||
not failed while it is still arriving, and one whose download stops is
|
||
failed that long after.
|
||
|
||
## Watching it come up
|
||
|
||
| What | Where |
|
||
|---|---|
|
||
| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` |
|
||
| Why it is not placed | `llmgw ctl explain-placement MODEL` |
|
||
| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` |
|
||
| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` |
|
||
| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin |
|
||
| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) |
|
||
|
||
With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes:
|
||
`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the
|
||
model will run, slowly.
|
||
|
||
## Bringing it up on new hardware
|
||
|
||
A first run on a fabric, in order. Each step is worth its output if
|
||
something goes wrong.
|
||
|
||
1. **The nodes.** Add each with its fabric:
|
||
`llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then
|
||
`llmgw ctl nodes` shows every node ready, on `ib-a`.
|
||
2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its
|
||
rated speed (a 400 Gb/s rail near 45 GB/s).
|
||
3. **One node, several cards.** A dense 70B at bf16 on four or eight
|
||
cards of one node, by the plan:
|
||
`llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`.
|
||
Ready, then a question answered.
|
||
4. **Two nodes.** The same model forced over two:
|
||
`--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under
|
||
another `--name`. Ready, answering, and `NET/IB` in both shards' logs.
|
||
5. **Experts.** A mixture-of-experts model on one node's eight cards,
|
||
for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile
|
||
carries `--enable-expert-parallel` (`llmgw ctl models`, or its
|
||
definition from the admin API).
|
||
6. **A node lost.** With the two-node replica serving, stop one node's
|
||
agent (`systemctl stop llmgw-agent`): the replica fails and is placed
|
||
again once the node is back or another has room. Nothing is left
|
||
holding cards.
|
||
|
||
For each, worth keeping:
|
||
- the time from serve to ready;
|
||
- tokens a second from `llmgw ctl bench`;
|
||
- the plan's per-card figure against what `nvidia-smi` shows once loaded;
|
||
- anything it said that was wrong.
|