diff --git a/BACKLOG.md b/BACKLOG.md index 162323f..dfd79b9 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -47,6 +47,12 @@ What is left, most needed first. The detail is in the item each names. licences in video and audio (H5); nginx in front of several control planes (J7). +14. **K** — large models on several cards and nodes (2026-10-20): code + done (expert parallelism, any number of nodes, memory per card, the KV + head rule, the startup timeout from the last download), and + docs/large-models.md. Left for the operator: trying it on a fabric with + the checklist there. Then fixes from what it shows. + Not code, the operator's to decide or do: the PoC on v0.14.7 (it runs a v0.14.5 dev build; no hurry); where Alertmanager sends (its receiver is "none"). Settled 2026-10-16: Apertus 8B is on demand (min 0); the HA test diff --git a/CHANGELOG.md b/CHANGELOG.md index 0a396bd..a5ae33f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,14 @@ APIs change between minors without a deprecation period. ### Added +- **Large models across cards and nodes.** For a model of experts on more than one card, the plan now adds vLLM's + `--enable-expert-parallel`. It reads the expert count from `config.json` under any of its names, and sizes a model + that also reads pictures by its `text_config`. Several nodes are offered in any number, not only powers of two. Each + way now says what it needs of a card, part by part (weights, context, engine), and of all its cards together. This + shows in `ctl model plan`, in the console's Serve a model, and in a serve job's first step. The serve form checks that + a longer context still fits. `ctl model serve --env KEY=VALUE` puts engine variables such as `NCCL_DEBUG=INFO` on + every node. docs/large-models.md has the rules, the commands, and a checklist for bringing a model up on a fabric. + - **Machines of the judges' own in a three-machine install.** The setup screen asks for up to two judge machines after the store's two; `server init` takes `--judge-node user@host` for each. Each is added as a node labelled `llmgw.role: judge`, and the judge's profile selects it, so the judges run there alone. Under load on a 4-core @@ -37,6 +45,13 @@ APIs change between minors without a deprecation period. ### Fixed +- **The plan refused tensor parallelism past a model's KV heads.** Qwen3-235B, with four KV heads, had no way to run + on eight cards, though vLLM gives each card a copy of a head. A width that is a multiple of the KV heads is now + offered, as is one that divides them. +- **A big model on a slow mirror was failed while it downloaded.** The startup timeout counted the download, so + hundreds of gigabytes at a few hundred MB/s ran past twenty minutes and the replica was started again from nothing. + The timeout now runs from when a node's download last moved. + - **The judge init installs is no longer replaced by the leader's own.** Just after the control plane started, the leader saw a policy naming a judge that init was still fetching, and served one itself. Its entry replaced init's, so a second judge ran with the leader's sizing, and with judge machines it ran off them. The leader now waits ten minutes diff --git a/docs/admin-api.openapi.json b/docs/admin-api.openapi.json index 7f5a334..d9ea6ef 100644 --- a/docs/admin-api.openapi.json +++ b/docs/admin-api.openapi.json @@ -10639,6 +10639,10 @@ "format": "int64", "description": "KVBytesPerToken is one token of context, the whole model: what an option needs for another context is worked out from it. 0 is unknown." }, + "experts": { + "type": "integer", + "description": "Experts is a mixture-of-experts model's routed experts; 0 for dense." + }, "gated": { "type": "boolean" }, @@ -10789,6 +10793,13 @@ "type": "integer", "description": "CPUCores is what a replica computes on; zero takes half the smallest node it fits on." }, + "env": { + "type": "object", + "additionalProperties": { + "type": "string" + }, + "description": "Env is set in the engine's environment, on every node of a replica: NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them. Only the variables engines are tuned with; the catalog refuses others." + }, "memory_gb": { "type": "integer", "description": "MemoryGB is what a replica on CPUs reserves, in place of the figure worked out from its weights and context: a judge's, whose own sizing is known." diff --git a/docs/install.md b/docs/install.md index fd111bd..0e11a52 100644 --- a/docs/install.md +++ b/docs/install.md @@ -3219,6 +3219,8 @@ requests. On a slow model, set `timeouts.generation` higher. - `docs/choosing-models.md` — which models to run and how many, if the answer to "we have GPUs, now what" is not already obvious +- `docs/large-models.md` — a model on several cards or several nodes: how it + is split, what each way needs of a card, and bringing one up on a fabric - `docs/data.md` — what is stored and what leaves the machine, for the review that happens before this is allowed near anyone's documents - `docs/reference.md` — every configuration key, generated from the source, so diff --git a/docs/large-models.md b/docs/large-models.md new file mode 100644 index 0000000..d8e7824 --- /dev/null +++ b/docs/large-models.md @@ -0,0 +1,145 @@ +# Large models: several cards, several nodes + +A model too big for one card runs on several. llmgw chooses how from the +model's own figures and the cluster as it is; this page is the rules it +follows, the commands, and what to look at when a first one is brought up +on new hardware. + +## How a model is split + +In order, the first that fits: + +1. **One card.** Nothing to split. +2. **Several cards of one node, tensor parallel.** Every layer is cut + across the cards, so they talk for every token: they must be on one + NVLink domain. The agent reads `nvidia-smi topo -m` and gives a replica + the cards with the most NVLink between them + ([install.md](install.md#nvlink-if-your-gpus-have-it)). +3. **Several nodes, pipeline parallel between them.** Each node holds a + run of layers on all its cards (tensor parallel inside it), and only a + layer's output crosses to the next node. The nodes must share one + InfiniBand or RoCE fabric: a replica never spans two, and never a node + with none. Any number of nodes, not only two or four. + +The plan takes the fewest cards, then the fewest nodes, then the better +precision up to eight bits. + +**Tensor width and the KV heads.** The cards split the model's KV heads +among them, or past their number each card holds a copy of one. So the +width divides the KV heads or is a multiple of them: a model with 8 KV +heads runs 2, 4, 8 or 16 wide, not 12. Qwen3-235B has 4 and runs on 8. + +**Mixture of experts.** A model of experts (Qwen3-235B-A22B, DeepSeek-V3, +Llama 4, Mixtral) on more than one card is served with vLLM's +`--enable-expert-parallel`: each card holds whole experts rather than a +slice of every one, which is less traffic per token. It is read from +`config.json` (`num_experts`, `n_routed_experts`, `num_local_experts`). + +**When it does not fit.** A smaller precision first: an FP8 or AWQ +repository of the same model, or a GGUF quantisation on llama.cpp if one +node's cards hold it. Then more nodes. `llmgw ctl model plan REPO +--precision bf16,fp8` shows each precision's fit from the model's config. + +## What a way needs + +Each way the plan offers says what a replica needs of each card, part by +part, and of all its cards together: + +``` +$ llmgw ctl model plan meta-llama/Llama-3.3-70B-Instruct --format safetensors --context 32768 +FILE SIZE BEST WAY A CARD OF ALL CARDS NOW +original 131 GB 4 × h100-80g 37.4 GB 72 GB 150 GB free + +Best: original on 4 × h100-80g: 37.4 GB a card (weights 32.9 + context 2.5 + engine 2.0) of 72 usable; 150 GB on 4 cards. +The context costs 0.31 GB per 1000 tokens, the whole model (--context to plan another). +``` + +- **weights:** the files, divided over the cards. +- **context:** the KV cache for the context planned, divided likewise. +- **engine:** the engine's own: 2 GB and a fiftieth of the weights for vLLM. +- **usable:** 90% of a card; the rest is the driver's and fragmentation's. + +The console's **Serve a model** shows the same for every way a file runs. +A context longer than planned is checked against it before anything is +fetched. + +## Serving one + +Without flags the plan's best way is taken: + +``` +llmgw ctl model serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 --format safetensors --context-length 32768 +``` + +Or a way given by hand, for instance two nodes of eight: + +``` +llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors \ + --gpu-class h100-80g --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO +``` + +`--gpus` is the cards on each node and `--nodes` the nodes. The pipeline +is the nodes and the tensor width the cards of each. + +`--env KEY=VALUE` goes into the engine's environment on every node: +`NCCL_DEBUG=INFO` while bringing a cluster up, `NCCL_IB_*` to steer it. +Only the variables engines are tuned with are taken. llmgw already sets +these: +- `NCCL_IB_HCA` to the fabric's adapters; +- `NCCL_IB_GID_INDEX` on RoCE; +- `NCCL_SOCKET_IFNAME` to the management network; +- for an NVLink domain spanning nodes, `NCCL_MNNVL_ENABLE`. + +The job's first step says what it needs of each card. The weights then +come to the mirror once and from it to each node. The startup timeout (20 +minutes unless the profile's `startup_timeout` says otherwise) runs from +when a node's download last moved. So a model of hundreds of gigabytes is +not failed while it is still arriving, and one whose download stops is +failed that long after. + +## Watching it come up + +| What | Where | +|---|---| +| The replica and its shards, one per node, each with its download | `llmgw ctl replicas` | +| Why it is not placed | `llmgw ctl explain-placement MODEL` | +| A node's engine output (NCCL's lines with `NCCL_DEBUG=INFO`) | `llmgw ctl logs NODE --source SHARD` | +| The bandwidth between two nodes, rail by rail | `llmgw ctl fabric-test NODE NODE` | +| Each node's GPUs, NVLink and adapters | `llmgw doctor` on the node; its page under /admin | +| Every step of a start, across nodes | the `replica start` trace ([install.md](install.md)) | + +With `NCCL_DEBUG=INFO`, NCCL says which transport it took between nodes: +`NET/IB` is the fabric. `NET/Socket` means it fell back to TCP, and the +model will run, slowly. + +## Bringing it up on new hardware + +A first run on a fabric, in order. Each step is worth its output if +something goes wrong. + +1. **The nodes.** Add each with its fabric: + `llmgw node add host --fabric ib-a:mlx5_0,mlx5_1`. Then + `llmgw ctl nodes` shows every node ready, on `ib-a`. +2. **The fabric.** `llmgw ctl fabric-test node1 node2`: each rail at its + rated speed (a 400 Gb/s rail near 45 GB/s). +3. **One node, several cards.** A dense 70B at bf16 on four or eight + cards of one node, by the plan: + `llmgw ctl model serve meta-llama/Llama-3.3-70B-Instruct --format safetensors`. + Ready, then a question answered. +4. **Two nodes.** The same model forced over two: + `--gpu-class CLASS --gpus 8 --nodes 2 --env NCCL_DEBUG=INFO`, under + another `--name`. Ready, answering, and `NET/IB` in both shards' logs. +5. **Experts.** A mixture-of-experts model on one node's eight cards, + for instance `Qwen/Qwen3-235B-A22B-Instruct-2507-FP8`: its profile + carries `--enable-expert-parallel` (`llmgw ctl models`, or its + definition from the admin API). +6. **A node lost.** With the two-node replica serving, stop one node's + agent (`systemctl stop llmgw-agent`): the replica fails and is placed + again once the node is back or another has room. Nothing is left + holding cards. + +For each, worth keeping: +- the time from serve to ready; +- tokens a second from `llmgw ctl bench`; +- the plan's per-card figure against what `nvidia-smi` shows once loaded; +- anything it said that was wrong. diff --git a/internal/cli/model_serve.go b/internal/cli/model_serve.go index e993b32..143a8ad 100644 --- a/internal/cli/model_serve.go +++ b/internal/cli/model_serve.go @@ -69,6 +69,8 @@ func newModelServeCmd(client func() *ctlClient) *cobra.Command { f.IntVar(&req.Nodes, "nodes", 0, "with --gpu-class: the nodes one replica spans, over a fabric (vLLM only)") f.IntVar(&req.TP, "tp", 0, "the older name of --gpus") _ = f.MarkHidden("tp") + f.StringToStringVar(&req.Env, "env", nil, + "KEY=VALUE in the engine's environment on every node, e.g. NCCL_DEBUG=INFO; repeatable") f.BoolVar(&req.Fallbacks, "fallbacks", false, "also add the next best ways to run it, used when the first is busy") f.BoolVar(&detach, "detach", false, "start it and return: it runs on the control plane; `llmgw ctl jobs` follows it") f.StringVar(&req.Name, "name", "", "the catalog name (default: from the repository)") diff --git a/internal/control/modelserve/modelserve.go b/internal/control/modelserve/modelserve.go index d5363b4..357c710 100644 --- a/internal/control/modelserve/modelserve.go +++ b/internal/control/modelserve/modelserve.go @@ -163,6 +163,10 @@ type Request struct { // CPUCores is what a replica computes on; zero takes half the smallest // node it fits on. CPUCores int `json:"cpu_cores,omitempty"` + // Env is set in the engine's environment, on every node of a replica: + // NCCL_DEBUG=INFO to see how its cards talk, NCCL_IB_* to steer them. + // Only the variables engines are tuned with; the catalog refuses others. + Env map[string]string `json:"env,omitempty"` // IfAbsent adds the model only if nobody has meanwhile: the leader's own // serving of a judge a policy names gives way to the one init or an // operator added while it fetched, rather than replacing it. @@ -580,6 +584,8 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model o = p } } + // A way the plan did not offer is still a model of experts. + o.ExpertParallel = engine == "vllm" && plan.Experts > 1 && o.GPUs() > 1 out = append(out, o) case len(opts) > 0: out = append(out, opts[0]) @@ -614,7 +620,7 @@ func chooseLayout(req Request, plan Plan, planned bool, set hf.FileSet) ([]model func profilesFor(layout []modelplan.Option, req Request, memGB, cores int) []catalog.Profile { var out []catalog.Profile for i, o := range layout { - p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i} + p := catalog.Profile{Engine: o.Engine, Weight: 1, Priority: 100 - 10*i, Env: req.Env} switch { case o.CPU: p.Name = "cpu" diff --git a/internal/control/modelserve/modelserve_test.go b/internal/control/modelserve/modelserve_test.go index e38539c..c33ea42 100644 --- a/internal/control/modelserve/modelserve_test.go +++ b/internal/control/modelserve/modelserve_test.go @@ -475,3 +475,26 @@ func TestAJobAddingOnlyIfAbsentGivesWayToAModelAddedMeanwhile(t *testing.T) { t.Errorf("job %s, added %d: %+v", j.State, len(taken.added), j.Steps) } } + +// A model of experts given its cards by hand spreads them still, and the +// environment asked for reaches every profile. +func TestAHandGivenLayoutOfExpertsKeepsExpertParallelAndTheEnvironment(t *testing.T) { + plan := Plan{Repo: "Qwen/Qwen3-235B-A22B", Experts: 128} + req := Request{Format: "safetensors", GPUClass: "h100-80g", GPUsPerNode: 8, Nodes: 2, ContextLength: 8192, + Env: map[string]string{"NCCL_DEBUG": "INFO"}} + layout, err := chooseLayout(req, plan, true, hf.FileSet{Name: "original"}) + if err != nil { + t.Fatal(err) + } + profs := profilesFor(layout, req, 0, 0) + p := profs[0] + if p.TP != 8 || p.PP != 2 || p.Nodes != 2 || !slices.Contains(p.Args, "--enable-expert-parallel") || p.Env["NCCL_DEBUG"] != "INFO" { + t.Errorf("profile %+v", p) + } + // A dense model given the same cards does not. + plan.Experts = 0 + layout, _ = chooseLayout(req, plan, true, hf.FileSet{Name: "original"}) + if slices.Contains(profilesFor(layout, req, 0, 0)[0].Args, "--enable-expert-parallel") { + t.Error("a dense model was given expert parallelism") + } +} diff --git a/internal/control/modelserve/plan.go b/internal/control/modelserve/plan.go index 38fa16d..3b3170d 100644 --- a/internal/control/modelserve/plan.go +++ b/internal/control/modelserve/plan.go @@ -45,7 +45,9 @@ type Plan struct { // KVBytesPerToken is one token of context, the whole model: what an // option needs for another context is worked out from it. 0 is unknown. KVBytesPerToken int64 `json:"kv_bytes_per_token,omitempty"` - Gated bool `json:"gated,omitempty"` + // Experts is a mixture-of-experts model's routed experts; 0 for dense. + Experts int `json:"experts,omitempty"` + Gated bool `json:"gated,omitempty"` // Servable is false when the nodes' engine cannot run the model at all, // and Reason says why. Servable bool `json:"servable"` @@ -161,7 +163,7 @@ func (s *Server) plan(ctx context.Context, repo hf.Repo, req PlanRequest) (Plan, p.Note = "the model's shape was not readable, so the context's memory is not counted" } shape.Params = p.Params - p.KVBytesPerToken = shape.KVBytesPerToken + p.KVBytesPerToken, p.Experts = shape.KVBytesPerToken, shape.Experts p.Options = modelplan.Options(arts, shape, s.deps.Pool(), p.Context) if !p.Servable { p.Options = nil diff --git a/internal/control/reconciler/reconciler.go b/internal/control/reconciler/reconciler.go index 9abdb43..42f525a 100644 --- a/internal/control/reconciler/reconciler.go +++ b/internal/control/reconciler/reconciler.go @@ -582,7 +582,11 @@ func (r *Reconciler) expireStuckReplicas() { if rep.State != Starting && rep.State != Placing { continue } - if now.Sub(rep.CreatedAt) < r.startupTimeout(rep) { + since := rep.CreatedAt + if rep.fetchedAt.After(since) { + since = rep.fetchedAt + } + if now.Sub(since) < r.startupTimeout(rep) { continue } rep.State = Failed @@ -1328,6 +1332,9 @@ func (r *Reconciler) OnFetchProgress(replicaID, shardID, source string, done, to if rep.Shards[i].ID != shardID { continue } + if prev := rep.Shards[i].Fetch; prev == nil || done > prev.BytesDone { + rep.fetchedAt = r.now() + } rep.Shards[i].Fetch = &FetchProgress{ Source: source, BytesDone: done, BytesTotal: total, } diff --git a/internal/control/reconciler/replica.go b/internal/control/reconciler/replica.go index 1f9a213..a8fb3a2 100644 --- a/internal/control/reconciler/replica.go +++ b/internal/control/reconciler/replica.go @@ -160,6 +160,11 @@ type Replica struct { AdoptedAt time.Time CreatedAt time.Time + // fetchedAt is when a shard's download last moved: the startup timeout + // runs from then, so a model of hundreds of gigabytes on a slow mirror + // is not failed while it is still arriving, and one whose download + // stalls is failed a timeout after it stopped. + fetchedAt time.Time ReadyAt time.Time // LastUsedAt is when a request was last routed to this replica, zero if // none has been since the router started. diff --git a/internal/control/reconciler/stuck_test.go b/internal/control/reconciler/stuck_test.go index 6ec2e07..2319961 100644 --- a/internal/control/reconciler/stuck_test.go +++ b/internal/control/reconciler/stuck_test.go @@ -406,3 +406,34 @@ func TestAMultiNodeReplicaKnowsHowManyShardsItWants(t *testing.T) { t.Errorf("WantShards = %d, want 2", reps[0].WantShards) } } + +// A model of hundreds of gigabytes on a slow mirror takes longer to arrive +// than to load: the startup timeout runs from when its download last moved, +// and a download that stops is failed a timeout after it stopped. +func TestAReplicaStillDownloadingIsNotFailed(t *testing.T) { + m := testModel(1, catalog.Profile{ + Name: "tp4", GPUClass: "h100-80g", TP: 4, StartupTimeout: 20 * time.Minute, + }) + h := newHarness(t, m, gpuNode("1", 8)) + h.r.Reconcile(context.Background()) + rep := h.r.Replicas()[0] + shard := rep.Shards[0].ID + + // Forty minutes of download, moving all the while. + for i := int64(1); i <= 8; i++ { + h.advance(5 * time.Minute) + h.r.OnFetchProgress(rep.ID, shard, "mirror", i<<35, 400<<30) + h.r.expireStuckReplicas() + } + if got := h.r.Replicas()[0].State; got == Failed { + t.Fatal("a replica whose download kept moving was failed") + } + // The same figure again is not progress: stalled, it fails a timeout on. + h.advance(10 * time.Minute) + h.r.OnFetchProgress(rep.ID, shard, "mirror", 8<<35, 400<<30) + h.advance(11 * time.Minute) + h.r.expireStuckReplicas() + if got := h.r.Replicas()[0].State; got != Failed { + t.Errorf("a stalled download was never failed: %s", got) + } +}