BACKLOG P: training beyond one card, in llmgw or Slurm beside it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

BACKLOG.md

@@ -1983,6 +1983,36 @@ one control plane's router can. In order:
 - O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's
   control plane, as many as the traffic needs.
 
+## P. Training beyond one card (written 2026-11-11) — decision open
+
+F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume.
+Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways:
+
+**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's
+venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by
+draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from
+storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from
+the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for
+several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with
+inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again;
+multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit
+our job format.
+
+**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control
+planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two:
+llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint
+or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size:
+M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature
+multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures,
+upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training
+data are Slurm's and the file system's, not the KB's.
+
+Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4.
+
+Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger;
+build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for
+mid-size jobs without wanting Slurm.
+
 ## D. Facts worth knowing before starting
 
 - Admin API: `127.0.0.1:9090`, behind the token in