diff --git a/BACKLOG.md b/BACKLOG.md index 3a7275d..1058091 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1983,6 +1983,36 @@ one control plane's router can. In order: - O3. Only if O2 is not enough: stateless gateway machines (`llmgw server --role gateway`) in front of the leader's control plane, as many as the traffic needs. +## P. Training beyond one card (written 2026-11-11) — decision open + +F4 trains a LoRA on one card of a node the caller names (`ctl train start --node`), one job per node, no resume. +Larger work (full fine-tuning, a LoRA on a 70B, several GPUs or machines) needs more. Two ways: + +**A. In llmgw.** Add, in order: a job that takes N whole GPUs on one node (torchrun, FSDP/DeepSpeed in the agent's +venv); placement by the control plane instead of `--node`, freeing GPUs from replicas by the scaling schedule or by +draining; checkpoints to shared storage and resume when an agent or node goes; datasets uploaded or read from +storage, not only KB pairs; then gang placement of N nodes with a rendezvous on the first, the NCCL environment from +the fabric discovery (IB/NVLink, `llmgw doctor`); a console screen with logs and loss. Size: L for one node, XL for +several. For: one product, one install, one set of users and grants (KB rules carry over), GPUs shared with +inference without a second scheduler. Against: it is Slurm's gang scheduling, requeue and accounting written again; +multi-node NCCL cannot be tried on the PoC (one GPU node, no IB); every customer's training code would have to fit +our job format. + +**B. Slurm beside llmgw.** The TUI installs Slurm on the nodes the customer picks (munge, slurmctld on the control +planes, slurmd, users from the same LDAP/AD), with pyxis/enroot for containers. A node is handed between the two: +llmgw drains its replicas and `scontrol update state=resume`, and back. llmgw's part is what comes out: a checkpoint +or adapter from a Slurm job published to the mirror (`ctl train import`), served through canary and the eval. Size: +M. For: what research and university customers already run (`sbatch`, `srun torchrun`), any training code, mature +multi-node and requeue; we write only the install and the handover. Against: a second system with its own failures, +upgrades and users to keep in step; GPUs split by hand or by the handover, not by demand; access rules on training +data are Slurm's and the file system's, not the KB's. + +Both need shared storage (NFS, BeeGFS, Lustre) the customer provides; neither changes F4. + +Recommendation: keep F4 in llmgw for small single-card adapters tied to KB grants; take B for anything larger; +build only A's first step (N GPUs on one node, placed by the control plane, with resume) if customers ask for +mid-size jobs without wanting Slurm. + ## D. Facts worth knowing before starting - Admin API: `127.0.0.1:9090`, behind the token in