Files
werewulf/docs/06-slurm-mpi.md
T
ilker f29e8785ef Add a long-running MPI benchmark
mpi-ring finishes in about two seconds, so there is nothing to observe in
squeue, no meaningful sacct record and no load on the nodes.

heat_mpi solves 2D steady-state heat diffusion by Jacobi iteration over a
grid split into horizontal strips, one per rank. Every iteration performs a
five-point stencil, a halo exchange with the neighbouring ranks via
MPI_Sendrecv, and an MPI_Allreduce for the global residual - so it exercises
compute and both point-to-point and collective communication rather than just
starting up and exiting.

It runs to a wall-clock budget (-m minutes) instead of a fixed iteration
count, which is what makes it useful for watching the scheduler. Stopping is
collective: rank 0 decides and broadcasts, so every rank leaves on the same
iteration. The residual falls monotonically, which doubles as a check that the
halo exchange is correct.

Measured on this cluster: 6144 grid, 8 ranks over 4 nodes, ~12 GFLOP/s with
all nodes at ~95% CPU and ~72 MiB per rank.

heat-long.sbatch defaults to 20 minutes and takes GRID, MINUTES and REPORT
through --export.
2026-08-31 10:35:56 +02:00

8.5 KiB

6. Slurm and MPI

6.1 Software stack

Everything comes from OpenHPC 3 for EL_9 (http://repos.openhpc.community/OpenHPC/3/EL_9/x86_64/).

Component Version Package
Slurm 25.11.4 slurm-ohpc
OpenMPI 5.0.7 openmpi5-gnu14-ohpc
GCC 14.2.0 gnu14-compilers-ohpc
Lmod 9.2 lmod-ohpc
MUNGE 0.5.13 munge (base OS)
PMIx 4.2.9 pmix-ohpc

OpenHPC installs under /opt/ohpc/pub, which is NFS-exported read-only from head01 — so nodes get the identical toolchain without it being baked into the image.

6.2 Slurm configuration

/etc/slurm/slurm.conf on head01, distributed to the nodes through the hpc overlay:

ClusterName=wwtest
SlurmctldHost=head01(10.10.10.1)

AuthType=auth/munge
ProctrackType=proctrack/cgroup
TaskPlugin=task/affinity,task/cgroup
MpiDefault=pmix

SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
ReturnToService=2

NodeName=wwnode[1-4] CPUs=2 Sockets=1 CoresPerSocket=2 ThreadsPerCore=1 \
         RealMemory=7000 State=UNKNOWN
PartitionName=normal Nodes=wwnode[1-4] Default=YES MaxTime=INFINITE State=UP

RealMemory=7000 is set below the 8192 MB the VMs actually have. The gap is deliberate: the rootfs occupies roughly 1.8 GB of RAM as tmpfs, so advertising the full amount would let Slurm oversubscribe the node into OOM.

ReturnToService=2 matters for stateless nodes: a node that reboots comes back with a fresh slurmd and should rejoin without an operator manually clearing its DOWN state.

MUNGE

slurm.conf and munge.key both ship in the hpc overlay. The key must be byte-identical across head and nodes, mode 0400, owned by munge.

Two permission details caused real failures here — the overlay must set both /etc/munge ownership (to munge:munge, mode 0700) and the key itself. munged checks the directory, not just the file, and refuses to start with Failed to check keyfile: Permission denied if it cannot traverse it.

slurmd spool directory

/var/spool/slurm/d must exist before slurmd starts. It is runtime state, not configuration, so it belongs in the image rather than an overlay. It is created inside the image and re-asserted at every boot via /etc/tmpfiles.d/slurm.conf.

6.3 Running jobs

ssh head01
sudo -u hpcuser -i          # or log in as any IPA user

module avail                # what Lmod offers
module load gnu14 openmpi5  # hierarchical: openmpi5 appears after gnu14

sinfo                       # partition state
squeue                      # queue
sbatch mpi-ring.sbatch      # submit
scancel <jobid>

The test job

mpi/mpi-ring.sbatch:

#SBATCH --job-name=mpi-ring
#SBATCH --partition=normal
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=2
#SBATCH --output=/home/hpcuser/mpi-ring-%j.out

module purge
module load gnu14 openmpi5
srun --mpi=pmix /home/hpcuser/hello_mpi

mpi/hello_mpi.c reports each rank's hostname, then passes a token around a ring across all ranks and checks the result. With 8 ranks the token starts at 42 and returns as 49 — ranks 1..7 each add one.

Compile:

module load gnu14 openmpi5
mpicc -O2 -o hello_mpi hello_mpi.c

srun --mpi=pmix is used rather than mpirun. Slurm launches the ranks directly through PMIx, so no SSH between nodes is required — which matters because the nodes are stateless and have no user SSH keys of their own.

6.4 Verified result

=== Slurm ===
Job:    4
Nodes:  wwnode[1-4]
Tasks:  8

Rank  0 /  8  host=wwnode1    mpi_proc=wwnode1
Rank  1 /  8  host=wwnode1    mpi_proc=wwnode1
Rank  2 /  8  host=wwnode2    mpi_proc=wwnode2
...
Ring complete. start=42, returned=49, expected=49, hops=8  -> SUCCESS

The job ran as the IPA user hpcuser (uid 721800005) with its home on the NFS share, writing output back to /home/hpcuser — exercising the scheduler, MPI, identity and storage layers in a single test.

A long-running job

mpi-ring finishes in about two seconds, which is useless for watching the scheduler. mpi/heat_mpi.c solves 2D steady-state heat diffusion by Jacobi iteration and runs for a wall-clock budget you choose.

Each iteration does both kinds of work a real job does: a five-point stencil over the rank's strip of the grid, a halo exchange with the ranks above and below (MPI_Sendrecv, so it cannot deadlock), and one MPI_Allreduce for the global convergence measure.

sbatch heat-long.sbatch                        # ~20 min, 4 nodes, 8 ranks
sbatch --export=MINUTES=60 heat-long.sbatch    # an hour
sbatch --export=GRID=8192,MINUTES=45 heat-long.sbatch

srun -N4 -n8 --mpi=pmix ./heat_mpi -n 4096 -m 5 -r 200   # interactive
Flag Meaning Default
-n grid points per side 4096
-m wall-clock budget in minutes 10
-i stop after this many iterations unlimited
-r report interval in iterations 200
-e stop early below this residual never

Progress is printed by rank 0 while it runs:

    iter       residual   elapsed(s)       iter/s    GFLOP/s
       1   2.500000e+01          0.1        17.22       3.90
     500   4.839573e-02          9.2        54.57      12.35

The residual falls monotonically — that is the physics being right, and a useful sanity check that the halo exchange is correct. On this cluster (8 ranks, 6144² grid) it sustains roughly 12 GFLOP/s with all four nodes at ~95% CPU.

Stopping is collective: rank 0 decides and broadcasts, so all ranks leave the loop on the same iteration rather than deadlocking on a straggler.

Memory per rank is about 2 x (rows+2) x N x 8 bytes — at -n 6144 with 8 ranks that is ~72 MiB each. Keep an eye on it if you raise the grid, because the node rootfs is already in RAM.

6.5 The default login environment

hello_mpi links against UCX (libucp, libucs, libucm, libuct) and libhwloc, which live under /opt/ohpc/pub and only appear on LD_LIBRARY_PATH once the modules are loaded. A user who logs in and runs the binary directly would otherwise get:

./hello_mpi: error while loading shared libraries: libucp.so.0: cannot open
shared object file: No such file or directory

lmod-ohpc installs /etc/profile.d/lmod.sh, which ends with module try-add ohpc — so every login shell loads a module named ohpc if one exists. OpenHPC ships that modulefile in its lmod-defaults-* packages, but only for gnu12 and gnu15 combinations; this cluster is gnu14, so /opt/ohpc/pub/modulefiles/ohpc is written by 60-slurm-openhpc.sh instead.

It lives on /opt/ohpc, exported read-only to the nodes, so a single copy serves head and compute alike and changing it needs no image rebuild.

A fresh login now has:

prun/2.2  gnu14/14.2.0  hwloc/2.13.0  ucx/1.20.1  libfabric/1.18.0
openmpi5/5.0.7  ohpc

module purge removes them and the library error returns — that is expected. module reset restores this default set.

6.6 Accounting

Enabled by A85-slurm-accounting.sh: slurmdbd 25.11.4 against MariaDB 10.5 on the head node.

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=head01
AccountingStoragePort=6819
JobAcctGatherType=jobacct_gather/cgroup
JobAcctGatherFrequency=30

Accounting is recorded but not enforced — AccountingStorageEnforce is deliberately unset, so a user without an association can still submit. To require one:

AccountingStorageEnforce=associations,limits,qos

Associations created so far: cluster wwtest → account hpc → users hpcuser, testuser.

sacct -j <id>                  # one job, with steps
sacct -a -X --starttime=today  # everything today, jobs only
sreport cluster utilization start=today
sacctmgr show assoc format=Cluster,Account,User
sacctmgr -i add user ayse Account=hpc

Database credentials are in head01:/root/.slurmdbd-credentials (mode 0600); slurmdbd.conf holds the same password and is 0600 slurm:slurm.

slurmdbd logs "Database settings not recommended values: innodb_buffer_pool_size" on start. That is deliberate: it wants 4 GB, head01 has 8 GB in total and also runs slurmctld, warewulfd, NFS, dhcpd, tftp and the overlay UI. A four-node lab's accounting database is a few megabytes, so the pool is set to 1 GB and the warning is cosmetic.

6.7 Not configured

  • Fair-share / QOS / limits — the associations now exist, so these are a sacctmgr change away, but no policy is defined.
  • High availability — a single slurmctld, no SlurmctldHost backup, and slurmdbd and MariaDB are on that same host.