mpi-ring finishes in about two seconds, so there is nothing to observe in squeue, no meaningful sacct record and no load on the nodes. heat_mpi solves 2D steady-state heat diffusion by Jacobi iteration over a grid split into horizontal strips, one per rank. Every iteration performs a five-point stencil, a halo exchange with the neighbouring ranks via MPI_Sendrecv, and an MPI_Allreduce for the global residual - so it exercises compute and both point-to-point and collective communication rather than just starting up and exiting. It runs to a wall-clock budget (-m minutes) instead of a fixed iteration count, which is what makes it useful for watching the scheduler. Stopping is collective: rank 0 decides and broadcasts, so every rank leaves on the same iteration. The residual falls monotonically, which doubles as a check that the halo exchange is correct. Measured on this cluster: 6144 grid, 8 ranks over 4 nodes, ~12 GFLOP/s with all nodes at ~95% CPU and ~72 MiB per rank. heat-long.sbatch defaults to 20 minutes and takes GRID, MINUTES and REPORT through --export.
8.5 KiB
6. Slurm and MPI
6.1 Software stack
Everything comes from OpenHPC 3 for EL_9
(http://repos.openhpc.community/OpenHPC/3/EL_9/x86_64/).
| Component | Version | Package |
|---|---|---|
| Slurm | 25.11.4 | slurm-ohpc |
| OpenMPI | 5.0.7 | openmpi5-gnu14-ohpc |
| GCC | 14.2.0 | gnu14-compilers-ohpc |
| Lmod | 9.2 | lmod-ohpc |
| MUNGE | 0.5.13 | munge (base OS) |
| PMIx | 4.2.9 | pmix-ohpc |
OpenHPC installs under /opt/ohpc/pub, which is NFS-exported read-only from
head01 — so nodes get the identical toolchain without it being baked into the
image.
6.2 Slurm configuration
/etc/slurm/slurm.conf on head01, distributed to the nodes through the hpc
overlay:
ClusterName=wwtest
SlurmctldHost=head01(10.10.10.1)
AuthType=auth/munge
ProctrackType=proctrack/cgroup
TaskPlugin=task/affinity,task/cgroup
MpiDefault=pmix
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core
ReturnToService=2
NodeName=wwnode[1-4] CPUs=2 Sockets=1 CoresPerSocket=2 ThreadsPerCore=1 \
RealMemory=7000 State=UNKNOWN
PartitionName=normal Nodes=wwnode[1-4] Default=YES MaxTime=INFINITE State=UP
RealMemory=7000 is set below the 8192 MB the VMs actually have. The gap is
deliberate: the rootfs occupies roughly 1.8 GB of RAM as tmpfs, so advertising
the full amount would let Slurm oversubscribe the node into OOM.
ReturnToService=2 matters for stateless nodes: a node that reboots comes back
with a fresh slurmd and should rejoin without an operator manually clearing
its DOWN state.
MUNGE
slurm.conf and munge.key both ship in the hpc overlay. The key must be
byte-identical across head and nodes, mode 0400, owned by munge.
Two permission details caused real failures here — the overlay must set both
/etc/munge ownership (to munge:munge, mode 0700) and the key itself.
munged checks the directory, not just the file, and refuses to start with
Failed to check keyfile: Permission denied if it cannot traverse it.
slurmd spool directory
/var/spool/slurm/d must exist before slurmd starts. It is runtime state,
not configuration, so it belongs in the image rather than an overlay. It is
created inside the image and re-asserted at every boot via
/etc/tmpfiles.d/slurm.conf.
6.3 Running jobs
ssh head01
sudo -u hpcuser -i # or log in as any IPA user
module avail # what Lmod offers
module load gnu14 openmpi5 # hierarchical: openmpi5 appears after gnu14
sinfo # partition state
squeue # queue
sbatch mpi-ring.sbatch # submit
scancel <jobid>
The test job
mpi/mpi-ring.sbatch:
#SBATCH --job-name=mpi-ring
#SBATCH --partition=normal
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=2
#SBATCH --output=/home/hpcuser/mpi-ring-%j.out
module purge
module load gnu14 openmpi5
srun --mpi=pmix /home/hpcuser/hello_mpi
mpi/hello_mpi.c reports each rank's hostname, then passes a token around a
ring across all ranks and checks the result. With 8 ranks the token starts at
42 and returns as 49 — ranks 1..7 each add one.
Compile:
module load gnu14 openmpi5
mpicc -O2 -o hello_mpi hello_mpi.c
srun --mpi=pmix is used rather than mpirun. Slurm launches the ranks
directly through PMIx, so no SSH between nodes is required — which matters
because the nodes are stateless and have no user SSH keys of their own.
6.4 Verified result
=== Slurm ===
Job: 4
Nodes: wwnode[1-4]
Tasks: 8
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
Rank 1 / 8 host=wwnode1 mpi_proc=wwnode1
Rank 2 / 8 host=wwnode2 mpi_proc=wwnode2
...
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
The job ran as the IPA user hpcuser (uid 721800005) with its home on the
NFS share, writing output back to /home/hpcuser — exercising the scheduler,
MPI, identity and storage layers in a single test.
A long-running job
mpi-ring finishes in about two seconds, which is useless for watching the
scheduler. mpi/heat_mpi.c solves 2D steady-state heat diffusion by Jacobi
iteration and runs for a wall-clock budget you choose.
Each iteration does both kinds of work a real job does: a five-point stencil
over the rank's strip of the grid, a halo exchange with the ranks above and
below (MPI_Sendrecv, so it cannot deadlock), and one MPI_Allreduce for the
global convergence measure.
sbatch heat-long.sbatch # ~20 min, 4 nodes, 8 ranks
sbatch --export=MINUTES=60 heat-long.sbatch # an hour
sbatch --export=GRID=8192,MINUTES=45 heat-long.sbatch
srun -N4 -n8 --mpi=pmix ./heat_mpi -n 4096 -m 5 -r 200 # interactive
| Flag | Meaning | Default |
|---|---|---|
-n |
grid points per side | 4096 |
-m |
wall-clock budget in minutes | 10 |
-i |
stop after this many iterations | unlimited |
-r |
report interval in iterations | 200 |
-e |
stop early below this residual | never |
Progress is printed by rank 0 while it runs:
iter residual elapsed(s) iter/s GFLOP/s
1 2.500000e+01 0.1 17.22 3.90
500 4.839573e-02 9.2 54.57 12.35
The residual falls monotonically — that is the physics being right, and a useful sanity check that the halo exchange is correct. On this cluster (8 ranks, 6144² grid) it sustains roughly 12 GFLOP/s with all four nodes at ~95% CPU.
Stopping is collective: rank 0 decides and broadcasts, so all ranks leave the loop on the same iteration rather than deadlocking on a straggler.
Memory per rank is about 2 x (rows+2) x N x 8 bytes — at -n 6144 with 8
ranks that is ~72 MiB each. Keep an eye on it if you raise the grid, because
the node rootfs is already in RAM.
6.5 The default login environment
hello_mpi links against UCX (libucp, libucs, libucm, libuct) and
libhwloc, which live under /opt/ohpc/pub and only appear on
LD_LIBRARY_PATH once the modules are loaded. A user who logs in and runs the
binary directly would otherwise get:
./hello_mpi: error while loading shared libraries: libucp.so.0: cannot open
shared object file: No such file or directory
lmod-ohpc installs /etc/profile.d/lmod.sh, which ends with
module try-add ohpc — so every login shell loads a module named ohpc if one
exists. OpenHPC ships that modulefile in its lmod-defaults-* packages, but
only for gnu12 and gnu15 combinations; this cluster is gnu14, so
/opt/ohpc/pub/modulefiles/ohpc is written by 60-slurm-openhpc.sh instead.
It lives on /opt/ohpc, exported read-only to the nodes, so a single copy
serves head and compute alike and changing it needs no image rebuild.
A fresh login now has:
prun/2.2 gnu14/14.2.0 hwloc/2.13.0 ucx/1.20.1 libfabric/1.18.0
openmpi5/5.0.7 ohpc
module purge removes them and the library error returns — that is expected.
module reset restores this default set.
6.6 Accounting
Enabled by A85-slurm-accounting.sh: slurmdbd 25.11.4 against MariaDB 10.5
on the head node.
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=head01
AccountingStoragePort=6819
JobAcctGatherType=jobacct_gather/cgroup
JobAcctGatherFrequency=30
Accounting is recorded but not enforced — AccountingStorageEnforce is
deliberately unset, so a user without an association can still submit. To
require one:
AccountingStorageEnforce=associations,limits,qos
Associations created so far: cluster wwtest → account hpc →
users hpcuser, testuser.
sacct -j <id> # one job, with steps
sacct -a -X --starttime=today # everything today, jobs only
sreport cluster utilization start=today
sacctmgr show assoc format=Cluster,Account,User
sacctmgr -i add user ayse Account=hpc
Database credentials are in head01:/root/.slurmdbd-credentials (mode 0600);
slurmdbd.conf holds the same password and is 0600 slurm:slurm.
slurmdbd logs "Database settings not recommended values:
innodb_buffer_pool_size" on start. That is deliberate: it wants 4 GB, head01
has 8 GB in total and also runs slurmctld, warewulfd, NFS, dhcpd, tftp and the
overlay UI. A four-node lab's accounting database is a few megabytes, so the
pool is set to 1 GB and the warning is cosmetic.
6.7 Not configured
- Fair-share / QOS / limits — the associations now exist, so these are a
sacctmgrchange away, but no policy is defined. - High availability — a single
slurmctld, noSlurmctldHostbackup, andslurmdbdand MariaDB are on that same host.