Files
ilkerandClaude Opus 5 d6e30e35cc hpcod: srun works from a compute node, not only from the head
"srun ls" in the desktop terminal died on

  dlopen(/usr/lib64/slurm/mpi_pmix.so): libhwloc.so.15: cannot open
  shared object file

MpiDefault is pmix, so even a plain srun loads that plugin. The node
image shipped the plugin without the library it links against, because
install_weak_deps=False - set on the Slurm line to keep singularity-ce
out - dropped hwloc-libs along with it. Naming the two libraries
explicitly keeps both properties.

Nothing to do with the container. The same command fails the same way on
a bare compute node; it only looked container-specific because every
test I had run launched srun from the head, which has the library for
other reasons. A failure that appears only when somebody opens a
terminal on a node is one that testing from the head cannot find.

The build now refuses an image whose pmix plugin has unresolved
libraries, so the next version of this cannot ship quietly.

Verified after rebuilding and rebooting the nodes: libhwloc.so.15
resolves to 15.4.1 on the node and ldd reports nothing missing for
mpi_pmix.so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-21 23:12:27 +02:00

126 lines
7.1 KiB
Docker

# The compute node image, built once and published to a registry.
#
# Warewulf imports node images from OCI, so the natural place to build one is a
# container build - not a chroot copied between hosts. An OCI layer carries
# ownership and permissions in its metadata, so it does not care who runs the
# transport; that is precisely what broke when this was rsync driven by an
# unprivileged control plane.
#
# Building here also takes the external package repositories out of the
# provisioning path. Every tenant used to re-run this dnf against EPEL and
# OpenHPC, so an outage at either one meant no tenant could be provisioned.
# Now it is a build-time dependency and tenants pull one immutable artefact.
#
# podman build -t gitea.mynodes.xyz/ilker/hpc-node:rocky9-ohpc3 .
# podman push gitea.mynodes.xyz/ilker/hpc-node:rocky9-ohpc3
#
# The tag is the cache key: it names the stack a tenant asked for.
FROM ghcr.io/warewulf/warewulf-rockylinux:9
ARG WAREWULF_VERSION=4.7.0
ARG OHPC_RELEASE=http://repos.openhpc.community/OpenHPC/3/EL_9/x86_64/ohpc-release-3-1.el9.x86_64.rpm
# Device firmware is 657 MB of an unslimmed image - by far the largest single
# item - and a guest using virtio needs none of it. Set KEEP_FIRMWARE=1 for a
# site whose compute nodes are physical, or whose passthrough devices need
# kernel firmware blobs.
ARG KEEP_FIRMWARE=0
# Kernel and the wwinit dracut module. The base image has no kernel at all, so
# without this it cannot netboot.
RUN dnf install -y -q --nodocs --setopt=install_weak_deps=False epel-release dnf-plugins-core \
&& dnf config-manager --set-enabled crb \
&& dnf install -y -q --nodocs --setopt=install_weak_deps=False \
kernel kernel-modules dracut dracut-network \
NetworkManager openssh-server openssh-clients \
nfs-utils chrony \
iproute iputils procps-ng util-linux passwd sudo less which tar \
&& curl -sLo /tmp/ww.rpm \
"https://github.com/warewulf/warewulf/releases/download/v${WAREWULF_VERSION}/warewulf-dracut-${WAREWULF_VERSION}-1.el9.noarch.rpm" \
&& dnf install -y -q /tmp/ww.rpm \
&& rm -f /tmp/ww.rpm \
&& systemctl enable sshd NetworkManager chronyd
# OpenHPC compute stack.
#
# install_weak_deps=False on the ohpc-base-compute line is not cosmetic: its
# recommends pulled in singularity-ce, which nothing here requires and which
# installs a setuid-root starter on every compute node. The container runtime
# is chosen explicitly below instead of arriving as a suggestion.
RUN dnf install -y -q --nodocs "${OHPC_RELEASE}" \
&& dnf install -y -q --nodocs --setopt=install_weak_deps=False ohpc-base-compute lmod-ohpc \
&& dnf install -y -q --nodocs --setopt=install_weak_deps=False slurm-slurmd-ohpc munge \
hwloc-libs numactl-libs \
&& systemctl enable munge slurmd
# hwloc-libs and numactl-libs are named above because they are weak
# dependencies of Slurm, and install_weak_deps=False - set on that line to
# keep singularity-ce out - drops them too. The node then ships
# mpi_pmix.so without the libhwloc it links against, and since
# MpiDefault=pmix, plain "srun ls" run from a compute node dies on
# "dlopen(/usr/lib64/slurm/mpi_pmix.so): libhwloc.so.15: cannot open
# shared object file". It works from the head, which has the library for
# other reasons, so the failure only appears once somebody opens a
# terminal on a node.
#
# Checked here rather than discovered there.
RUN test -z "$(ldd /usr/lib64/slurm/mpi_pmix.so 2>/dev/null | grep 'not found')" \
|| { echo "the pmix plugin is missing libraries:"; ldd /usr/lib64/slurm/mpi_pmix.so | grep 'not found'; exit 1; }
# Apptainer, not SingularityCE, and without the suid package.
#
# They are the same project either side of a 2021 split - Apptainer went to
# the Linux Foundation, SingularityCE stayed with Sylabs above a commercial
# edition - and both read the same SIF images and answer to the same
# `singularity` command, which apptainer also provides. Three things decided
# it.
#
# The setuid binary is the first and the reason. SingularityCE installs
# starter-suid unconditionally; Apptainer splits it into apptainer-suid, so
# not installing that package leaves no setuid-root binary on a machine that
# runs other people's jobs. Apptainer then uses user namespaces instead.
#
# Second, governance: one vendor's open core against multi-vendor
# foundation governance, for a platform meant to run for years.
#
# Third, size: 36 MB against 55 MB. On its own that would not decide
# anything, but this image is unpacked into tmpfs, so every megabyte is paid
# twice - once in each node's RAM and once in every PXE transfer.
RUN dnf install -y -q --nodocs --setopt=install_weak_deps=False apptainer \
&& ! rpm -q apptainer-suid >/dev/null 2>&1 \
&& ! find /usr/libexec -perm -4000 -type f 2>/dev/null | grep -q . \
&& command -v singularity >/dev/null
# slurmd will not start without its spool directory. It is runtime state, so
# tmpfiles.d recreates it on every boot of a filesystem that lives in RAM.
RUN mkdir -p /var/spool/slurm/d /var/spool/slurm/ctld \
&& chown -R slurm:slurm /var/spool/slurm \
&& chown root:root /var/spool/slurm/d \
&& chmod 0755 /var/spool/slurm /var/spool/slurm/d \
&& printf 'd /var/spool/slurm 0755 slurm slurm -\nd /var/spool/slurm/d 0755 root root -\n' \
> /etc/tmpfiles.d/slurm.conf
# NFSv4 maps users by user@domain, so this must match the cluster's setting or
# files arrive owned by nobody:nobody.
RUN sed -i 's/^#\?Domain =.*/Domain = cluster.local/' /etc/idmapd.conf
# Slimming. Measured on the unslimmed image: firmware 657 MB, the base image's
# leftover kernel modules 100 MB, locales 55 MB, vim 29 MB. Every megabyte here
# is paid twice - once pulling the image to the head node, once again sending
# it to each compute node over PXE - and a third time as RAM, because the root
# filesystem is a tmpfs.
RUN set -eux; if [ "${KEEP_FIRMWARE}" = "0" ]; then rm -rf /usr/lib/firmware/*; fi; running=$(rpm -q --qf '%{VERSION}-%{RELEASE}.%{ARCH}\n' kernel | tail -1); for d in /usr/lib/modules/*; do [ "$(basename "$d")" = "${running}" ] || rm -rf "$d"; done; for f in /boot/initramfs-*.img /boot/symvers-*.gz; do case "$f" in *"${running}"*) : ;; *) rm -f "$f" ;; esac; done; rm -rf /usr/share/locale/* /usr/share/doc/* /usr/share/man/* /usr/share/info/*; rm -rf /usr/share/licenses/* /usr/share/cracklib/* /usr/share/i18n/locales/*; dnf clean all; rm -rf /var/cache/dnf /var/log/dnf* /var/log/hawkey*
# The initramfs must be regenerated after the slimming above, or it still
# references firmware and modules that are no longer there.
#
# --add wwinit is what makes two-stage boot possible, and it is not optional
# here. Without it the node boots this initramfs and has no way to fetch its
# root filesystem: it fails, iPXE waits 15s, reboots, and repeats. --no-hostonly
# because dracut is running in a container against a build host whose hardware
# has nothing to do with the machines that will boot this.
RUN kver=$(rpm -q --qf '%{VERSION}-%{RELEASE}.%{ARCH}\n' kernel | tail -1) \
&& dracut --force --no-hostonly --add wwinit --kver "${kver}" /boot/initramfs-${kver}.img \
&& lsinitrd /boot/initramfs-${kver}.img | grep -q wwinit