Files
werewulf/hpcod/image/apps/spack.Containerfile.in
ilkerandClaude Opus 5 275997c52a hpcod: make the Spack path work, and give it a build cache
Four things stood between a spack.yaml recipe and an image, and all
four were invisible because nothing had ever run this path: every
Spack recipe in the catalogue was still draft.

The default builder image was spack/rockylinux9:v0.23.1, a tag that
has never existed - Spack's images are tagged by release and Spack
went to 1.x. Pinned to 1.2.2 rather than :latest, because the
concretizer decides the whole catalogue's ABI and should not change
under us between two applications.

Spack was assumed to be on PATH. `spack env activate` changes the
calling shell, so it is a function that setup-env.sh defines; running
the binary prints what it would have set and changes nothing.

config: install_tree took a bare path, which Spack 1.x rejects.

And the MPI line could not concretize at all: +pmi and +legacylaunchers
exist only up to OpenMPI 4, so asking for them with @5.0.6 collapsed
the spec to 4.1.8 and then refused it. OpenMPI 5 is launched by srun
through PMIx, and ~internal-pmix is what makes it use the external
pmix@4.2.9 the recipes pin - without it OpenMPI bundles its own and
the pin means nothing.

The build cache was a dead wire at both ends. The template read from
$SPACK_MIRROR and build-push.sh never passed it, and nothing ever
pushed, so it could only have been empty. It is now passed by default,
pointing at the same registry the images go to, and the install
autopushes - during, not after, so a build that fails on its last
package still leaves the half hour of OpenMPI before it.

Reading needs no credential, because the registry is public by
design; writing does, and it arrives as a mounted secret rather than
a build argument. Arguments are recorded in the image metadata, and
these images are published - a token passed that way would ship
inside every application in the catalogue.

osu-micro-benchmarks is the first entry that claims to run across
nodes. Everything else says single-node, so the path that carries
ranks between two machines had never been exercised by anything the
catalogue offers. Its binaries are declared by absolute path because
that is where OSU puts them: under libexec, which the view carries
and nothing puts on PATH, so a bare name would be a promise the image
does not keep.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 13:22:36 +02:00

117 lines
5.6 KiB
Docker

# Two-stage Spack build, rendered by build-push.sh from the application's
# spack.yaml. @SPACK_IMAGE@ and @RUNTIME_IMAGE@ are substituted; everything
# else is the same for every application, which is the point - one template
# means one MPI, one compiler and one set of runtime libraries across the
# whole catalogue.
#
# Stage 1 carries a compiler toolchain and the Spack source tree, several
# gigabytes of it. Stage 2 carries the installed software and nothing that
# built it. Copying /opt/software and /opt/view together matters: the view is
# a tree of symlinks *into* the install tree, and a view without its targets
# is a directory of dangling links that looks fine until something runs.
FROM @SPACK_IMAGE@ AS builder
# The environment is copied in rather than created with `spack env create`,
# so the file in the repository is exactly the file that was concretised.
COPY spack.yaml /opt/spack-env/spack.yaml
# An OCI build cache, when one is configured. A cold build of GROMACS is
# around 40 minutes and almost all of it is dependencies that every other
# application in the catalogue also needs; with the cache populated the same
# build is a download. Unset means build from source, which is correct on
# the first run and merely slow afterwards.
# The build cache, read unauthenticated. Reading needs no credential
# because the registry is public by design, and that is what keeps this an
# ARG rather than a secret: a build argument is recorded in the image's own
# metadata, so a token passed this way ships inside every application we
# publish. Pushing does need one, and takes the secret mount below.
ARG SPACK_MIRROR=""
# Spack has to be sourced, not merely found on PATH. `spack env activate`
# changes the calling shell, so it is a shell function that setup-env.sh
# defines - running the binary directly prints the environment it would
# have set and changes nothing, and the install that follows then builds
# into the wrong place or, as here, does not run at all. Bash because that
# is what the function is written for; a RUN is /bin/sh otherwise.
SHELL ["/bin/bash", "-c"]
# Build once, reuse everywhere.
#
# Almost everything in an MPI application's dependency tree - the whole of
# OpenMPI, hwloc, libevent, pmix, ucx, libfabric - is shared with every
# other application in this catalogue. Compiled from source that is half an
# hour per application; from a cache it is a download.
#
# --autopush rather than one push at the end: a build that fails on its last
# package still leaves everything before it in the cache, which is the
# difference between losing half an hour and losing nothing.
#
# The token is a mounted secret, never an ARG. Build arguments are recorded
# in the image's own metadata, so a token passed that way would ship inside
# every application published from this template. Reading needs no
# credential at all, because the registry is public by design - so a build
# with no secret is slower and not broken, and says which it is.
RUN --mount=type=cache,target=/root/.spack/cache \
--mount=type=secret,id=mirror_token \
set -eux; \
. /opt/spack/share/spack/setup-env.sh; \
cd /opt/spack-env; \
if [ -n "$SPACK_MIRROR" ]; then \
spack mirror add --unsigned hpcod "$SPACK_MIRROR"; \
if [ -s /run/secrets/mirror_token ]; then \
spack mirror set --push --autopush \
--oci-username-variable SPACK_OCI_USER \
--oci-password-variable SPACK_OCI_TOKEN hpcod; \
export SPACK_OCI_USER="${SPACK_MIRROR_USER:-hpcod}"; \
export SPACK_OCI_TOKEN="$(cat /run/secrets/mirror_token)"; \
echo "build cache: reading and writing $SPACK_MIRROR"; \
else \
echo "build cache: reading $SPACK_MIRROR, no credential to write it"; \
fi; \
spack buildcache update-index hpcod || true; \
fi; \
spack env activate .; \
spack concretize --force; \
spack install --fail-fast --no-check-signature; \
spack gc -y; \
spack env deactivate
# Strip what a runtime image has no use for. Spack keeps the full source and
# build logs of every package it installed, and on a stack the size of an MPI
# application that is the larger half of the image.
RUN set -eux; \
find /opt/software -name '*.a' -delete; \
rm -rf /opt/software/*/*/*/.spack/archive /opt/spack-env/.spack-env/._view 2>/dev/null || true; \
strip -s /opt/view/bin/* 2>/dev/null || true
FROM @RUNTIME_IMAGE@
# The runtime libraries Spack-built binaries link against from the system
# rather than from the view: libstdc++ in particular, which has to be the
# same one the compiler in stage 1 used. Same base family, so it is.
RUN dnf install -y -q --nodocs --setopt=install_weak_deps=False \
libgomp libstdc++ libatomic numactl-libs hwloc-libs \
libibverbs librdmacm rdma-core \
&& dnf clean all -y && rm -rf /var/cache/dnf /usr/share/doc /usr/share/man
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
ENV PATH="/opt/view/bin:${PATH}" \
LD_LIBRARY_PATH="/opt/view/lib:/opt/view/lib64" \
MANPATH="/opt/view/share/man" \
PKG_CONFIG_PATH="/opt/view/lib/pkgconfig:/opt/view/lib64/pkgconfig"
# Launched by the host's srun over PMIx, not by a nested mpirun. Telling
# OpenMPI this up front turns a silent single-rank-per-node run into an
# error, which is the difference between a wrong answer and no answer.
ENV OMPI_MCA_btl_vader_single_copy_mechanism=none \
PMIX_MCA_gds=hash
# A view with a dangling link in it is a working image until the moment
# something runs. Catch it here.
RUN set -eux; \
test -d /opt/view/bin; \
find /opt/view/bin -xtype l -print -quit | grep -q . && { echo "view has dangling links"; exit 1; }; \
true