Four things stood between a spack.yaml recipe and an image, and all four were invisible because nothing had ever run this path: every Spack recipe in the catalogue was still draft. The default builder image was spack/rockylinux9:v0.23.1, a tag that has never existed - Spack's images are tagged by release and Spack went to 1.x. Pinned to 1.2.2 rather than :latest, because the concretizer decides the whole catalogue's ABI and should not change under us between two applications. Spack was assumed to be on PATH. `spack env activate` changes the calling shell, so it is a function that setup-env.sh defines; running the binary prints what it would have set and changes nothing. config: install_tree took a bare path, which Spack 1.x rejects. And the MPI line could not concretize at all: +pmi and +legacylaunchers exist only up to OpenMPI 4, so asking for them with @5.0.6 collapsed the spec to 4.1.8 and then refused it. OpenMPI 5 is launched by srun through PMIx, and ~internal-pmix is what makes it use the external pmix@4.2.9 the recipes pin - without it OpenMPI bundles its own and the pin means nothing. The build cache was a dead wire at both ends. The template read from $SPACK_MIRROR and build-push.sh never passed it, and nothing ever pushed, so it could only have been empty. It is now passed by default, pointing at the same registry the images go to, and the install autopushes - during, not after, so a build that fails on its last package still leaves the half hour of OpenMPI before it. Reading needs no credential, because the registry is public by design; writing does, and it arrives as a mounted secret rather than a build argument. Arguments are recorded in the image metadata, and these images are published - a token passed that way would ship inside every application in the catalogue. osu-micro-benchmarks is the first entry that claims to run across nodes. Everything else says single-node, so the path that carries ranks between two machines had never been exercised by anything the catalogue offers. Its binaries are declared by absolute path because that is where OSU puts them: under libexec, which the view carries and nothing puts on PATH, so a bare name would be a promise the image does not keep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
117 lines
5.6 KiB
Docker
117 lines
5.6 KiB
Docker
# Two-stage Spack build, rendered by build-push.sh from the application's
|
|
# spack.yaml. @SPACK_IMAGE@ and @RUNTIME_IMAGE@ are substituted; everything
|
|
# else is the same for every application, which is the point - one template
|
|
# means one MPI, one compiler and one set of runtime libraries across the
|
|
# whole catalogue.
|
|
#
|
|
# Stage 1 carries a compiler toolchain and the Spack source tree, several
|
|
# gigabytes of it. Stage 2 carries the installed software and nothing that
|
|
# built it. Copying /opt/software and /opt/view together matters: the view is
|
|
# a tree of symlinks *into* the install tree, and a view without its targets
|
|
# is a directory of dangling links that looks fine until something runs.
|
|
|
|
FROM @SPACK_IMAGE@ AS builder
|
|
|
|
# The environment is copied in rather than created with `spack env create`,
|
|
# so the file in the repository is exactly the file that was concretised.
|
|
COPY spack.yaml /opt/spack-env/spack.yaml
|
|
|
|
# An OCI build cache, when one is configured. A cold build of GROMACS is
|
|
# around 40 minutes and almost all of it is dependencies that every other
|
|
# application in the catalogue also needs; with the cache populated the same
|
|
# build is a download. Unset means build from source, which is correct on
|
|
# the first run and merely slow afterwards.
|
|
# The build cache, read unauthenticated. Reading needs no credential
|
|
# because the registry is public by design, and that is what keeps this an
|
|
# ARG rather than a secret: a build argument is recorded in the image's own
|
|
# metadata, so a token passed this way ships inside every application we
|
|
# publish. Pushing does need one, and takes the secret mount below.
|
|
ARG SPACK_MIRROR=""
|
|
|
|
# Spack has to be sourced, not merely found on PATH. `spack env activate`
|
|
# changes the calling shell, so it is a shell function that setup-env.sh
|
|
# defines - running the binary directly prints the environment it would
|
|
# have set and changes nothing, and the install that follows then builds
|
|
# into the wrong place or, as here, does not run at all. Bash because that
|
|
# is what the function is written for; a RUN is /bin/sh otherwise.
|
|
SHELL ["/bin/bash", "-c"]
|
|
# Build once, reuse everywhere.
|
|
#
|
|
# Almost everything in an MPI application's dependency tree - the whole of
|
|
# OpenMPI, hwloc, libevent, pmix, ucx, libfabric - is shared with every
|
|
# other application in this catalogue. Compiled from source that is half an
|
|
# hour per application; from a cache it is a download.
|
|
#
|
|
# --autopush rather than one push at the end: a build that fails on its last
|
|
# package still leaves everything before it in the cache, which is the
|
|
# difference between losing half an hour and losing nothing.
|
|
#
|
|
# The token is a mounted secret, never an ARG. Build arguments are recorded
|
|
# in the image's own metadata, so a token passed that way would ship inside
|
|
# every application published from this template. Reading needs no
|
|
# credential at all, because the registry is public by design - so a build
|
|
# with no secret is slower and not broken, and says which it is.
|
|
RUN --mount=type=cache,target=/root/.spack/cache \
|
|
--mount=type=secret,id=mirror_token \
|
|
set -eux; \
|
|
. /opt/spack/share/spack/setup-env.sh; \
|
|
cd /opt/spack-env; \
|
|
if [ -n "$SPACK_MIRROR" ]; then \
|
|
spack mirror add --unsigned hpcod "$SPACK_MIRROR"; \
|
|
if [ -s /run/secrets/mirror_token ]; then \
|
|
spack mirror set --push --autopush \
|
|
--oci-username-variable SPACK_OCI_USER \
|
|
--oci-password-variable SPACK_OCI_TOKEN hpcod; \
|
|
export SPACK_OCI_USER="${SPACK_MIRROR_USER:-hpcod}"; \
|
|
export SPACK_OCI_TOKEN="$(cat /run/secrets/mirror_token)"; \
|
|
echo "build cache: reading and writing $SPACK_MIRROR"; \
|
|
else \
|
|
echo "build cache: reading $SPACK_MIRROR, no credential to write it"; \
|
|
fi; \
|
|
spack buildcache update-index hpcod || true; \
|
|
fi; \
|
|
spack env activate .; \
|
|
spack concretize --force; \
|
|
spack install --fail-fast --no-check-signature; \
|
|
spack gc -y; \
|
|
spack env deactivate
|
|
|
|
# Strip what a runtime image has no use for. Spack keeps the full source and
|
|
# build logs of every package it installed, and on a stack the size of an MPI
|
|
# application that is the larger half of the image.
|
|
RUN set -eux; \
|
|
find /opt/software -name '*.a' -delete; \
|
|
rm -rf /opt/software/*/*/*/.spack/archive /opt/spack-env/.spack-env/._view 2>/dev/null || true; \
|
|
strip -s /opt/view/bin/* 2>/dev/null || true
|
|
|
|
FROM @RUNTIME_IMAGE@
|
|
|
|
# The runtime libraries Spack-built binaries link against from the system
|
|
# rather than from the view: libstdc++ in particular, which has to be the
|
|
# same one the compiler in stage 1 used. Same base family, so it is.
|
|
RUN dnf install -y -q --nodocs --setopt=install_weak_deps=False \
|
|
libgomp libstdc++ libatomic numactl-libs hwloc-libs \
|
|
libibverbs librdmacm rdma-core \
|
|
&& dnf clean all -y && rm -rf /var/cache/dnf /usr/share/doc /usr/share/man
|
|
|
|
COPY --from=builder /opt/software /opt/software
|
|
COPY --from=builder /opt/view /opt/view
|
|
|
|
ENV PATH="/opt/view/bin:${PATH}" \
|
|
LD_LIBRARY_PATH="/opt/view/lib:/opt/view/lib64" \
|
|
MANPATH="/opt/view/share/man" \
|
|
PKG_CONFIG_PATH="/opt/view/lib/pkgconfig:/opt/view/lib64/pkgconfig"
|
|
|
|
# Launched by the host's srun over PMIx, not by a nested mpirun. Telling
|
|
# OpenMPI this up front turns a silent single-rank-per-node run into an
|
|
# error, which is the difference between a wrong answer and no answer.
|
|
ENV OMPI_MCA_btl_vader_single_copy_mechanism=none \
|
|
PMIX_MCA_gds=hash
|
|
|
|
# A view with a dangling link in it is a working image until the moment
|
|
# something runs. Catch it here.
|
|
RUN set -eux; \
|
|
test -d /opt/view/bin; \
|
|
find /opt/view/bin -xtype l -print -quit | grep -q . && { echo "view has dangling links"; exit 1; }; \
|
|
true
|