Four things stood between a spack.yaml recipe and an image, and all four were invisible because nothing had ever run this path: every Spack recipe in the catalogue was still draft. The default builder image was spack/rockylinux9:v0.23.1, a tag that has never existed - Spack's images are tagged by release and Spack went to 1.x. Pinned to 1.2.2 rather than :latest, because the concretizer decides the whole catalogue's ABI and should not change under us between two applications. Spack was assumed to be on PATH. `spack env activate` changes the calling shell, so it is a function that setup-env.sh defines; running the binary prints what it would have set and changes nothing. config: install_tree took a bare path, which Spack 1.x rejects. And the MPI line could not concretize at all: +pmi and +legacylaunchers exist only up to OpenMPI 4, so asking for them with @5.0.6 collapsed the spec to 4.1.8 and then refused it. OpenMPI 5 is launched by srun through PMIx, and ~internal-pmix is what makes it use the external pmix@4.2.9 the recipes pin - without it OpenMPI bundles its own and the pin means nothing. The build cache was a dead wire at both ends. The template read from $SPACK_MIRROR and build-push.sh never passed it, and nothing ever pushed, so it could only have been empty. It is now passed by default, pointing at the same registry the images go to, and the install autopushes - during, not after, so a build that fails on its last package still leaves the half hour of OpenMPI before it. Reading needs no credential, because the registry is public by design; writing does, and it arrives as a mounted secret rather than a build argument. Arguments are recorded in the image metadata, and these images are published - a token passed that way would ship inside every application in the catalogue. osu-micro-benchmarks is the first entry that claims to run across nodes. Everything else says single-node, so the path that carries ranks between two machines had never been exercised by anything the catalogue offers. Its binaries are declared by absolute path because that is where OSU puts them: under libexec, which the view carries and nothing puts on PATH, so a bare name would be a promise the image does not keep. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Application images
What a tenant means by "I want GROMACS" is one SIF in /opt/apps, fetched
once onto the head and visible on every compute node because /opt is
already exported read-only. This directory is how those SIFs are made.
The existing image/ builds the node image and image/desktop/ builds the
interactive desktop. This is the same idea for everything else, with one
driver instead of a script per application.
The shape of an application
One directory per application, holding app.yaml and a build recipe.
An application that has one version states it at the top level:
name: stress-ng
version: "0.18.07"
binaries: [stress-ng]
published: true
One that has several lists them, and the top level holds what they share:
name: gromacs
description: Molecular dynamics.
module: gromacs
binaries: [gmx_mpi] # shared by every version
versions:
- version: "2024.4"
recipe: 2024.4 # its own subdirectory, with its own spack.yaml
published: true
status: deprecated
- version: "2025.1"
recipe: 2025.1
published: true
build_args: { CUDA_ARCH: "80,90" }
Both are read into the same shape, so nothing downstream knows which was
written. The list form exists because an update must not destroy what it
replaces: gromacs resolves to the newest published version, and a tenant
who pinned gromacs:2024.4 keeps getting 2024.4.
published.json beside the recipes is generated, not written by hand. It
records the hash of what each push actually produced, which the recipe
cannot know in advance and which the head node checks the pulled file
against. Recipes are intent; that file is measurement.
Every version carries exactly one build recipe:
| recipe | what it means | cost |
|---|---|---|
upstream: key in app.yaml |
re-publish somebody else's image as a SIF | seconds |
Containerfile |
a small image we assemble ourselves | minutes |
spack.yaml |
compiled from source by Spack against our MPI | 30–90 min |
The rule for choosing: Spack when the application has to span more than one node, upstream or a Containerfile when it does not. A multi-node MPI job is the only case where the container's MPI has to match the host's, and matching it is the whole reason to spend an hour compiling.
The MPI contract
A container that runs on one node can carry whatever MPI it likes. A
container that runs across nodes cannot: the launcher is the host's
srun, and it speaks to the process inside the container over PMI. If the
two disagree the job starts, hangs, and eventually times out — which looks
like a slow application rather than a broken one.
So every Spack-built image in here pins the same three things:
- OpenMPI 5.x, matching OpenHPC 3's
openmpi5-gnu13-ohpcmajor version. - PMIx 4.x, because that is what Slurm 25.11 on the head offers.
schedulers=slurm fabrics=ofi,ucx, sosruncan launch directly rather than through a nestedmpirun.
Jobs launch as:
module load gromacs
srun --mpi=pmix gmx_mpi mdrun -s topol.tpr
module load is real: the apps role turns the module: and binaries:
fields into an Lmod modulefile and one wrapper per binary, both under
/opt/apps, which is already exported to the compute nodes. The wrapper
decides --nv from whether the node it is running on has an NVIDIA device,
so one image serves both kinds of node - passing it where there is no driver
makes Apptainer fail outright, and omitting it where there is one leaves the
job on the CPU saying nothing at all.
The image is still there for anyone who wants it, and the module says where:
$HPCOD_GROMACS_SIF.
This has not been measured yet. The first Spack image built from here must be checked against the host before anything is published as MPI-capable:
srun --mpi=list # what the host offers
srun -N2 -n2 --mpi=pmix apptainer exec /opt/apps/gromacs.sif \
gmx_mpi -version | grep -i "MPI library" # what the container used
Two ranks on two nodes that both report the host's PMIx is the pass
condition. Until that has been seen, mark the entry mpi: single-node in
app.yaml — an application that quietly runs four copies of a one-node job
is worse than one that refuses.
Before the first build
The control plane host is where these are built, and it is not ready as it stands. Both of these were found by trying, not by reading:
-
Apptainer is not installed. It is what turns the OCI image podman builds into the SIF a head node pulls, and
build-push.shrefuses to start without it. -
Docker is installed alongside podman, and that stops podman building anything that needs the network. Docker sets
iptables -P FORWARD DROPand permits only its own bridges. netavark accepts podman's10.88.0.0/16in its own nftables table, but a DROP in another table is still a DROP. The symptom is misleading: the base image pulls fine, because podman does that over the host's own network, and then the firstRUNthat fetches anything fails with a name-resolution error that blames DNS.podman build --network=hostgets past it without touching the firewall, and is how the stress-ng image was verified. The lasting fix is an accept forpodman0in the legacy FORWARD chain, saved so it survives a reboot.
Building
./build-push.sh # everything that has changed
./build-push.sh gromacs # one application
./build-push.sh --catalog # regenerate ../../config/apps.yaml from the app.yaml files
The script pushes a SIF over ORAS and then pulls it back, because a push that printed no error has not necessarily left anything a puller can fetch — the same read-back the desktop build does for the same reason.
REGISTRY_TOKEN unset means build and verify locally without publishing.
Publishing
Three steps, and none of them is "restart the portal":
./build-push.sh stress-ng # build, verify, push, pull back
./build-push.sh --catalog # regenerate ../../config/apps.yaml
../../../scripts/C20-deploy-tree.sh # send it to /etc/hpcod/apps.yaml
The catalogue generator asks the registry whether each image is really there, so an entry only appears once it can be pulled. The portal re-reads the file when it changes, so the new application is on the create page within one request of the deploy. That is three moving parts that used to be four, and the fourth was the one nobody remembered.
Adding one
mkdir apps/<name>, writeapp.yamland one recipe../build-push.sh <name>— it verifies the binaries exist in the image before it pushes, so a missing dependency fails here and not inside a job that has already queued and started../build-push.sh --catalogand commit the regeneratedconfig/apps.yaml.
What is deliberately not here
Licensed software. VASP, Gaussian, ANSYS, ABAQUS, AMBER, MATLAB and
Materials Studio cannot be published in a shared registry — the licence is
per-site or per-seat and the image would redistribute it. These need a
different mechanism (a tenant-private image plus a licence server address),
not another directory here. See the marketplace notes in
docs/22-app-marketplace.md.