Files
ilkerandClaude Opus 5.5 8556a50e94
ci / derle (push) Failing after 34s
hpcod: DEPLOY.md, and the changelog for v0.2.0
How someone else installs hpcod: the binary, sudo hpcod setup, what it
asks and what each step does, the simulated Proxmox, a closed network
with its own CA and DNS, verify, selftest, and what to back up. The
changelog says what was proven and what was not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:10 +01:00
..

Application images

What a tenant means by "I want GROMACS" is one SIF in /opt/apps, fetched once onto the head and visible on every compute node because /opt is already exported read-only. This directory is how those SIFs are made.

The existing image/ builds the node image and image/desktop/ builds the interactive desktop. This is the same idea for everything else, with one driver instead of a script per application.

The shape of an application

One directory per application, holding app.yaml and a build recipe.

An application that has one version states it at the top level:

name: stress-ng
version: "0.18.07"
binaries: [stress-ng]
published: true

One that has several lists them, and the top level holds what they share:

name: gromacs
description: Molecular dynamics.
module: gromacs
binaries: [gmx_mpi]          # shared by every version
versions:
  - version: "2024.4"
    recipe: 2024.4           # its own subdirectory, with its own spack.yaml
    published: true
    status: deprecated
  - version: "2025.1"
    recipe: 2025.1
    published: true
    build_args: { CUDA_ARCH: "80,90" }

Both are read into the same shape, so nothing downstream knows which was written. The list form exists because an update must not destroy what it replaces: gromacs resolves to the newest published version, and a tenant who pinned gromacs:2024.4 keeps getting 2024.4.

published.json beside the recipes is generated, not written by hand. It records the hash of what each push actually produced, which the recipe cannot know in advance and which the head node checks the pulled file against. Recipes are intent; that file is measurement.

Every version carries exactly one build recipe:

recipe what it means cost
upstream: key in app.yaml re-publish somebody else's image as a SIF seconds
Containerfile a small image we assemble ourselves minutes
spack.yaml compiled from source by Spack against our MPI 30–90 min

The rule for choosing: Spack when the application has to span more than one node, upstream or a Containerfile when it does not. A multi-node MPI job is the only case where the container's MPI has to match the host's, and matching it is the whole reason to spend an hour compiling.

The MPI contract

A container that runs on one node can carry whatever MPI it likes. A container that runs across nodes cannot: the launcher is the host's srun, and it speaks to the process inside the container over PMI. If the two disagree the job starts, hangs, and eventually times out — which looks like a slow application rather than a broken one.

So every Spack-built image in here pins the same three things:

  • OpenMPI 5.x, matching OpenHPC 3's openmpi5-gnu13-ohpc major version.
  • PMIx 4.x, because that is what Slurm 25.11 on the head offers.
  • schedulers=slurm fabrics=ofi,ucx, so srun can launch directly rather than through a nested mpirun.

Jobs launch as:

module load gromacs
srun --mpi=pmix gmx_mpi mdrun -s topol.tpr

module load is real: the apps role turns the module: and binaries: fields into an Lmod modulefile and one wrapper per binary, both under /opt/apps, which is already exported to the compute nodes. The wrapper decides --nv from whether the node it is running on has an NVIDIA device, so one image serves both kinds of node - passing it where there is no driver makes Apptainer fail outright, and omitting it where there is one leaves the job on the CPU saying nothing at all.

The image is still there for anyone who wants it, and the module says where: $HPCOD_GROMACS_SIF.

This has not been measured yet. The first Spack image built from here must be checked against the host before anything is published as MPI-capable:

srun --mpi=list                                    # what the host offers
srun -N2 -n2 --mpi=pmix apptainer exec /opt/apps/gromacs.sif \
     gmx_mpi -version | grep -i "MPI library"      # what the container used

Two ranks on two nodes that both report the host's PMIx is the pass condition. Until that has been seen, mark the entry mpi: single-node in app.yaml — an application that quietly runs four copies of a one-node job is worse than one that refuses.

Before the first build

The control plane host is where these are built, and it is not ready as it stands. Both of these were found by trying, not by reading:

  • Apptainer is not installed. It is what turns the OCI image podman builds into the SIF a head node pulls, and build-push.sh refuses to start without it.

  • Docker is installed alongside podman, and that stops podman building anything that needs the network. Docker sets iptables -P FORWARD DROP and permits only its own bridges. netavark accepts podman's 10.88.0.0/16 in its own nftables table, but a DROP in another table is still a DROP. The symptom is misleading: the base image pulls fine, because podman does that over the host's own network, and then the first RUN that fetches anything fails with a name-resolution error that blames DNS.

    podman build --network=host gets past it without touching the firewall, and is how the stress-ng image was verified. The lasting fix is an accept for podman0 in the legacy FORWARD chain, saved so it survives a reboot.

Building

./build-push.sh                # everything that has changed
./build-push.sh gromacs        # one application
./build-push.sh --catalog      # regenerate ../../config/apps.yaml from the app.yaml files

The script pushes a SIF over ORAS and then pulls it back, because a push that printed no error has not necessarily left anything a puller can fetch — the same read-back the desktop build does for the same reason.

REGISTRY_TOKEN unset means build and verify locally without publishing.

Publishing

Three steps, and none of them is "restart the portal":

./build-push.sh stress-ng     # build, verify, push, pull back
./build-push.sh --catalog     # regenerate ../../config/apps.yaml
cd ../.. && make deploy                # the binary carries the catalogue;
                                      # setup installs it as /etc/hpcod/apps.yaml

A site that keeps its own catalogue instead creates /etc/hpcod/apps.local (empty is enough); setup then leaves /etc/hpcod/apps.yaml alone.

The catalogue generator asks the registry whether each image is really there, so an entry only appears once it can be pulled. The portal re-reads the file when it changes, so the new application is on the create page within one request of the deploy. That is three moving parts that used to be four, and the fourth was the one nobody remembered.

Adding one

  1. mkdir apps/<name>, write app.yaml and one recipe.
  2. ./build-push.sh <name> — it verifies the binaries exist in the image before it pushes, so a missing dependency fails here and not inside a job that has already queued and started.
  3. ./build-push.sh --catalog and commit the regenerated config/apps.yaml.

What is deliberately not here

Licensed software. VASP, Gaussian, ANSYS, ABAQUS, AMBER, MATLAB and Materials Studio cannot be published in a shared registry — the licence is per-site or per-seat and the image would redistribute it. These need a different mechanism (a tenant-private image plus a licence server address), not another directory here. See the marketplace notes in docs/22-app-marketplace.md.