ci / derle (push) Failing after 34s
How someone else installs hpcod: the binary, sudo hpcod setup, what it asks and what each step does, the simulated Proxmox, a closed network with its own CA and DNS, verify, selftest, and what to back up. The changelog says what was proven and what was not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
183 lines
7.5 KiB
Markdown
183 lines
7.5 KiB
Markdown
# Application images
|
||
|
||
What a tenant means by "I want GROMACS" is one SIF in `/opt/apps`, fetched
|
||
once onto the head and visible on every compute node because `/opt` is
|
||
already exported read-only. This directory is how those SIFs are made.
|
||
|
||
The existing `image/` builds the node image and `image/desktop/` builds the
|
||
interactive desktop. This is the same idea for everything else, with one
|
||
driver instead of a script per application.
|
||
|
||
## The shape of an application
|
||
|
||
One directory per application, holding `app.yaml` and a build recipe.
|
||
|
||
An application that has one version states it at the top level:
|
||
|
||
```yaml
|
||
name: stress-ng
|
||
version: "0.18.07"
|
||
binaries: [stress-ng]
|
||
published: true
|
||
```
|
||
|
||
One that has several lists them, and the top level holds what they share:
|
||
|
||
```yaml
|
||
name: gromacs
|
||
description: Molecular dynamics.
|
||
module: gromacs
|
||
binaries: [gmx_mpi] # shared by every version
|
||
versions:
|
||
- version: "2024.4"
|
||
recipe: 2024.4 # its own subdirectory, with its own spack.yaml
|
||
published: true
|
||
status: deprecated
|
||
- version: "2025.1"
|
||
recipe: 2025.1
|
||
published: true
|
||
build_args: { CUDA_ARCH: "80,90" }
|
||
```
|
||
|
||
Both are read into the same shape, so nothing downstream knows which was
|
||
written. The list form exists because an update must not destroy what it
|
||
replaces: `gromacs` resolves to the newest published version, and a tenant
|
||
who pinned `gromacs:2024.4` keeps getting 2024.4.
|
||
|
||
`published.json` beside the recipes is generated, not written by hand. It
|
||
records the hash of what each push actually produced, which the recipe
|
||
cannot know in advance and which the head node checks the pulled file
|
||
against. Recipes are intent; that file is measurement.
|
||
|
||
Every version carries exactly one build recipe:
|
||
|
||
| recipe | what it means | cost |
|
||
|-----------------|----------------------------------------------------|-----------|
|
||
| `upstream:` key in `app.yaml` | re-publish somebody else's image as a SIF | seconds |
|
||
| `Containerfile` | a small image we assemble ourselves | minutes |
|
||
| `spack.yaml` | compiled from source by Spack against our MPI | 30–90 min |
|
||
|
||
The rule for choosing: **Spack when the application has to span more than
|
||
one node, upstream or a Containerfile when it does not.** A multi-node MPI
|
||
job is the only case where the container's MPI has to match the host's, and
|
||
matching it is the whole reason to spend an hour compiling.
|
||
|
||
## The MPI contract
|
||
|
||
A container that runs on one node can carry whatever MPI it likes. A
|
||
container that runs across nodes cannot: the launcher is the *host's*
|
||
`srun`, and it speaks to the process inside the container over PMI. If the
|
||
two disagree the job starts, hangs, and eventually times out — which looks
|
||
like a slow application rather than a broken one.
|
||
|
||
So every Spack-built image in here pins the same three things:
|
||
|
||
- **OpenMPI 5.x**, matching OpenHPC 3's `openmpi5-gnu13-ohpc` major version.
|
||
- **PMIx 4.x**, because that is what Slurm 25.11 on the head offers.
|
||
- **`schedulers=slurm fabrics=ofi,ucx`**, so `srun` can launch directly
|
||
rather than through a nested `mpirun`.
|
||
|
||
Jobs launch as:
|
||
|
||
```bash
|
||
module load gromacs
|
||
srun --mpi=pmix gmx_mpi mdrun -s topol.tpr
|
||
```
|
||
|
||
`module load` is real: the `apps` role turns the `module:` and `binaries:`
|
||
fields into an Lmod modulefile and one wrapper per binary, both under
|
||
`/opt/apps`, which is already exported to the compute nodes. The wrapper
|
||
decides `--nv` from whether the node it is running on has an NVIDIA device,
|
||
so one image serves both kinds of node - passing it where there is no driver
|
||
makes Apptainer fail outright, and omitting it where there is one leaves the
|
||
job on the CPU saying nothing at all.
|
||
|
||
The image is still there for anyone who wants it, and the module says where:
|
||
`$HPCOD_GROMACS_SIF`.
|
||
|
||
**This has not been measured yet.** The first Spack image built from here
|
||
must be checked against the host before anything is published as
|
||
MPI-capable:
|
||
|
||
```bash
|
||
srun --mpi=list # what the host offers
|
||
srun -N2 -n2 --mpi=pmix apptainer exec /opt/apps/gromacs.sif \
|
||
gmx_mpi -version | grep -i "MPI library" # what the container used
|
||
```
|
||
|
||
Two ranks on two nodes that both report the host's PMIx is the pass
|
||
condition. Until that has been seen, mark the entry `mpi: single-node` in
|
||
`app.yaml` — an application that quietly runs four copies of a one-node job
|
||
is worse than one that refuses.
|
||
|
||
## Before the first build
|
||
|
||
The control plane host is where these are built, and it is not ready as it
|
||
stands. Both of these were found by trying, not by reading:
|
||
|
||
- **Apptainer is not installed.** It is what turns the OCI image podman
|
||
builds into the SIF a head node pulls, and `build-push.sh` refuses to
|
||
start without it.
|
||
- **Docker is installed alongside podman, and that stops podman building
|
||
anything that needs the network.** Docker sets `iptables -P FORWARD DROP`
|
||
and permits only its own bridges. netavark accepts podman's `10.88.0.0/16`
|
||
in its own nftables table, but a DROP in another table is still a DROP.
|
||
The symptom is misleading: the base image pulls fine, because podman does
|
||
that over the host's own network, and then the first `RUN` that fetches
|
||
anything fails with a name-resolution error that blames DNS.
|
||
|
||
`podman build --network=host` gets past it without touching the firewall,
|
||
and is how the stress-ng image was verified. The lasting fix is an accept
|
||
for `podman0` in the legacy FORWARD chain, saved so it survives a reboot.
|
||
|
||
## Building
|
||
|
||
```bash
|
||
./build-push.sh # everything that has changed
|
||
./build-push.sh gromacs # one application
|
||
./build-push.sh --catalog # regenerate ../../config/apps.yaml from the app.yaml files
|
||
```
|
||
|
||
The script pushes a SIF over ORAS and then pulls it back, because a push
|
||
that printed no error has not necessarily left anything a puller can fetch —
|
||
the same read-back the desktop build does for the same reason.
|
||
|
||
`REGISTRY_TOKEN` unset means build and verify locally without publishing.
|
||
|
||
## Publishing
|
||
|
||
Three steps, and none of them is "restart the portal":
|
||
|
||
```bash
|
||
./build-push.sh stress-ng # build, verify, push, pull back
|
||
./build-push.sh --catalog # regenerate ../../config/apps.yaml
|
||
cd ../.. && make deploy # the binary carries the catalogue;
|
||
# setup installs it as /etc/hpcod/apps.yaml
|
||
```
|
||
|
||
A site that keeps its own catalogue instead creates `/etc/hpcod/apps.local`
|
||
(empty is enough); setup then leaves `/etc/hpcod/apps.yaml` alone.
|
||
|
||
The catalogue generator asks the registry whether each image is really
|
||
there, so an entry only appears once it can be pulled. The portal re-reads
|
||
the file when it changes, so the new application is on the create page
|
||
within one request of the deploy. That is three moving parts that used to
|
||
be four, and the fourth was the one nobody remembered.
|
||
|
||
## Adding one
|
||
|
||
1. `mkdir apps/<name>`, write `app.yaml` and one recipe.
|
||
2. `./build-push.sh <name>` — it verifies the binaries exist in the image
|
||
before it pushes, so a missing dependency fails here and not inside a job
|
||
that has already queued and started.
|
||
3. `./build-push.sh --catalog` and commit the regenerated `config/apps.yaml`.
|
||
|
||
## What is deliberately not here
|
||
|
||
**Licensed software.** VASP, Gaussian, ANSYS, ABAQUS, AMBER, MATLAB and
|
||
Materials Studio cannot be published in a shared registry — the licence is
|
||
per-site or per-seat and the image would redistribute it. These need a
|
||
different mechanism (a tenant-private image plus a licence server address),
|
||
not another directory here. See the marketplace notes in
|
||
`docs/22-app-marketplace.md`.
|