Files
ilkerandClaude Opus 5.5 8556a50e94
ci / derle (push) Failing after 34s
hpcod: DEPLOY.md, and the changelog for v0.2.0
How someone else installs hpcod: the binary, sudo hpcod setup, what it
asks and what each step does, the simulated Proxmox, a closed network
with its own CA and DNS, verify, selftest, and what to back up. The
changelog says what was proven and what was not.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:20:10 +01:00

183 lines
7.5 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Application images
What a tenant means by "I want GROMACS" is one SIF in `/opt/apps`, fetched
once onto the head and visible on every compute node because `/opt` is
already exported read-only. This directory is how those SIFs are made.
The existing `image/` builds the node image and `image/desktop/` builds the
interactive desktop. This is the same idea for everything else, with one
driver instead of a script per application.
## The shape of an application
One directory per application, holding `app.yaml` and a build recipe.
An application that has one version states it at the top level:
```yaml
name: stress-ng
version: "0.18.07"
binaries: [stress-ng]
published: true
```
One that has several lists them, and the top level holds what they share:
```yaml
name: gromacs
description: Molecular dynamics.
module: gromacs
binaries: [gmx_mpi] # shared by every version
versions:
- version: "2024.4"
recipe: 2024.4 # its own subdirectory, with its own spack.yaml
published: true
status: deprecated
- version: "2025.1"
recipe: 2025.1
published: true
build_args: { CUDA_ARCH: "80,90" }
```
Both are read into the same shape, so nothing downstream knows which was
written. The list form exists because an update must not destroy what it
replaces: `gromacs` resolves to the newest published version, and a tenant
who pinned `gromacs:2024.4` keeps getting 2024.4.
`published.json` beside the recipes is generated, not written by hand. It
records the hash of what each push actually produced, which the recipe
cannot know in advance and which the head node checks the pulled file
against. Recipes are intent; that file is measurement.
Every version carries exactly one build recipe:
| recipe | what it means | cost |
|-----------------|----------------------------------------------------|-----------|
| `upstream:` key in `app.yaml` | re-publish somebody else's image as a SIF | seconds |
| `Containerfile` | a small image we assemble ourselves | minutes |
| `spack.yaml` | compiled from source by Spack against our MPI | 30–90 min |
The rule for choosing: **Spack when the application has to span more than
one node, upstream or a Containerfile when it does not.** A multi-node MPI
job is the only case where the container's MPI has to match the host's, and
matching it is the whole reason to spend an hour compiling.
## The MPI contract
A container that runs on one node can carry whatever MPI it likes. A
container that runs across nodes cannot: the launcher is the *host's*
`srun`, and it speaks to the process inside the container over PMI. If the
two disagree the job starts, hangs, and eventually times out — which looks
like a slow application rather than a broken one.
So every Spack-built image in here pins the same three things:
- **OpenMPI 5.x**, matching OpenHPC 3's `openmpi5-gnu13-ohpc` major version.
- **PMIx 4.x**, because that is what Slurm 25.11 on the head offers.
- **`schedulers=slurm fabrics=ofi,ucx`**, so `srun` can launch directly
rather than through a nested `mpirun`.
Jobs launch as:
```bash
module load gromacs
srun --mpi=pmix gmx_mpi mdrun -s topol.tpr
```
`module load` is real: the `apps` role turns the `module:` and `binaries:`
fields into an Lmod modulefile and one wrapper per binary, both under
`/opt/apps`, which is already exported to the compute nodes. The wrapper
decides `--nv` from whether the node it is running on has an NVIDIA device,
so one image serves both kinds of node - passing it where there is no driver
makes Apptainer fail outright, and omitting it where there is one leaves the
job on the CPU saying nothing at all.
The image is still there for anyone who wants it, and the module says where:
`$HPCOD_GROMACS_SIF`.
**This has not been measured yet.** The first Spack image built from here
must be checked against the host before anything is published as
MPI-capable:
```bash
srun --mpi=list # what the host offers
srun -N2 -n2 --mpi=pmix apptainer exec /opt/apps/gromacs.sif \
gmx_mpi -version | grep -i "MPI library" # what the container used
```
Two ranks on two nodes that both report the host's PMIx is the pass
condition. Until that has been seen, mark the entry `mpi: single-node` in
`app.yaml` — an application that quietly runs four copies of a one-node job
is worse than one that refuses.
## Before the first build
The control plane host is where these are built, and it is not ready as it
stands. Both of these were found by trying, not by reading:
- **Apptainer is not installed.** It is what turns the OCI image podman
builds into the SIF a head node pulls, and `build-push.sh` refuses to
start without it.
- **Docker is installed alongside podman, and that stops podman building
anything that needs the network.** Docker sets `iptables -P FORWARD DROP`
and permits only its own bridges. netavark accepts podman's `10.88.0.0/16`
in its own nftables table, but a DROP in another table is still a DROP.
The symptom is misleading: the base image pulls fine, because podman does
that over the host's own network, and then the first `RUN` that fetches
anything fails with a name-resolution error that blames DNS.
`podman build --network=host` gets past it without touching the firewall,
and is how the stress-ng image was verified. The lasting fix is an accept
for `podman0` in the legacy FORWARD chain, saved so it survives a reboot.
## Building
```bash
./build-push.sh # everything that has changed
./build-push.sh gromacs # one application
./build-push.sh --catalog # regenerate ../../config/apps.yaml from the app.yaml files
```
The script pushes a SIF over ORAS and then pulls it back, because a push
that printed no error has not necessarily left anything a puller can fetch —
the same read-back the desktop build does for the same reason.
`REGISTRY_TOKEN` unset means build and verify locally without publishing.
## Publishing
Three steps, and none of them is "restart the portal":
```bash
./build-push.sh stress-ng # build, verify, push, pull back
./build-push.sh --catalog # regenerate ../../config/apps.yaml
cd ../.. && make deploy # the binary carries the catalogue;
# setup installs it as /etc/hpcod/apps.yaml
```
A site that keeps its own catalogue instead creates `/etc/hpcod/apps.local`
(empty is enough); setup then leaves `/etc/hpcod/apps.yaml` alone.
The catalogue generator asks the registry whether each image is really
there, so an entry only appears once it can be pulled. The portal re-reads
the file when it changes, so the new application is on the create page
within one request of the deploy. That is three moving parts that used to
be four, and the fourth was the one nobody remembered.
## Adding one
1. `mkdir apps/<name>`, write `app.yaml` and one recipe.
2. `./build-push.sh <name>` — it verifies the binaries exist in the image
before it pushes, so a missing dependency fails here and not inside a job
that has already queued and started.
3. `./build-push.sh --catalog` and commit the regenerated `config/apps.yaml`.
## What is deliberately not here
**Licensed software.** VASP, Gaussian, ANSYS, ABAQUS, AMBER, MATLAB and
Materials Studio cannot be published in a shared registry — the licence is
per-site or per-seat and the image would redistribute it. These need a
different mechanism (a tenant-private image plus a licence server address),
not another directory here. See the marketplace notes in
`docs/22-app-marketplace.md`.