Files
werewulf/docs/22-spack-chapar-integration.md
T
ilkerandClaude Opus 5.5 861aceb283 docs: bring sections 01-23 in line with hpcod setup and what is built
- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for
  installing the control plane.
- 09: repository layout now includes hpcod/, docs 20-23 and the B/C
  scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is
  recorded, not absent (matches 6.6). Handbook PDF regenerated.
- 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run
  from the control plane; Terraform state per cluster, not in Postgres;
  no reaper yet; usage per project.
- 21: c<NNN> portals, publishing wired into provisioning, closed-network
  mode.
- 22: three published images; chosen apps now reach the cluster.
- 23: setup installs the catalogue; apps.local keeps a site's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:50:33 +01:00

10 KiB

22 — Spack, and what Chapar would give us

Written for whoever maintains hpcod/ next. It assumes §20 and §21, and it assumes nothing about Spack.

The application catalogue currently publishes three images: busybox, osu-micro-benchmarks and stress-ng. Filling it is not a matter of writing more Containerfiles — it is a question of who compiles the software, against which MPI, and how a second cluster gets it without paying for the first cluster's build again. Chapar answers all three, and it answers them from production rather than from a design document, which is the reason this section exists.

What Chapar is, in our terms

A site's Spack policy, kept outside Spack. One envs/<name>/spack.yaml holds the root specs; etc/system and etc/user hold the compiler, libc, mirror and concretiser policy that every environment shares. From that it builds a release: an install tree plus a tree of Lmod modulefiles named {name}/{version} with no hashes in them.

Four things in it are worth more than the package list:

  • Releases are atomic. A build goes into <os>/.<release-id>.staging.<pid> and is renamed into place only after install and module generation succeed. Publication is a symlink flip. Rollback is the same flip backwards. Nothing overwrites a tree a running job is reading from.
  • The buildcache is the point. Concretised specs that already exist as binaries are downloaded, not rebuilt. Source-built packages are pushed as they complete, so even a failed build leaves the next one faster.
  • The MPI runtime policy lives in the generated module, not in a wiki. Chapar's openmpi module picks UCX transports from the node it is loaded on, so one build serves CPU and GPU nodes; its Intel MPI module exposes a CUDA driver stub so a CPU-only node can start a CUDA-aware binary instead of dying on a missing libcuda.
  • It already made the PMIx decision. openmpi@5 ~internal-pmix ^pmix@6 schedulers=slurm — external PMIx, not the bundled one, so the host's srun can launch ranks directly.

That last line is the one we would otherwise have had to discover by watching a two-node job hang.

Where it does not fit, stated plainly

Chapar's container is a thin runtime wrapper. Its own README is explicit: "It does not embed the Spack install tree. At runtime, bind the site resources tree so the container sees the promoted release." The software lives at a stable absolute path — /resources/chapar/hpcsim/rocky9/current — on a filesystem that outlives every job.

Our containers are the opposite. A SIF in /opt/apps is self-contained and is pulled fresh onto a head node that will be destroyed in fourteen days. There is no site filesystem. /opt is exported by each cluster's own head, which is the whole reason a tenant's software is a tenant's business.

So Chapar cannot simply be switched on. Its buildcache mirror is also a file:// path, which assumes the same shared filesystem; we have a registry and no shared filesystem at all.

This is a mismatch about lifetime, not about correctness — and once it is named that way the fit becomes obvious, because we have three lifetimes and have been trying to serve them with one mechanism:

lives for where built by
the software stack months the site Chapar, once per release
an application image months the registry image/apps/build-push.sh
a cluster fourteen days Proxmox hpcod, per tenant

Three levels, and which one to take

Level 1 — Chapar as the image factory

Nothing about our runtime changes. A SIF is still fat, still pulled per cluster, still lands in /opt/apps. What changes is who compiles it.

Today's sketch in image/apps/ gives each application its own spack.yaml, which means each application concretises independently. Two images built a month apart get different OpenMPI patch levels and nobody finds out until somebody tries to run them in the same job script. Instead:

  • one Chapar environment, envs/hpcod/spack.yaml, whose root specs are the application catalogue;
  • one concretisation, therefore one compiler and one MPI across every image we publish;
  • an OCI buildcache in Gitea rather than Chapar's file:// mirror — Spack has supported oci:// mirrors since 0.21, and we already run a registry, which is exactly the shared store we do not otherwise have;
  • per-application images that install only their own spec with --use-buildcache only, turning a forty-minute GROMACS build into a download.

Cost: a fork or a vendored checkout of Chapar, and one full cold build to seed the cache. Everything after that is cheap.

Level 2 — the release tree mounted into every cluster

Build one Chapar release for the site and let every tenant cluster read it. Users get module avail over the whole stack — several hundred packages, benchmarks and profilers included — with no per-cluster pull cost at all.

The topology objection answers itself: the head already exports /opt read-only to its compute nodes. Mount the site tree on the head at /opt/chapar and the nodes see it through an export that already exists. No new route across vmbr6, no node-image change beyond dropping in a profile.d hook of the kind Chapar already ships.

How the head gets it is the real choice, and NFS from a long-lived host is the obvious answer rather than the best one. A read-only disk attached by Terraform is a better fit for what we actually run: Proxmox can clone a volume instantly, the clone costs nothing until written, the cluster gains no dependency on a host outside itself, and a release is promoted by attaching a different volume. We already create disks in the tenant module.

Level 2 subsumes Level 1 — the SIFs remain the right answer for anything with an awkward licence, a vendor binary, or a Python stack users expect to pip install into.

Level 3 — the operational machinery

Independent of the above, and all of it is work we would otherwise write:

  • ci/cve-checker.py and its systemd timer. We publish images to tenants; a marketplace that cannot say when it is shipping a known vulnerability is not one an institution can adopt.
  • validation/ — osu-micro-benchmarks, ior, mdtest, hpl, stream, and MPI+RDMA programs including CUDA. This is a better post-provision acceptance suite than roles/verify and 70-mpi-test.sh, and it is the thing that would tell a tenant their fourteen-day cluster is actually healthy.
  • The staging-then-rename discipline applies to our catalogue as much as to a module tree.

Recommendation

Level 1 now, Level 3 opportunistically, Level 2 when a second site exists.

Level 2's payoff scales with how many clusters read one tree, and we have one Proxmox node. Level 1's payoff is immediate and its blast radius is a build host.

Measure this before anything else

Chapar's hpcsim pins PMIx 6. Our head runs OpenHPC 3 with Slurm 25.11, whose PMIx is its own. PMIx keeps cross-version compatibility, but not without limit, and the failure mode is not an error: the job starts, the ranks never rendezvous, and it sits there until the wall clock kills it. That reads as a slow application, not a broken stack.

So, on a real head, before a single image is built against Chapar's MPI:

srun --mpi=list                 # what our Slurm offers
rpm -q pmix pmix-libs           # what version it is built against

If those do not line up with ^pmix@6, pin Chapar's PMIx to our Slurm's major version rather than the other way round — or build Slurm through Chapar as well, which is already on its own TODO and which would retire the question entirely by making both sides come out of one build.

Two further things to check on the same visit, both cheap and both capable of silently wasting a build:

  • Microarchitecture. Chapar concretises with targets: granularity: generic, so it produces x86_64_v2/v3/v4 rather than zen5. Good for us — but our Proxmox guests must actually advertise the level the cache was built for, and x86_64_v4 needs AVX-512 that a default VM CPU model does not expose. A mismatch does not fail; it misses the cache and rebuilds everything from source.
  • CUDA. hpcsim builds cuda@13 with a cuda_arch list of thirteen architectures. Our GPU story is whole-device passthrough on hardware we should confirm is in that list before paying for the build.

Open questions

  • Licence. The repository carries no LICENSE file — the only one in the tree belongs to a bundled skill. Vendoring it, forking it, or building our published images with it all need a licence statement from the author first. This is the one item that blocks everything else and it is a question, not a task.
  • Ownership. A fork we patch is a fork we maintain. Chapar's own TODO notes that its two release.sh copies have already diverged; a third environment of ours would be a fourth place for that to happen. Upstreaming an envs/hpcod is the cheaper posture if the author is willing.
  • Our catalogue's shape. Chapar's unit is a module in a release. Ours is a named application with hardware requirements attached, because placement reads it (internal/catalog). A Chapar release does not carry "this needs a GPU" — that stays ours, and something has to map one onto the other.

What is already written

hpcod/image/apps/ holds the driver, the catalogue generator, and the per-application metadata. Under Level 1 the app.yaml files, build-push.sh and the upstream: entries all stand. The per-application spack.yaml files and spack.Containerfile.in are the part Chapar replaces, and should not be extended before this decision is made.

Separately: what a tenant ticks now reaches their cluster. It is recorded in cluster_app at allocation, provision.TFVars emits it as the apps variable, and the apps role pulls each image onto the head at first boot. When this section was first written none of that was connected, and every cluster got the module's apps = [] default whatever the catalogue resolved.