- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for installing the control plane. - 09: repository layout now includes hpcod/, docs 20-23 and the B/C scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is recorded, not absent (matches 6.6). Handbook PDF regenerated. - 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run from the control plane; Terraform state per cluster, not in Postgres; no reaper yet; usage per project. - 21: c<NNN> portals, publishing wired into provisioning, closed-network mode. - 22: three published images; chosen apps now reach the cluster. - 23: setup installs the catalogue; apps.local keeps a site's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10 KiB
22 — Spack, and what Chapar would give us
Written for whoever maintains hpcod/ next. It assumes §20 and §21, and it
assumes nothing about Spack.
The application catalogue currently publishes three images: busybox, osu-micro-benchmarks and stress-ng. Filling it is not a matter of writing more Containerfiles — it is a question of who compiles the software, against which MPI, and how a second cluster gets it without paying for the first cluster's build again. Chapar answers all three, and it answers them from production rather than from a design document, which is the reason this section exists.
What Chapar is, in our terms
A site's Spack policy, kept outside Spack. One envs/<name>/spack.yaml holds
the root specs; etc/system and etc/user hold the compiler, libc, mirror
and concretiser policy that every environment shares. From that it builds a
release: an install tree plus a tree of Lmod modulefiles named
{name}/{version} with no hashes in them.
Four things in it are worth more than the package list:
- Releases are atomic. A build goes into
<os>/.<release-id>.staging.<pid>and is renamed into place only after install and module generation succeed. Publication is a symlink flip. Rollback is the same flip backwards. Nothing overwrites a tree a running job is reading from. - The buildcache is the point. Concretised specs that already exist as binaries are downloaded, not rebuilt. Source-built packages are pushed as they complete, so even a failed build leaves the next one faster.
- The MPI runtime policy lives in the generated module, not in a wiki.
Chapar's
openmpimodule picks UCX transports from the node it is loaded on, so one build serves CPU and GPU nodes; its Intel MPI module exposes a CUDA driver stub so a CPU-only node can start a CUDA-aware binary instead of dying on a missinglibcuda. - It already made the PMIx decision.
openmpi@5 ~internal-pmix ^pmix@6 schedulers=slurm— external PMIx, not the bundled one, so the host'ssruncan launch ranks directly.
That last line is the one we would otherwise have had to discover by watching a two-node job hang.
Where it does not fit, stated plainly
Chapar's container is a thin runtime wrapper. Its own README is explicit:
"It does not embed the Spack install tree. At runtime, bind the site
resources tree so the container sees the promoted release." The software
lives at a stable absolute path — /resources/chapar/hpcsim/rocky9/current —
on a filesystem that outlives every job.
Our containers are the opposite. A SIF in /opt/apps is self-contained and
is pulled fresh onto a head node that will be destroyed in fourteen days.
There is no site filesystem. /opt is exported by each cluster's own head,
which is the whole reason a tenant's software is a tenant's business.
So Chapar cannot simply be switched on. Its buildcache mirror is also a
file:// path, which assumes the same shared filesystem; we have a registry
and no shared filesystem at all.
This is a mismatch about lifetime, not about correctness — and once it is named that way the fit becomes obvious, because we have three lifetimes and have been trying to serve them with one mechanism:
| lives for | where | built by | |
|---|---|---|---|
| the software stack | months | the site | Chapar, once per release |
| an application image | months | the registry | image/apps/build-push.sh |
| a cluster | fourteen days | Proxmox | hpcod, per tenant |
Three levels, and which one to take
Level 1 — Chapar as the image factory
Nothing about our runtime changes. A SIF is still fat, still pulled per
cluster, still lands in /opt/apps. What changes is who compiles it.
Today's sketch in image/apps/ gives each application its own spack.yaml,
which means each application concretises independently. Two images built a
month apart get different OpenMPI patch levels and nobody finds out until
somebody tries to run them in the same job script. Instead:
- one Chapar environment,
envs/hpcod/spack.yaml, whose root specs are the application catalogue; - one concretisation, therefore one compiler and one MPI across every image we publish;
- an OCI buildcache in Gitea rather than Chapar's
file://mirror — Spack has supportedoci://mirrors since 0.21, and we already run a registry, which is exactly the shared store we do not otherwise have; - per-application images that install only their own spec with
--use-buildcache only, turning a forty-minute GROMACS build into a download.
Cost: a fork or a vendored checkout of Chapar, and one full cold build to seed the cache. Everything after that is cheap.
Level 2 — the release tree mounted into every cluster
Build one Chapar release for the site and let every tenant cluster read it.
Users get module avail over the whole stack — several hundred packages,
benchmarks and profilers included — with no per-cluster pull cost at all.
The topology objection answers itself: the head already exports /opt
read-only to its compute nodes. Mount the site tree on the head at
/opt/chapar and the nodes see it through an export that already exists.
No new route across vmbr6, no node-image change beyond dropping in a
profile.d hook of the kind Chapar already ships.
How the head gets it is the real choice, and NFS from a long-lived host is the obvious answer rather than the best one. A read-only disk attached by Terraform is a better fit for what we actually run: Proxmox can clone a volume instantly, the clone costs nothing until written, the cluster gains no dependency on a host outside itself, and a release is promoted by attaching a different volume. We already create disks in the tenant module.
Level 2 subsumes Level 1 — the SIFs remain the right answer for anything with
an awkward licence, a vendor binary, or a Python stack users expect to pip install into.
Level 3 — the operational machinery
Independent of the above, and all of it is work we would otherwise write:
ci/cve-checker.pyand its systemd timer. We publish images to tenants; a marketplace that cannot say when it is shipping a known vulnerability is not one an institution can adopt.validation/—osu-micro-benchmarks,ior,mdtest,hpl,stream, and MPI+RDMA programs including CUDA. This is a better post-provision acceptance suite thanroles/verifyand70-mpi-test.sh, and it is the thing that would tell a tenant their fourteen-day cluster is actually healthy.- The staging-then-rename discipline applies to our catalogue as much as to a module tree.
Recommendation
Level 1 now, Level 3 opportunistically, Level 2 when a second site exists.
Level 2's payoff scales with how many clusters read one tree, and we have one Proxmox node. Level 1's payoff is immediate and its blast radius is a build host.
Measure this before anything else
Chapar's hpcsim pins PMIx 6. Our head runs OpenHPC 3 with Slurm 25.11, whose PMIx is its own. PMIx keeps cross-version compatibility, but not without limit, and the failure mode is not an error: the job starts, the ranks never rendezvous, and it sits there until the wall clock kills it. That reads as a slow application, not a broken stack.
So, on a real head, before a single image is built against Chapar's MPI:
srun --mpi=list # what our Slurm offers
rpm -q pmix pmix-libs # what version it is built against
If those do not line up with ^pmix@6, pin Chapar's PMIx to our Slurm's
major version rather than the other way round — or build Slurm through
Chapar as well, which is already on its own TODO and which would retire the
question entirely by making both sides come out of one build.
Two further things to check on the same visit, both cheap and both capable of silently wasting a build:
- Microarchitecture. Chapar concretises with
targets: granularity: generic, so it producesx86_64_v2/v3/v4rather thanzen5. Good for us — but our Proxmox guests must actually advertise the level the cache was built for, andx86_64_v4needs AVX-512 that a default VM CPU model does not expose. A mismatch does not fail; it misses the cache and rebuilds everything from source. - CUDA. hpcsim builds
cuda@13with acuda_archlist of thirteen architectures. Our GPU story is whole-device passthrough on hardware we should confirm is in that list before paying for the build.
Open questions
- Licence. The repository carries no
LICENSEfile — the only one in the tree belongs to a bundled skill. Vendoring it, forking it, or building our published images with it all need a licence statement from the author first. This is the one item that blocks everything else and it is a question, not a task. - Ownership. A fork we patch is a fork we maintain. Chapar's own TODO
notes that its two
release.shcopies have already diverged; a third environment of ours would be a fourth place for that to happen. Upstreaming anenvs/hpcodis the cheaper posture if the author is willing. - Our catalogue's shape. Chapar's unit is a module in a release. Ours is
a named application with hardware requirements attached, because placement
reads it (
internal/catalog). A Chapar release does not carry "this needs a GPU" — that stays ours, and something has to map one onto the other.
What is already written
hpcod/image/apps/ holds the driver, the catalogue generator, and the
per-application metadata. Under Level 1 the app.yaml files, build-push.sh
and the upstream: entries all stand. The per-application spack.yaml files
and spack.Containerfile.in are the part Chapar replaces, and should not be
extended before this decision is made.
Separately: what a tenant ticks now reaches their cluster. It is recorded
in cluster_app at allocation, provision.TFVars emits it as the apps
variable, and the apps role pulls each image onto the head at first boot.
When this section was first written none of that was connected, and every
cluster got the module's apps = [] default whatever the catalogue
resolved.