Files
werewulf/docs/22-spack-chapar-integration.md
ilkerandClaude Opus 5.5 861aceb283 docs: bring sections 01-23 in line with hpcod setup and what is built
- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for
  installing the control plane.
- 09: repository layout now includes hpcod/, docs 20-23 and the B/C
  scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is
  recorded, not absent (matches 6.6). Handbook PDF regenerated.
- 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run
  from the control plane; Terraform state per cluster, not in Postgres;
  no reaper yet; usage per project.
- 21: c<NNN> portals, publishing wired into provisioning, closed-network
  mode.
- 22: three published images; chosen apps now reach the cluster.
- 23: setup installs the catalogue; apps.local keeps a site's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:50:33 +01:00

207 lines
10 KiB
Markdown

# 22 — Spack, and what Chapar would give us
Written for whoever maintains `hpcod/` next. It assumes §20 and §21, and it
assumes nothing about Spack.
The application catalogue currently publishes three images: busybox,
osu-micro-benchmarks and stress-ng. Filling it is not a matter of writing
more Containerfiles — it is a question of who
compiles the software, against which MPI, and how a second cluster gets it
without paying for the first cluster's build again. [Chapar][chapar] answers
all three, and it answers them from production rather than from a design
document, which is the reason this section exists.
[chapar]: https://github.com/mmrezaie/chapar
## What Chapar is, in our terms
A site's Spack policy, kept outside Spack. One `envs/<name>/spack.yaml` holds
the root specs; `etc/system` and `etc/user` hold the compiler, libc, mirror
and concretiser policy that every environment shares. From that it builds a
**release**: an install tree plus a tree of Lmod modulefiles named
`{name}/{version}` with no hashes in them.
Four things in it are worth more than the package list:
- **Releases are atomic.** A build goes into `<os>/.<release-id>.staging.<pid>`
and is renamed into place only after install *and* module generation
succeed. Publication is a symlink flip. Rollback is the same flip backwards.
Nothing overwrites a tree a running job is reading from.
- **The buildcache is the point.** Concretised specs that already exist as
binaries are downloaded, not rebuilt. Source-built packages are pushed as
they complete, so even a failed build leaves the next one faster.
- **The MPI runtime policy lives in the generated module,** not in a wiki.
Chapar's `openmpi` module picks UCX transports from the node it is loaded
on, so one build serves CPU and GPU nodes; its Intel MPI module exposes a
CUDA driver stub so a CPU-only node can start a CUDA-aware binary instead
of dying on a missing `libcuda`.
- **It already made the PMIx decision.** `openmpi@5 ~internal-pmix ^pmix@6
schedulers=slurm` — external PMIx, not the bundled one, so the host's
`srun` can launch ranks directly.
That last line is the one we would otherwise have had to discover by watching
a two-node job hang.
## Where it does not fit, stated plainly
Chapar's container is a **thin runtime wrapper**. Its own README is explicit:
"It does not embed the Spack install tree. At runtime, bind the site
resources tree so the container sees the promoted release." The software
lives at a stable absolute path — `/resources/chapar/hpcsim/rocky9/current` —
on a filesystem that outlives every job.
Our containers are the opposite. A SIF in `/opt/apps` is self-contained and
is pulled fresh onto a head node that will be destroyed in fourteen days.
There is no site filesystem. `/opt` is exported by *each cluster's own head*,
which is the whole reason a tenant's software is a tenant's business.
So Chapar cannot simply be switched on. Its buildcache mirror is also a
`file://` path, which assumes the same shared filesystem; we have a registry
and no shared filesystem at all.
This is a mismatch about **lifetime**, not about correctness — and once it is
named that way the fit becomes obvious, because we have three lifetimes and
have been trying to serve them with one mechanism:
| | lives for | where | built by |
|---|---|---|---|
| the software stack | months | the site | Chapar, once per release |
| an application image | months | the registry | `image/apps/build-push.sh` |
| a cluster | fourteen days | Proxmox | hpcod, per tenant |
## Three levels, and which one to take
### Level 1 — Chapar as the image factory
Nothing about our runtime changes. A SIF is still fat, still pulled per
cluster, still lands in `/opt/apps`. What changes is who compiles it.
Today's sketch in `image/apps/` gives each application its own `spack.yaml`,
which means each application concretises independently. Two images built a
month apart get different OpenMPI patch levels and nobody finds out until
somebody tries to run them in the same job script. Instead:
- one Chapar environment, `envs/hpcod/spack.yaml`, whose root specs **are**
the application catalogue;
- one concretisation, therefore one compiler and one MPI across every image
we publish;
- an **OCI buildcache in Gitea** rather than Chapar's `file://` mirror —
Spack has supported `oci://` mirrors since 0.21, and we already run a
registry, which is exactly the shared store we do not otherwise have;
- per-application images that install only their own spec with
`--use-buildcache only`, turning a forty-minute GROMACS build into a
download.
Cost: a fork or a vendored checkout of Chapar, and one full cold build to
seed the cache. Everything after that is cheap.
### Level 2 — the release tree mounted into every cluster
Build one Chapar release for the site and let every tenant cluster read it.
Users get `module avail` over the whole stack — several hundred packages,
benchmarks and profilers included — with no per-cluster pull cost at all.
The topology objection answers itself: **the head already exports `/opt`
read-only to its compute nodes.** Mount the site tree on the head at
`/opt/chapar` and the nodes see it through an export that already exists.
No new route across `vmbr6`, no node-image change beyond dropping in a
`profile.d` hook of the kind Chapar already ships.
How the head gets it is the real choice, and NFS from a long-lived host is
the obvious answer rather than the best one. A **read-only disk attached by
Terraform** is a better fit for what we actually run: Proxmox can clone a
volume instantly, the clone costs nothing until written, the cluster gains no
dependency on a host outside itself, and a release is promoted by attaching a
different volume. We already create disks in the tenant module.
Level 2 subsumes Level 1 — the SIFs remain the right answer for anything with
an awkward licence, a vendor binary, or a Python stack users expect to `pip
install` into.
### Level 3 — the operational machinery
Independent of the above, and all of it is work we would otherwise write:
- **`ci/cve-checker.py`** and its systemd timer. We publish images to
tenants; a marketplace that cannot say when it is shipping a known
vulnerability is not one an institution can adopt.
- **`validation/`** — `osu-micro-benchmarks`, `ior`, `mdtest`, `hpl`,
`stream`, and MPI+RDMA programs including CUDA. This is a better
post-provision acceptance suite than `roles/verify` and `70-mpi-test.sh`,
and it is the thing that would tell a tenant their fourteen-day cluster is
actually healthy.
- **The staging-then-rename discipline** applies to our catalogue as much as
to a module tree.
## Recommendation
Level 1 now, Level 3 opportunistically, Level 2 when a second site exists.
Level 2's payoff scales with how many clusters read one tree, and we have one
Proxmox node. Level 1's payoff is immediate and its blast radius is a build
host.
## Measure this before anything else
Chapar's hpcsim pins **PMIx 6**. Our head runs OpenHPC 3 with Slurm 25.11,
whose PMIx is its own. PMIx keeps cross-version compatibility, but not
without limit, and the failure mode is not an error: the job starts, the
ranks never rendezvous, and it sits there until the wall clock kills it.
That reads as a slow application, not a broken stack.
So, on a real head, before a single image is built against Chapar's MPI:
```bash
srun --mpi=list # what our Slurm offers
rpm -q pmix pmix-libs # what version it is built against
```
If those do not line up with `^pmix@6`, pin Chapar's PMIx to our Slurm's
major version rather than the other way round — or build Slurm through
Chapar as well, which is already on its own TODO and which would retire the
question entirely by making both sides come out of one build.
Two further things to check on the same visit, both cheap and both capable of
silently wasting a build:
- **Microarchitecture.** Chapar concretises with `targets: granularity:
generic`, so it produces `x86_64_v2/v3/v4` rather than `zen5`. Good for us
— but our Proxmox guests must actually advertise the level the cache was
built for, and `x86_64_v4` needs AVX-512 that a default VM CPU model does
not expose. A mismatch does not fail; it misses the cache and rebuilds
everything from source.
- **CUDA.** hpcsim builds `cuda@13` with a `cuda_arch` list of thirteen
architectures. Our GPU story is whole-device passthrough on hardware we
should confirm is in that list before paying for the build.
## Open questions
- **Licence.** The repository carries no `LICENSE` file — the only one in the
tree belongs to a bundled skill. Vendoring it, forking it, or building our
published images with it all need a licence statement from the author
first. This is the one item that blocks everything else and it is a
question, not a task.
- **Ownership.** A fork we patch is a fork we maintain. Chapar's own TODO
notes that its two `release.sh` copies have already diverged; a third
environment of ours would be a fourth place for that to happen. Upstreaming
an `envs/hpcod` is the cheaper posture if the author is willing.
- **Our catalogue's shape.** Chapar's unit is a module in a release. Ours is
a named application with hardware requirements attached, because placement
reads it (`internal/catalog`). A Chapar release does not carry "this needs
a GPU" — that stays ours, and something has to map one onto the other.
## What is already written
`hpcod/image/apps/` holds the driver, the catalogue generator, and the
per-application metadata. Under Level 1 the `app.yaml` files, `build-push.sh`
and the `upstream:` entries all stand. The per-application `spack.yaml` files
and `spack.Containerfile.in` are the part Chapar replaces, and should not be
extended before this decision is made.
Separately: what a tenant ticks now reaches their cluster. It is recorded
in `cluster_app` at allocation, `provision.TFVars` emits it as the `apps`
variable, and the `apps` role pulls each image onto the head at first boot.
When this section was first written none of that was connected, and every
cluster got the module's `apps = []` default whatever the catalogue
resolved.