- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for installing the control plane. - 09: repository layout now includes hpcod/, docs 20-23 and the B/C scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is recorded, not absent (matches 6.6). Handbook PDF regenerated. - 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run from the control plane; Terraform state per cluster, not in Postgres; no reaper yet; usage per project. - 21: c<NNN> portals, publishing wired into provisioning, closed-network mode. - 22: three published images; chosen apps now reach the cluster. - 23: setup installs the catalogue; apps.local keeps a site's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
207 lines
10 KiB
Markdown
207 lines
10 KiB
Markdown
# 22 — Spack, and what Chapar would give us
|
|
|
|
Written for whoever maintains `hpcod/` next. It assumes §20 and §21, and it
|
|
assumes nothing about Spack.
|
|
|
|
The application catalogue currently publishes three images: busybox,
|
|
osu-micro-benchmarks and stress-ng. Filling it is not a matter of writing
|
|
more Containerfiles — it is a question of who
|
|
compiles the software, against which MPI, and how a second cluster gets it
|
|
without paying for the first cluster's build again. [Chapar][chapar] answers
|
|
all three, and it answers them from production rather than from a design
|
|
document, which is the reason this section exists.
|
|
|
|
[chapar]: https://github.com/mmrezaie/chapar
|
|
|
|
## What Chapar is, in our terms
|
|
|
|
A site's Spack policy, kept outside Spack. One `envs/<name>/spack.yaml` holds
|
|
the root specs; `etc/system` and `etc/user` hold the compiler, libc, mirror
|
|
and concretiser policy that every environment shares. From that it builds a
|
|
**release**: an install tree plus a tree of Lmod modulefiles named
|
|
`{name}/{version}` with no hashes in them.
|
|
|
|
Four things in it are worth more than the package list:
|
|
|
|
- **Releases are atomic.** A build goes into `<os>/.<release-id>.staging.<pid>`
|
|
and is renamed into place only after install *and* module generation
|
|
succeed. Publication is a symlink flip. Rollback is the same flip backwards.
|
|
Nothing overwrites a tree a running job is reading from.
|
|
- **The buildcache is the point.** Concretised specs that already exist as
|
|
binaries are downloaded, not rebuilt. Source-built packages are pushed as
|
|
they complete, so even a failed build leaves the next one faster.
|
|
- **The MPI runtime policy lives in the generated module,** not in a wiki.
|
|
Chapar's `openmpi` module picks UCX transports from the node it is loaded
|
|
on, so one build serves CPU and GPU nodes; its Intel MPI module exposes a
|
|
CUDA driver stub so a CPU-only node can start a CUDA-aware binary instead
|
|
of dying on a missing `libcuda`.
|
|
- **It already made the PMIx decision.** `openmpi@5 ~internal-pmix ^pmix@6
|
|
schedulers=slurm` — external PMIx, not the bundled one, so the host's
|
|
`srun` can launch ranks directly.
|
|
|
|
That last line is the one we would otherwise have had to discover by watching
|
|
a two-node job hang.
|
|
|
|
## Where it does not fit, stated plainly
|
|
|
|
Chapar's container is a **thin runtime wrapper**. Its own README is explicit:
|
|
"It does not embed the Spack install tree. At runtime, bind the site
|
|
resources tree so the container sees the promoted release." The software
|
|
lives at a stable absolute path — `/resources/chapar/hpcsim/rocky9/current` —
|
|
on a filesystem that outlives every job.
|
|
|
|
Our containers are the opposite. A SIF in `/opt/apps` is self-contained and
|
|
is pulled fresh onto a head node that will be destroyed in fourteen days.
|
|
There is no site filesystem. `/opt` is exported by *each cluster's own head*,
|
|
which is the whole reason a tenant's software is a tenant's business.
|
|
|
|
So Chapar cannot simply be switched on. Its buildcache mirror is also a
|
|
`file://` path, which assumes the same shared filesystem; we have a registry
|
|
and no shared filesystem at all.
|
|
|
|
This is a mismatch about **lifetime**, not about correctness — and once it is
|
|
named that way the fit becomes obvious, because we have three lifetimes and
|
|
have been trying to serve them with one mechanism:
|
|
|
|
| | lives for | where | built by |
|
|
|---|---|---|---|
|
|
| the software stack | months | the site | Chapar, once per release |
|
|
| an application image | months | the registry | `image/apps/build-push.sh` |
|
|
| a cluster | fourteen days | Proxmox | hpcod, per tenant |
|
|
|
|
## Three levels, and which one to take
|
|
|
|
### Level 1 — Chapar as the image factory
|
|
|
|
Nothing about our runtime changes. A SIF is still fat, still pulled per
|
|
cluster, still lands in `/opt/apps`. What changes is who compiles it.
|
|
|
|
Today's sketch in `image/apps/` gives each application its own `spack.yaml`,
|
|
which means each application concretises independently. Two images built a
|
|
month apart get different OpenMPI patch levels and nobody finds out until
|
|
somebody tries to run them in the same job script. Instead:
|
|
|
|
- one Chapar environment, `envs/hpcod/spack.yaml`, whose root specs **are**
|
|
the application catalogue;
|
|
- one concretisation, therefore one compiler and one MPI across every image
|
|
we publish;
|
|
- an **OCI buildcache in Gitea** rather than Chapar's `file://` mirror —
|
|
Spack has supported `oci://` mirrors since 0.21, and we already run a
|
|
registry, which is exactly the shared store we do not otherwise have;
|
|
- per-application images that install only their own spec with
|
|
`--use-buildcache only`, turning a forty-minute GROMACS build into a
|
|
download.
|
|
|
|
Cost: a fork or a vendored checkout of Chapar, and one full cold build to
|
|
seed the cache. Everything after that is cheap.
|
|
|
|
### Level 2 — the release tree mounted into every cluster
|
|
|
|
Build one Chapar release for the site and let every tenant cluster read it.
|
|
Users get `module avail` over the whole stack — several hundred packages,
|
|
benchmarks and profilers included — with no per-cluster pull cost at all.
|
|
|
|
The topology objection answers itself: **the head already exports `/opt`
|
|
read-only to its compute nodes.** Mount the site tree on the head at
|
|
`/opt/chapar` and the nodes see it through an export that already exists.
|
|
No new route across `vmbr6`, no node-image change beyond dropping in a
|
|
`profile.d` hook of the kind Chapar already ships.
|
|
|
|
How the head gets it is the real choice, and NFS from a long-lived host is
|
|
the obvious answer rather than the best one. A **read-only disk attached by
|
|
Terraform** is a better fit for what we actually run: Proxmox can clone a
|
|
volume instantly, the clone costs nothing until written, the cluster gains no
|
|
dependency on a host outside itself, and a release is promoted by attaching a
|
|
different volume. We already create disks in the tenant module.
|
|
|
|
Level 2 subsumes Level 1 — the SIFs remain the right answer for anything with
|
|
an awkward licence, a vendor binary, or a Python stack users expect to `pip
|
|
install` into.
|
|
|
|
### Level 3 — the operational machinery
|
|
|
|
Independent of the above, and all of it is work we would otherwise write:
|
|
|
|
- **`ci/cve-checker.py`** and its systemd timer. We publish images to
|
|
tenants; a marketplace that cannot say when it is shipping a known
|
|
vulnerability is not one an institution can adopt.
|
|
- **`validation/`** — `osu-micro-benchmarks`, `ior`, `mdtest`, `hpl`,
|
|
`stream`, and MPI+RDMA programs including CUDA. This is a better
|
|
post-provision acceptance suite than `roles/verify` and `70-mpi-test.sh`,
|
|
and it is the thing that would tell a tenant their fourteen-day cluster is
|
|
actually healthy.
|
|
- **The staging-then-rename discipline** applies to our catalogue as much as
|
|
to a module tree.
|
|
|
|
## Recommendation
|
|
|
|
Level 1 now, Level 3 opportunistically, Level 2 when a second site exists.
|
|
|
|
Level 2's payoff scales with how many clusters read one tree, and we have one
|
|
Proxmox node. Level 1's payoff is immediate and its blast radius is a build
|
|
host.
|
|
|
|
## Measure this before anything else
|
|
|
|
Chapar's hpcsim pins **PMIx 6**. Our head runs OpenHPC 3 with Slurm 25.11,
|
|
whose PMIx is its own. PMIx keeps cross-version compatibility, but not
|
|
without limit, and the failure mode is not an error: the job starts, the
|
|
ranks never rendezvous, and it sits there until the wall clock kills it.
|
|
That reads as a slow application, not a broken stack.
|
|
|
|
So, on a real head, before a single image is built against Chapar's MPI:
|
|
|
|
```bash
|
|
srun --mpi=list # what our Slurm offers
|
|
rpm -q pmix pmix-libs # what version it is built against
|
|
```
|
|
|
|
If those do not line up with `^pmix@6`, pin Chapar's PMIx to our Slurm's
|
|
major version rather than the other way round — or build Slurm through
|
|
Chapar as well, which is already on its own TODO and which would retire the
|
|
question entirely by making both sides come out of one build.
|
|
|
|
Two further things to check on the same visit, both cheap and both capable of
|
|
silently wasting a build:
|
|
|
|
- **Microarchitecture.** Chapar concretises with `targets: granularity:
|
|
generic`, so it produces `x86_64_v2/v3/v4` rather than `zen5`. Good for us
|
|
— but our Proxmox guests must actually advertise the level the cache was
|
|
built for, and `x86_64_v4` needs AVX-512 that a default VM CPU model does
|
|
not expose. A mismatch does not fail; it misses the cache and rebuilds
|
|
everything from source.
|
|
- **CUDA.** hpcsim builds `cuda@13` with a `cuda_arch` list of thirteen
|
|
architectures. Our GPU story is whole-device passthrough on hardware we
|
|
should confirm is in that list before paying for the build.
|
|
|
|
## Open questions
|
|
|
|
- **Licence.** The repository carries no `LICENSE` file — the only one in the
|
|
tree belongs to a bundled skill. Vendoring it, forking it, or building our
|
|
published images with it all need a licence statement from the author
|
|
first. This is the one item that blocks everything else and it is a
|
|
question, not a task.
|
|
- **Ownership.** A fork we patch is a fork we maintain. Chapar's own TODO
|
|
notes that its two `release.sh` copies have already diverged; a third
|
|
environment of ours would be a fourth place for that to happen. Upstreaming
|
|
an `envs/hpcod` is the cheaper posture if the author is willing.
|
|
- **Our catalogue's shape.** Chapar's unit is a module in a release. Ours is
|
|
a named application with hardware requirements attached, because placement
|
|
reads it (`internal/catalog`). A Chapar release does not carry "this needs
|
|
a GPU" — that stays ours, and something has to map one onto the other.
|
|
|
|
## What is already written
|
|
|
|
`hpcod/image/apps/` holds the driver, the catalogue generator, and the
|
|
per-application metadata. Under Level 1 the `app.yaml` files, `build-push.sh`
|
|
and the `upstream:` entries all stand. The per-application `spack.yaml` files
|
|
and `spack.Containerfile.in` are the part Chapar replaces, and should not be
|
|
extended before this decision is made.
|
|
|
|
Separately: what a tenant ticks now reaches their cluster. It is recorded
|
|
in `cluster_app` at allocation, `provision.TFVars` emits it as the `apps`
|
|
variable, and the `apps` role pulls each image onto the head at first boot.
|
|
When this section was first written none of that was connected, and every
|
|
cluster got the module's `apps = []` default whatever the catalogue
|
|
resolved.
|