The catalogue published one image. Filling it is not nine more Containerfiles: it is a question of who compiles the software, against which MPI, and how a second cluster gets it without paying for the first one's build. image/apps holds the answer to the first of those - one driver over one directory per application, with three recipes (somebody else's image, one we assemble, or Spack against our own MPI) and the rule for choosing between them. The catalogue is generated from the recipes rather than kept beside them, because an entry in it is a promise that a head node can pull the image, and a promise nobody checks is how the first generated catalogue came to point busybox at a tag that had never existed - guessed from the upstream version instead of read from the registry. The generator now asks the registry, and an application whose image is not there is reported rather than offered. Recipes are intent and published.json is measurement, kept apart so that recording the hash of a push does not rewrite the reasoning in the recipe that produced it. Two entries to start: busybox, which is the cheapest way to prove the whole delivery path, and stress-ng, which is what somebody actually runs first on a cluster they have just been handed. docs/22 is the Spack half of the question, from reading Chapar: what it would give us, the three levels of integration, and the PMIx version to settle on a real head before a single image is built against it. Two things about the build host, both found by trying: Apptainer is not installed, and Docker is, which sets FORWARD DROP and leaves podman unable to build anything that needs the network. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8.5 KiB
Warewulf HPC Test Cluster
A working Warewulf 4 HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity (FreeIPA) and shared NFS home directories.
Status: built and verified. Four diskless nodes PXE-boot from the head
node, enrol into the IPA domain, mount NFS homes, register idle in Slurm, and
run an 8-rank MPI job across all four as a directory user.
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ Warewulf · DHCP │ │ FreeIPA (hpc.local) │
│ TFTP/iPXE · Slurm │ │ LDAP · Kerberos │
│ MUNGE · NAT gateway │ │ CA · DNS │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4 (VM 301-304)
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
/home → idm1:/export/home /opt → head01:/opt (ro)
Documentation
A 22-page PDF edition of the whole handbook: docs/warewulf-cluster-handbook.pdf.
Regenerate it with ./scripts/build-pdf.sh after editing docs/handbook-print.html.
| Document | Contents |
|---|---|
| 01 — Architecture | Topology, why vmbr6 has no IP, why idm1 is dual-homed, the stateless boot chain |
| 02 — Installation | Building from scratch, what each script does, timings |
| 03 — Warewulf | Images, overlays, templates, node management |
| 04 — Identity | FreeIPA vs AD, enrolling stateless nodes, users and groups |
| 05 — Storage | NFS layout, id mapping, security posture, path to Kerberised NFS |
| 06 — Slurm & MPI | Scheduler config, running jobs, what is not configured |
| 07 — Operations | Runbook: access, power, health checks, common tasks, logs |
| 08 — Troubleshooting | The 12 pitfalls hit during the build, plus a diagnostic guide |
| 09 — Reference | Inventory, addressing, ports, versions, file locations |
| 20 — HPC-on-Demand | Design: multi-site control plane that provisions clusters like this one on demand |
| 21 — Portal layer | What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to |
| 22 — Spack & Chapar | Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first |
| 23 — Application lifecycle | What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them |
| hpcod/ | Implementation: multi-provider discovery, placement, and the portal at hpc.example.org |
| CHANGELOG | What is in each release, and what is knowingly not |
Quick start
ssh head01 # head node
ssh idm1 # identity + home server
sinfo # cluster state
sudo wwctl node status # provisioning state
./scripts/A99-verify.sh # full end-to-end health check
Compute nodes live on the isolated subnet — reach them through the head node:
ssh head01
sudo ssh root@wwnode1
HPC-on-Demand portal — https://hpc.example.org (see hpcod/). Read-only: what the estate looks like, and where a request would be placed.
Power management:
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes
Software
| Component | Version |
|---|---|
| Proxmox VE | 8.4.19 |
| Host / node OS | Rocky Linux 9.8 |
| Warewulf | 4.7.0 |
| OpenHPC | 3 (EL_9) |
| Slurm | 25.11.4 |
| OpenMPI | 5.0.7 (GCC 14.2) |
| FreeIPA | 4.13.1 |
| SSSD | 2.9.8 |
Three decisions worth knowing
vmbr6 has no IP on Proxmox. Warewulf needs to own the entire subnet —
DHCP with PXE options, TFTP, address management. A second authority on the same
segment would collide. The bridge is a pure L2 switch and head01 is the
gateway. See 01-architecture §1.3.
idm1 is dual-homed, and idm1.hpc.local resolves to the cluster side.
Behind head01's NAT all four nodes would appear as 10.0.0.30, which breaks
Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every
client a direct, symmetric path. See 01-architecture §1.4.
Stateless nodes cannot run ipa-client-install. The rootfs is rebuilt in
RAM every boot, so enrolment happens once and the result is distributed through
Warewulf overlays — shared config as templates rendered per node, and a
separate keytab per node so wwnode1 cannot impersonate wwnode2.
See 04-identity §4.3.
Verified output
================= IPA SERVER (idm1) =================
IPA services running: 9
/export/home 10.10.10.0/24(rw,no_root_squash,sec=sys,...)
IPA users: 3
================= HEAD NODE (head01) =================
hostname : head01.hpc.local
DNS : 10.10.10.5
/home : idm1.hpc.local:/export/home
hpcuser : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
services : warewulfd=active dhcpd=active tftp.socket=active
slurmctld=active munge=active sssd=active
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
normal* up infinite 4 idle wwnode[1-4]
================= COMPUTE NODES =================
wwnode1: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode2: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode3: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode4: sssd=active slurmd=active munge=active home=nfs krb5=OK
================= MPI JOB (IPA user, NFS home) =================
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
krb5=OK means the node actually obtained a ticket from the KDC using its own
host keytab (kinit -k host/wwnodeN.hpc.local) — not merely that the file is
present.
Additionally verified by hand: an IPA user SSH'd into a stateless node with a password, received a Kerberos TGT and landed in their NFS home; and a brand-new IPA user's home was auto-created on the NFS share at first login.
Known gaps
- NFS is
sec=sys, not Kerberised. Host keytabs and thenfs/idm1.hpc.localservice principal already exist, sosec=krb5pis a short step — but it changes the home-provisioning flow. - Slurm accounting is recorded but not enforced —
sacctandsreportwork;AccountingStorageEnforceis unset, so no association is required to submit. - SELinux permissive on head and nodes.
- FreeIPA is not Active Directory. It is the RHEL-native equivalent for Linux clients; Windows joins or the AD schema would need a Samba AD DC or an AD trust. See 04-identity §4.1.
Full list with rationale in 09-reference §9.9.