The repository had no license, so it was public but not open source: nobody could use, change or redistribute it. LICENSE is the text from apache.org, unmodified. NOTICE says that the IBM Plex fonts embedded in the print handbook are under the SIL Open Font License, not Apache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Warewulf HPC Test Cluster
A working Warewulf 4 HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity (FreeIPA) and shared NFS home directories.
Status: built and verified. Four diskless nodes PXE-boot from the head
node, enrol into the IPA domain, mount NFS homes, register idle in Slurm, and
run an 8-rank MPI job across all four as a directory user.
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ Warewulf · DHCP │ │ FreeIPA (hpc.local) │
│ TFTP/iPXE · Slurm │ │ LDAP · Kerberos │
│ MUNGE · NAT gateway │ │ CA · DNS │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4 (VM 301-304)
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
/home → idm1:/export/home /opt → head01:/opt (ro)
Documentation
A 25-page PDF edition of the whole handbook: docs/warewulf-cluster-handbook.pdf.
Regenerate it with ./scripts/build-pdf.sh after editing docs/handbook-print.html.
| Document | Contents |
|---|---|
| 01 — Architecture | Topology, why vmbr6 has no IP, why idm1 is dual-homed, the stateless boot chain |
| 02 — Installation | Installing hpcod from scratch: one binary, sudo ./hpcod setup, verify and a test cluster |
| 03 — Warewulf | Images, overlays, templates, node management |
| 04 — Identity | FreeIPA vs AD, enrolling stateless nodes, users and groups |
| 05 — Storage | NFS layout, id mapping, security posture, path to Kerberised NFS |
| 06 — Slurm & MPI | Scheduler config, running jobs, what is not configured |
| 07 — Operations | Runbook: access, power, health checks, common tasks, logs |
| 08 — Troubleshooting | The 12 pitfalls hit during the build, plus a diagnostic guide |
| 09 — Reference | Inventory, addressing, ports, versions, file locations |
| 10 — Building the lab | The reference lab built by hand: script order, what each does, the two manual steps, timings |
| 20 — HPC-on-Demand | Design: multi-site control plane that provisions clusters like this one on demand |
| 21 — Portal layer | What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to |
| 22 — Spack & Chapar | Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first |
| 23 — Application lifecycle | What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them |
| hpcod/ | Implementation: multi-provider discovery, placement, and the portal at hpc.example.org |
| hpcod/DEPLOY.md | Installing hpcod against your own Proxmox: sudo ./hpcod setup, from the binary alone |
| CHANGELOG | What is in each release, and what is knowingly not |
Quick start
ssh head01 # head node
ssh idm1 # identity + home server
sinfo # cluster state
sudo wwctl node status # provisioning state
./scripts/A99-verify.sh # full end-to-end health check
Compute nodes live on the isolated subnet — reach them through the head node:
ssh head01
sudo ssh root@wwnode1
HPC-on-Demand portal — https://hpc.example.org (see hpcod/).
Users ask for a cluster there, and it is built, published and handed over.
To install hpcod somewhere else, download the release binary and run
sudo ./hpcod setup (docs/02-installation.md).
Power management:
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes
License
Apache License 2.0. The fonts embedded in the print handbook are under the SIL Open Font License; see NOTICE.
Software
| Component | Version |
|---|---|
| Proxmox VE | 8.4.19 |
| Host / node OS | Rocky Linux 9.8 |
| Warewulf | 4.7.0 |
| OpenHPC | 3 (EL_9) |
| Slurm | 25.11.4 |
| OpenMPI | 5.0.7 (GCC 14.2) |
| FreeIPA | 4.13.1 |
| SSSD | 2.9.8 |
Three decisions worth knowing
vmbr6 has no IP on Proxmox. Warewulf needs to own the entire subnet —
DHCP with PXE options, TFTP, address management. A second authority on the same
segment would collide. The bridge is a pure L2 switch and head01 is the
gateway. See 01-architecture §1.3.
idm1 is dual-homed, and idm1.hpc.local resolves to the cluster side.
Behind head01's NAT all four nodes would appear as 10.0.0.30, which breaks
Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every
client a direct, symmetric path. See 01-architecture §1.4.
Stateless nodes cannot run ipa-client-install. The rootfs is rebuilt in
RAM every boot, so enrolment happens once and the result is distributed through
Warewulf overlays — shared config as templates rendered per node, and a
separate keytab per node so wwnode1 cannot impersonate wwnode2.
See 04-identity §4.3.
Verified output
================= IPA SERVER (idm1) =================
IPA services running: 9
/export/home 10.10.10.0/24(rw,no_root_squash,sec=sys,...)
IPA users: 3
================= HEAD NODE (head01) =================
hostname : head01.hpc.local
DNS : 10.10.10.5
/home : idm1.hpc.local:/export/home
hpcuser : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
services : warewulfd=active dhcpd=active tftp.socket=active
slurmctld=active munge=active sssd=active
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
normal* up infinite 4 idle wwnode[1-4]
================= COMPUTE NODES =================
wwnode1: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode2: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode3: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode4: sssd=active slurmd=active munge=active home=nfs krb5=OK
================= MPI JOB (IPA user, NFS home) =================
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
krb5=OK means the node actually obtained a ticket from the KDC using its own
host keytab (kinit -k host/wwnodeN.hpc.local) — not merely that the file is
present.
Additionally verified by hand: an IPA user SSH'd into a stateless node with a password, received a Kerberos TGT and landed in their NFS home; and a brand-new IPA user's home was auto-created on the NFS share at first login.
Known gaps
- NFS is
sec=sys, not Kerberised. Host keytabs and thenfs/idm1.hpc.localservice principal already exist, sosec=krb5pis a short step — but it changes the home-provisioning flow. - Slurm accounting is recorded but not enforced —
sacctandsreportwork;AccountingStorageEnforceis unset, so no association is required to submit. - SELinux permissive on head and nodes.
- FreeIPA is not Active Directory. It is the RHEL-native equivalent for Linux clients; Windows joins or the AD schema would need a Samba AD DC or an AD trust. See 04-identity §4.1.
Full list with rationale in 09-reference §9.9.