How someone else installs hpcod: the binary, sudo hpcod setup, what it asks and what each step does, the simulated Proxmox, a closed network with its own CA and DNS, verify, selftest, and what to back up. The changelog says what was proven and what was not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Warewulf HPC Test Cluster
A working Warewulf 4 HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity (FreeIPA) and shared NFS home directories.
Status: built and verified. Four diskless nodes PXE-boot from the head
node, enrol into the IPA domain, mount NFS homes, register idle in Slurm, and
run an 8-rank MPI job across all four as a directory user.
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ Warewulf · DHCP │ │ FreeIPA (hpc.local) │
│ TFTP/iPXE · Slurm │ │ LDAP · Kerberos │
│ MUNGE · NAT gateway │ │ CA · DNS │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4 (VM 301-304)
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
/home → idm1:/export/home /opt → head01:/opt (ro)
Documentation
A 22-page PDF edition of the whole handbook: docs/warewulf-cluster-handbook.pdf.
Regenerate it with ./scripts/build-pdf.sh after editing docs/handbook-print.html.
| Document | Contents |
|---|---|
| 01 — Architecture | Topology, why vmbr6 has no IP, why idm1 is dual-homed, the stateless boot chain |
| 02 — Installation | Building from scratch, what each script does, timings |
| 03 — Warewulf | Images, overlays, templates, node management |
| 04 — Identity | FreeIPA vs AD, enrolling stateless nodes, users and groups |
| 05 — Storage | NFS layout, id mapping, security posture, path to Kerberised NFS |
| 06 — Slurm & MPI | Scheduler config, running jobs, what is not configured |
| 07 — Operations | Runbook: access, power, health checks, common tasks, logs |
| 08 — Troubleshooting | The 12 pitfalls hit during the build, plus a diagnostic guide |
| 09 — Reference | Inventory, addressing, ports, versions, file locations |
| 20 — HPC-on-Demand | Design: multi-site control plane that provisions clusters like this one on demand |
| 21 — Portal layer | What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to |
| 22 — Spack & Chapar | Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first |
| 23 — Application lifecycle | What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them |
| hpcod/ | Implementation: multi-provider discovery, placement, and the portal at hpc.example.org |
| hpcod/DEPLOY.md | Installing hpcod against your own Proxmox: sudo hpcod setup, from the binary alone |
| CHANGELOG | What is in each release, and what is knowingly not |
Quick start
ssh head01 # head node
ssh idm1 # identity + home server
sinfo # cluster state
sudo wwctl node status # provisioning state
./scripts/A99-verify.sh # full end-to-end health check
Compute nodes live on the isolated subnet — reach them through the head node:
ssh head01
sudo ssh root@wwnode1
HPC-on-Demand portal — https://hpc.example.org (see hpcod/). Read-only: what the estate looks like, and where a request would be placed.
Power management:
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes
Software
| Component | Version |
|---|---|
| Proxmox VE | 8.4.19 |
| Host / node OS | Rocky Linux 9.8 |
| Warewulf | 4.7.0 |
| OpenHPC | 3 (EL_9) |
| Slurm | 25.11.4 |
| OpenMPI | 5.0.7 (GCC 14.2) |
| FreeIPA | 4.13.1 |
| SSSD | 2.9.8 |
Three decisions worth knowing
vmbr6 has no IP on Proxmox. Warewulf needs to own the entire subnet —
DHCP with PXE options, TFTP, address management. A second authority on the same
segment would collide. The bridge is a pure L2 switch and head01 is the
gateway. See 01-architecture §1.3.
idm1 is dual-homed, and idm1.hpc.local resolves to the cluster side.
Behind head01's NAT all four nodes would appear as 10.0.0.30, which breaks
Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every
client a direct, symmetric path. See 01-architecture §1.4.
Stateless nodes cannot run ipa-client-install. The rootfs is rebuilt in
RAM every boot, so enrolment happens once and the result is distributed through
Warewulf overlays — shared config as templates rendered per node, and a
separate keytab per node so wwnode1 cannot impersonate wwnode2.
See 04-identity §4.3.
Verified output
================= IPA SERVER (idm1) =================
IPA services running: 9
/export/home 10.10.10.0/24(rw,no_root_squash,sec=sys,...)
IPA users: 3
================= HEAD NODE (head01) =================
hostname : head01.hpc.local
DNS : 10.10.10.5
/home : idm1.hpc.local:/export/home
hpcuser : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
services : warewulfd=active dhcpd=active tftp.socket=active
slurmctld=active munge=active sssd=active
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
normal* up infinite 4 idle wwnode[1-4]
================= COMPUTE NODES =================
wwnode1: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode2: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode3: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode4: sssd=active slurmd=active munge=active home=nfs krb5=OK
================= MPI JOB (IPA user, NFS home) =================
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
krb5=OK means the node actually obtained a ticket from the KDC using its own
host keytab (kinit -k host/wwnodeN.hpc.local) — not merely that the file is
present.
Additionally verified by hand: an IPA user SSH'd into a stateless node with a password, received a Kerberos TGT and landed in their NFS home; and a brand-new IPA user's home was auto-created on the NFS share at first login.
Known gaps
- NFS is
sec=sys, not Kerberised. Host keytabs and thenfs/idm1.hpc.localservice principal already exist, sosec=krb5pis a short step — but it changes the home-provisioning flow. - Slurm accounting is recorded but not enforced —
sacctandsreportwork;AccountingStorageEnforceis unset, so no association is required to submit. - SELinux permissive on head and nodes.
- FreeIPA is not Active Directory. It is the RHEL-native equivalent for Linux clients; Windows joins or the AD schema would need a Samba AD DC or an AD trust. See 04-identity §4.1.
Full list with rationale in 09-reference §9.9.