Files
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

4.6 KiB

5. Storage and Shared Filesystems

5.1 Layout

Mount Server Export options Client options Purpose
/home idm1 (10.10.10.5) rw,sync,no_subtree_check,no_root_squash defaults,nofail IPA user home directories
/opt head01 (10.10.10.1) ro,sync,no_root_squash defaults,nofail,ro OpenHPC modules (/opt/ohpc/pub)

Both are NFSv4, both exported only to 10.10.10.0/24 — the isolated cluster fabric. Neither is reachable from vmbr0.

The split is deliberate: identity owns user data, the provisioning server owns software. /opt/ohpc is built by the OpenHPC packages installed on head01 and is read-only everywhere else; nothing about it belongs on the identity server.

5.2 Client mounts

On the nodes the mounts come from the Warewulf fstab overlay, driven by the fstab resource in nodes.conf:

[{"file":"/home","mntops":"defaults,nofail",
  "spec":"idm1.hpc.local:/export/home","vfstype":"nfs"},
 {"file":"/opt","mntops":"defaults,nofail,ro",
  "spec":"warewulf:/opt","vfstype":"nfs"}]

Note /opt uses the name warewulf, an alias for the head node that Warewulf injects into /etc/hosts via the hosts overlay. /home uses idm1.hpc.local, which requires working DNS — one more reason node DNS must point at the IPA server.

Originally the default /opt entry carried noauto, so it never mounted. It was changed to auto-mount because the OpenHPC modules must be present before a job starts.

On head01 the same /home mount is a plain /etc/fstab line:

idm1.hpc.local:/export/home  /home  nfs4  defaults,_netdev,nofail  0 0

5.3 Identity mapping

NFSv4 maps users by user@domain strings, so /etc/idmapd.conf must agree across every host:

Domain = hpc.local

This is set on idm1, on head01, and inside the node image. A mismatch makes files appear as nobody:nobody even when uid/gid are otherwise correct.

Because SSSD supplies the same uid/gid everywhere from IPA, ownership stays consistent: hpcuser is 721800005 on the head and on all four nodes.

5.4 Security posture

The /export/home export uses sec=sys and no_root_squash. Both are deliberate lab trade-offs, and both are choices you would revisit in production.

sec=sys means the server trusts the uid the client asserts. There is no cryptographic authentication of NFS traffic. It is acceptable here only because vmbr6 is a closed L2 segment with four known hosts.

no_root_squash lets root on a client write as root on the server. It is needed so pam_oddjob_mkhomedir can create a home directory on first login. The alternative — pre-creating every home — was rejected because it makes user onboarding a two-step manual process.

Path to Kerberised NFS

The groundwork is already in place: every node holds a host keytab, and the nfs/idm1.hpc.local service principal was created during A40. Moving to sec=krb5p requires:

  1. Fetching a keytab for nfs/idm1.hpc.local on the server and enabling nfs-server with rpc.gssd/rpc.svcgssd
  2. Adding nfs/wwnodeN.hpc.local service principals, or relying on the host principal via rpc.gssd machine credentials
  3. Changing the export to sec=krb5p and the client mount options to match
  4. Dropping no_root_squash — with Kerberos, mkhomedir needs a different approach (IPA automember + a home-provisioning hook, or automount with krb5p and a pre-created skeleton)

Step 4 is the reason this was not done in the first pass: it changes the user onboarding flow, not just the mount options.

5.5 Capacity

Volume Backing Size Used
idm1:/export/home zfspool/vm-305-disk-0 100 GB ~3 GB
head01:/opt zfspool/vm-300-disk-0 100 GB ~15 GB (OpenHPC)
Node rootfs RAM (tmpfs) 8 GB per node ~1.8 GB

zfspool has roughly 5.4 TB free, so growth is not a near-term concern. The binding constraint is node RAM, not disk: every gigabyte added to the node image is a gigabyte taken from each node's usable memory.

5.6 Backup considerations

Nothing in this environment is currently backed up. If it mattered:

  • idm1:/export/home — the only irreplaceable user data.
  • FreeIPA — ipa-backup produces a full directory + CA dump. The CA private key in /root/cacert.p12 cannot be regenerated; losing it means re-enrolling every host.
  • head01:/etc/warewulf/ — nodes.conf and warewulf.conf are the whole cluster definition and are small enough to keep in version control.
  • Node images and overlays — reproducible from the scripts; not worth backing up.