A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
4.6 KiB
5. Storage and Shared Filesystems
5.1 Layout
| Mount | Server | Export options | Client options | Purpose |
|---|---|---|---|---|
/home |
idm1 (10.10.10.5) |
rw,sync,no_subtree_check,no_root_squash |
defaults,nofail |
IPA user home directories |
/opt |
head01 (10.10.10.1) |
ro,sync,no_root_squash |
defaults,nofail,ro |
OpenHPC modules (/opt/ohpc/pub) |
Both are NFSv4, both exported only to 10.10.10.0/24 — the isolated cluster
fabric. Neither is reachable from vmbr0.
The split is deliberate: identity owns user data, the provisioning server
owns software. /opt/ohpc is built by the OpenHPC packages installed on
head01 and is read-only everywhere else; nothing about it belongs on the
identity server.
5.2 Client mounts
On the nodes the mounts come from the Warewulf fstab overlay, driven by the
fstab resource in nodes.conf:
[{"file":"/home","mntops":"defaults,nofail",
"spec":"idm1.hpc.local:/export/home","vfstype":"nfs"},
{"file":"/opt","mntops":"defaults,nofail,ro",
"spec":"warewulf:/opt","vfstype":"nfs"}]
Note /opt uses the name warewulf, an alias for the head node that Warewulf
injects into /etc/hosts via the hosts overlay. /home uses
idm1.hpc.local, which requires working DNS — one more reason node DNS must
point at the IPA server.
Originally the default /opt entry carried noauto, so it never mounted. It
was changed to auto-mount because the OpenHPC modules must be present before a
job starts.
On head01 the same /home mount is a plain /etc/fstab line:
idm1.hpc.local:/export/home /home nfs4 defaults,_netdev,nofail 0 0
5.3 Identity mapping
NFSv4 maps users by user@domain strings, so /etc/idmapd.conf must agree
across every host:
Domain = hpc.local
This is set on idm1, on head01, and inside the node image. A mismatch makes
files appear as nobody:nobody even when uid/gid are otherwise correct.
Because SSSD supplies the same uid/gid everywhere from IPA, ownership stays
consistent: hpcuser is 721800005 on the head and on all four nodes.
5.4 Security posture
The /export/home export uses sec=sys and no_root_squash. Both are
deliberate lab trade-offs, and both are choices you would revisit in
production.
sec=sys means the server trusts the uid the client asserts. There is no
cryptographic authentication of NFS traffic. It is acceptable here only because
vmbr6 is a closed L2 segment with four known hosts.
no_root_squash lets root on a client write as root on the server. It is
needed so pam_oddjob_mkhomedir can create a home directory on first login.
The alternative — pre-creating every home — was rejected because it makes user
onboarding a two-step manual process.
Path to Kerberised NFS
The groundwork is already in place: every node holds a host keytab, and the
nfs/idm1.hpc.local service principal was created during A40. Moving to
sec=krb5p requires:
- Fetching a keytab for
nfs/idm1.hpc.localon the server and enablingnfs-serverwithrpc.gssd/rpc.svcgssd - Adding
nfs/wwnodeN.hpc.localservice principals, or relying on the host principal viarpc.gssdmachine credentials - Changing the export to
sec=krb5pand the client mount options to match - Dropping
no_root_squash— with Kerberos,mkhomedirneeds a different approach (IPA automember + a home-provisioning hook, or automount withkrb5pand a pre-created skeleton)
Step 4 is the reason this was not done in the first pass: it changes the user onboarding flow, not just the mount options.
5.5 Capacity
| Volume | Backing | Size | Used |
|---|---|---|---|
idm1:/export/home |
zfspool/vm-305-disk-0 |
100 GB | ~3 GB |
head01:/opt |
zfspool/vm-300-disk-0 |
100 GB | ~15 GB (OpenHPC) |
| Node rootfs | RAM (tmpfs) | 8 GB per node | ~1.8 GB |
zfspool has roughly 5.4 TB free, so growth is not a near-term concern. The
binding constraint is node RAM, not disk: every gigabyte added to the node
image is a gigabyte taken from each node's usable memory.
5.6 Backup considerations
Nothing in this environment is currently backed up. If it mattered:
idm1:/export/home— the only irreplaceable user data.- FreeIPA —
ipa-backupproduces a full directory + CA dump. The CA private key in/root/cacert.p12cannot be regenerated; losing it means re-enrolling every host. head01:/etc/warewulf/—nodes.confandwarewulf.confare the whole cluster definition and are small enough to keep in version control.- Node images and overlays — reproducible from the scripts; not worth backing up.