A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
6.0 KiB
3. Warewulf
Version 4.7.0-1.el9, installed from the official GitHub RPM — there is no Warewulf package in the Rocky or EPEL repositories.
3.1 Concepts
| Concept | Meaning |
|---|---|
| Image | The root filesystem nodes run. Derived from an OCI image, kept as a chroot under /var/lib/warewulf/chroots/<name>/rootfs, served as provision/images/<name>.img.gz. |
| Overlay | A set of files written on top of the image. Files ending in .ww are rendered as Go templates. |
| System overlay | Applied at boot, before /sbin/init. Network, hostname and keytab live here. |
| Runtime overlay | Pulled by wwclient every 60 s. For things that can change without a reboot (/etc/hosts, authorized_keys). |
| Profile | Shared node settings. The default profile is applied to every node automatically. |
3.2 The image in this build
IMAGE NAME rocky-9
Source docker://ghcr.io/warewulf/warewulf-rockylinux:9
Kernel 5.14.0-687.41.1.el9_8.x86_64
Size 1.8 GB (raw) / 1.2 GB (compressed)
Added on top of the base image:
kernel,kernel-modules,dracut,dracut-network— PXE bootwarewulf-dracut— thewwinitdracut module (builds the initramfs inside the image)NetworkManager,openssh-server,nfs-utils,chrony- OpenHPC:
ohpc-base-compute,lmod-ohpc,slurm-slurmd-ohpc,munge - Identity:
sssd,sssd-ipa,krb5-workstation,oddjob-mkhomedir,authselect /var/spool/slurm/{d,ctld}plus atmpfiles.d/slurm.confentry
Modifying the image
ssh head01
sudo wwctl image exec rocky-9 -- /bin/bash # shell inside the image
dnf install -y <package>
exit # rebuilds automatically on exit
sudo wwctl image build rocky-9 # or rebuild explicitly
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
image exec opens a chroot; on exit Warewulf resets machine-id and repacks
the image. Nodes keep running the old one until they are rebooted.
3.3 Overlay inventory
| Overlay | Type | Files | Contents |
|---|---|---|---|
hpc |
site, system | 6 | slurm.conf, cgroup.conf, munge.key |
ipa |
site, system | 7 | krb5.conf, ipa/ca.crt, sssd.conf.ww, ipa/default.conf.ww |
ipa.wwnode1…4 |
site, system | 2 | That node's /etc/krb5.keytab only |
Assignment (from the default profile):
SystemOverlay: wwinit, wwclient, hosts, fstab, hostname, ssh.host_keys,
issue, resolv, udev.netname, systemd.netname, ifcfg,
ifupdown, NetworkManager, wicked, ignition, hpc, ipa
RuntimeOverlay: hosts, ssh.authorized_keys
At node level only ipa.wwnodeN is added. Warewulf merges the profile
and node lists rather than overriding — repeating the full list at node level
applies every overlay twice.
Editing overlays
sudo wwctl overlay import -p -o <overlay> <source> <target-path>
sudo wwctl overlay edit <overlay> <path> # open in an editor
sudo wwctl overlay list <overlay> # contents
sudo wwctl overlay show --render <node> <overlay> <path> # render a template
sudo wwctl overlay build # rebuild everything
The import -p trap: parent directories it creates end up 0500 root:root.
Left uncorrected this locks /etc away from every non-root user on the node.
After any import -p:
sudo wwctl overlay chmod <overlay> /etc 0755
Templates
Files ending in .ww are rendered with Go text/template. The most useful
variable is {{.Id}} — the node name (wwnode1). From the ipa overlay:
ipa_hostname = {{.Id}}.hpc.local
Others include {{.NetDevs}}, {{.Tags}}, {{.ImageName}} and
{{.Kernel.Args}}. The stock resolv overlay uses {{$netdev.Tags.DNS1}}.
Templating is text-only. Binary files such as keytabs cannot be templated — which is precisely why per-node overlays were needed.
3.4 Node management
sudo wwctl node list # summary
sudo wwctl node list -a wwnode1 # all fields
sudo wwctl node status # provisioning stage / last seen
sudo wwctl node add wwnode5 --ipaddr 10.10.10.15 --hwaddr BC:24:11:E0:0A:15 \
--netdev eth0 --discoverable=false
sudo wwctl node set --yes wwnode5 -O ipa.wwnode5
sudo wwctl node delete --yes wwnode5
wwctl node status stages run IPXE → KERNEL → IMAGE → SYSTEM OVERLAY →
RUNTIME OVERLAY. In the final stage LASTSEEN should stay under 60 s; a
steadily climbing value means the node has died.
3.5 Network configuration
A node's address does not persist from DHCP. The sequence is:
- During PXE, a temporary lease from
10.10.10.100-199 - The system overlay writes
NetworkManager/system-connections/warewulf-default.confwithinterface-name=eth0,mac-address=…,address=10.10.10.11/24 - NM activates the connection and the node moves to its static address
--netdev must be the real interface name on the node (eth0). Warewulf
writes it into interface-name=; give it a name that does not exist and NM
never activates the connection, leaving the node stuck in the DHCP pool.
DNS comes from the resolv overlay, which reads netdev tags:
network devices:
default:
device: eth0
tags:
DNS1: 10.10.10.5
DNSSEARCH: hpc.local
These tags cannot be written with wwctl profile set --nettagadd (it fails
silently) — nodes.conf is edited directly instead.
3.6 File locations
| Path | Contents |
|---|---|
/etc/warewulf/warewulf.conf |
Network, DHCP, TFTP, NFS, paths |
/etc/warewulf/nodes.conf |
Node and profile definitions (YAML) |
/var/lib/warewulf/chroots/<img>/rootfs |
Image chroot |
/var/lib/warewulf/provision/images/ |
Served images (.img, .img.gz) |
/var/lib/warewulf/overlays/<name>/rootfs |
Site overlay sources |
/var/lib/warewulf/provision/overlays/<node>/ |
Per-node compiled overlays |
/usr/share/warewulf/overlays/ |
Distribution-supplied overlays |
/var/lib/tftpboot/warewulf/ |
iPXE bootloaders |