Files
werewulf/docs/03-warewulf.md
T
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

6.0 KiB

3. Warewulf

Version 4.7.0-1.el9, installed from the official GitHub RPM — there is no Warewulf package in the Rocky or EPEL repositories.

3.1 Concepts

Concept Meaning
Image The root filesystem nodes run. Derived from an OCI image, kept as a chroot under /var/lib/warewulf/chroots/<name>/rootfs, served as provision/images/<name>.img.gz.
Overlay A set of files written on top of the image. Files ending in .ww are rendered as Go templates.
System overlay Applied at boot, before /sbin/init. Network, hostname and keytab live here.
Runtime overlay Pulled by wwclient every 60 s. For things that can change without a reboot (/etc/hosts, authorized_keys).
Profile Shared node settings. The default profile is applied to every node automatically.

3.2 The image in this build

IMAGE NAME   rocky-9
Source       docker://ghcr.io/warewulf/warewulf-rockylinux:9
Kernel       5.14.0-687.41.1.el9_8.x86_64
Size         1.8 GB (raw) / 1.2 GB (compressed)

Added on top of the base image:

  • kernel, kernel-modules, dracut, dracut-network — PXE boot
  • warewulf-dracut — the wwinit dracut module (builds the initramfs inside the image)
  • NetworkManager, openssh-server, nfs-utils, chrony
  • OpenHPC: ohpc-base-compute, lmod-ohpc, slurm-slurmd-ohpc, munge
  • Identity: sssd, sssd-ipa, krb5-workstation, oddjob-mkhomedir, authselect
  • /var/spool/slurm/{d,ctld} plus a tmpfiles.d/slurm.conf entry

Modifying the image

ssh head01
sudo wwctl image exec rocky-9 -- /bin/bash    # shell inside the image
  dnf install -y <package>
  exit                                         # rebuilds automatically on exit
sudo wwctl image build rocky-9                 # or rebuild explicitly
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset

image exec opens a chroot; on exit Warewulf resets machine-id and repacks the image. Nodes keep running the old one until they are rebooted.

3.3 Overlay inventory

Overlay Type Files Contents
hpc site, system 6 slurm.conf, cgroup.conf, munge.key
ipa site, system 7 krb5.conf, ipa/ca.crt, sssd.conf.ww, ipa/default.conf.ww
ipa.wwnode1…4 site, system 2 That node's /etc/krb5.keytab only

Assignment (from the default profile):

SystemOverlay:  wwinit, wwclient, hosts, fstab, hostname, ssh.host_keys,
                issue, resolv, udev.netname, systemd.netname, ifcfg,
                ifupdown, NetworkManager, wicked, ignition, hpc, ipa
RuntimeOverlay: hosts, ssh.authorized_keys

At node level only ipa.wwnodeN is added. Warewulf merges the profile and node lists rather than overriding — repeating the full list at node level applies every overlay twice.

Editing overlays

sudo wwctl overlay import -p -o <overlay> <source> <target-path>
sudo wwctl overlay edit   <overlay> <path>                  # open in an editor
sudo wwctl overlay list   <overlay>                          # contents
sudo wwctl overlay show --render <node> <overlay> <path>     # render a template
sudo wwctl overlay build                                     # rebuild everything

The import -p trap: parent directories it creates end up 0500 root:root. Left uncorrected this locks /etc away from every non-root user on the node. After any import -p:

sudo wwctl overlay chmod <overlay> /etc 0755

Templates

Files ending in .ww are rendered with Go text/template. The most useful variable is {{.Id}} — the node name (wwnode1). From the ipa overlay:

ipa_hostname = {{.Id}}.hpc.local

Others include {{.NetDevs}}, {{.Tags}}, {{.ImageName}} and {{.Kernel.Args}}. The stock resolv overlay uses {{$netdev.Tags.DNS1}}.

Templating is text-only. Binary files such as keytabs cannot be templated — which is precisely why per-node overlays were needed.

3.4 Node management

sudo wwctl node list                  # summary
sudo wwctl node list -a wwnode1       # all fields
sudo wwctl node status                # provisioning stage / last seen
sudo wwctl node add wwnode5 --ipaddr 10.10.10.15 --hwaddr BC:24:11:E0:0A:15 \
     --netdev eth0 --discoverable=false
sudo wwctl node set --yes wwnode5 -O ipa.wwnode5
sudo wwctl node delete --yes wwnode5

wwctl node status stages run IPXE → KERNEL → IMAGE → SYSTEM OVERLAY → RUNTIME OVERLAY. In the final stage LASTSEEN should stay under 60 s; a steadily climbing value means the node has died.

3.5 Network configuration

A node's address does not persist from DHCP. The sequence is:

  1. During PXE, a temporary lease from 10.10.10.100-199
  2. The system overlay writes NetworkManager/system-connections/warewulf-default.conf with interface-name=eth0, mac-address=…, address=10.10.10.11/24
  3. NM activates the connection and the node moves to its static address

--netdev must be the real interface name on the node (eth0). Warewulf writes it into interface-name=; give it a name that does not exist and NM never activates the connection, leaving the node stuck in the DHCP pool.

DNS comes from the resolv overlay, which reads netdev tags:

network devices:
  default:
    device: eth0
    tags:
      DNS1: 10.10.10.5
      DNSSEARCH: hpc.local

These tags cannot be written with wwctl profile set --nettagadd (it fails silently) — nodes.conf is edited directly instead.

3.6 File locations

Path Contents
/etc/warewulf/warewulf.conf Network, DHCP, TFTP, NFS, paths
/etc/warewulf/nodes.conf Node and profile definitions (YAML)
/var/lib/warewulf/chroots/<img>/rootfs Image chroot
/var/lib/warewulf/provision/images/ Served images (.img, .img.gz)
/var/lib/warewulf/overlays/<name>/rootfs Site overlay sources
/var/lib/warewulf/provision/overlays/<node>/ Per-node compiled overlays
/usr/share/warewulf/overlays/ Distribution-supplied overlays
/var/lib/tftpboot/warewulf/ iPXE bootloaders