A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
60 lines
1.9 KiB
Bash
Executable File
60 lines
1.9 KiB
Bash
Executable File
#!/bin/bash
|
|
# Warewulf node image: Rocky 9 + kernel + wwinit dracut + OpenHPC compute stack
|
|
set -euo pipefail
|
|
IMG=rocky-9
|
|
WW_VER=4.7.0
|
|
|
|
echo "==> kernel and provisioning components into the image"
|
|
wwctl image exec ${IMG} -- /bin/bash -c "
|
|
set -e
|
|
dnf install -y -q epel-release
|
|
dnf install -y -q dnf-plugins-core
|
|
dnf config-manager --set-enabled crb
|
|
|
|
# kernel + everything PXE/wwinit needs
|
|
dnf install -y -q kernel kernel-modules dracut dracut-network \
|
|
NetworkManager openssh-server openssh-clients \
|
|
nfs-utils chrony ipmitool \
|
|
iproute iputils procps-ng util-linux passwd sudo vim less which tar
|
|
|
|
# Warewulf wwinit dracut module (initramfs is built inside the image)
|
|
curl -sLo /tmp/ww-dracut.rpm https://github.com/warewulf/warewulf/releases/download/v${WW_VER}/warewulf-dracut-${WW_VER}-1.el9.noarch.rpm
|
|
dnf install -y -q /tmp/ww-dracut.rpm
|
|
rm -f /tmp/ww-dracut.rpm
|
|
|
|
systemctl enable sshd NetworkManager chronyd
|
|
dnf clean all
|
|
"
|
|
|
|
echo "==> OpenHPC 3 compute stack (slurmd, munge, lmod)"
|
|
wwctl image exec ${IMG} -- /bin/bash -c "
|
|
set -e
|
|
dnf install -y -q http://repos.openhpc.community/OpenHPC/3/EL_9/x86_64/ohpc-release-3-1.el9.x86_64.rpm
|
|
dnf install -y -q ohpc-base-compute lmod-ohpc
|
|
dnf install -y -q slurm-slurmd-ohpc slurm-example-configs-ohpc munge
|
|
systemctl enable munge slurmd
|
|
dnf clean all
|
|
"
|
|
|
|
echo "==> slurmd spool directories (belong in the image, not an overlay)"
|
|
wwctl image exec ${IMG} -- /bin/bash -c "
|
|
mkdir -p /var/spool/slurm/d /var/spool/slurm/ctld
|
|
chown -R slurm:slurm /var/spool/slurm 2>/dev/null || true
|
|
chown root:root /var/spool/slurm/d
|
|
chmod 0755 /var/spool/slurm /var/spool/slurm/d
|
|
cat > /etc/tmpfiles.d/slurm.conf <<EOF
|
|
d /var/spool/slurm 0755 slurm slurm -
|
|
d /var/spool/slurm/d 0755 root root -
|
|
EOF
|
|
"
|
|
|
|
echo "==> kernel present in the image"
|
|
wwctl image exec ${IMG} -- /bin/bash -c "ls -1 /lib/modules; rpm -q kernel"
|
|
|
|
echo "==> image build (initramfs + compression)"
|
|
wwctl image build ${IMG}
|
|
|
|
echo "==> result"
|
|
wwctl image list
|
|
echo "NODE IMAGE OK"
|