A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
8.0 KiB
8. Troubleshooting
Section 8.1 lists the twelve failures actually hit while building this environment, each with its diagnosis and the fix that is now baked into the scripts. Section 8.2 is a general diagnostic guide.
8.1 Pitfalls encountered
1. 4 GB of node RAM is not enough
Symptom: node PXE-boots, downloads the image, then powers off silently.
qm status reports stopped. Nothing in any server log.
Cause: with Root: initramfs the whole rootfs (~1.8 GB) is unpacked into
RAM while the compressed image (~1.2 GB) is still held in memory. 4 GB is not
enough headroom.
Fix: nodes run 8 GB. Only visible on the serial console — this is why capturing serial output early is worth the setup.
2. --netdev default strands nodes in the DHCP pool
Symptom: node boots fully and checks in to warewulfd, but sits on
10.10.10.100 instead of its static address. Head cannot reach it by name.
Cause: Warewulf writes interface-name=default into the NetworkManager
connection. No interface is named default — the real one is eth0 — so NM
never activates the connection and the DHCP lease stays.
Fix: --netdev eth0, the node's real interface name. Note the node-level
value overrides the profile, so both may need correcting.
3. wwctl overlay import -p creates parents as 0500 root:root
Symptom: two separate failures. /etc becomes inaccessible to non-root
users on the node; and munged refuses to start with
"Failed to check keyfile: Permission denied".
Cause: the auto-created parent directories get restrictive ownership and
mode. munged validates the directory containing its key, not just the key.
Fix: explicitly correct after every import -p:
wwctl overlay chmod hpc /etc 0755
wwctl overlay chown hpc /etc/munge munge:munge
wwctl overlay chmod hpc /etc/munge 0700
4. slurmd dies without /var/spool/slurm/d
Symptom: fatal: mkdir(/var/spool/slurm/d): No such file or directory.
Cause: the directory is runtime state and does not exist in the OCI base image.
Fix: created inside the image, not in an overlay, plus a
tmpfiles.d entry so it is re-asserted at every boot.
5. cloud-init's authorized_keys locks root SSH to the nodes
Symptom: ssh root@wwnode1 from the head is refused, or prints
"Please login as the user ilker rather than root".
Cause: cloud-init writes a forced-command wrapper into
/root/.ssh/authorized_keys on the head. The ssh.authorized_keys overlay
copies that file verbatim to every node.
Fix: strip the wrapper on the head and generate a dedicated cluster key pair for head→node access.
6. --kernelargs splits on commas
Symptom: console=ttyS0,115200 arrives at the kernel as two arguments —
console=ttyS0 and a stray 115200.
Cause: wwctl treats the value as a comma-separated list.
Fix: drop the baud rate (console=ttyS0). Baud is meaningless on a QEMU
serial socket anyway.
7. wwctl profile set --nettagadd silently does nothing
Symptom: command succeeds, exit code 0, no error — and nodes.conf is
unchanged. Nodes end up with an empty /etc/resolv.conf.
Cause: unimplemented or broken in this version. Because the resolv
overlay reads DNS from netdev tags (DNS1, DNSSEARCH), the nodes got no DNS
at all — and without DNS neither SSSD nor NFS-by-name works.
Fix: write the tags into nodes.conf directly with python3-pyyaml.
This is the most dangerous item on the list precisely because it reports
success.
8. Node-level overlay lists append to the profile, not replace it
Symptom: wwctl node list -a shows every overlay twice.
Cause: Warewulf merges profile and node overlay lists.
Fix: give the node only what is unique to it (ipa.wwnodeN) and let the
profile supply the rest.
9. cloud-init pins DNS in two places
Symptom: nmcli con mod … ipv4.dns 10.10.10.5 appears to work, but
/etc/resolv.conf never changes.
Cause: cloud-init writes a static /etc/resolv.conf and sets
dns = none in /etc/NetworkManager/conf.d/99-cloud-init.conf, so NM never
manages the file.
Fix: set manage_resolv_conf: false for cloud-init, override NM with
dns = default, delete the stale resolv.conf, restart NetworkManager.
10. usermod -m refuses while the user is logged in
Symptom: usermod: user ilker is currently used by process NNNN when
moving the local admin's home out of /home before the NFS mount.
Cause: your own SSH session owns the account.
Fix: cp -a the directory and edit /etc/passwd with sed.
11. ipa user-show --raw returns lowercase attribute names
Symptom: install: invalid user '' while pre-creating home directories.
Cause: the attribute is uidnumber, not uidNumber. A case-sensitive
awk pattern matched nothing and produced an empty uid.
Fix: match uidnumber / gidnumber, and use --all --raw.
12. Local accounts shadow IPA accounts
Symptom: the same username resolves to different uids on different hosts; NFS home ownership looks wrong.
Cause: nsswitch.conf resolves files before sss. cloud-init creates a
local ilker (uid 1000) everywhere, and wwctl image syncuser had copied the
local hpcuser into the node image.
Fix: no IPA user named ilker; local hpcuser deleted from both the head
and the image. See 04-identity §4.5.
Bonus: tr | head kills a set -euo pipefail script
DM_PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 20) — tr receives
SIGPIPE when head exits, pipefail propagates the non-zero status and set -e aborts the script. Use head -c 40 /dev/urandom | base64 | tr -dc … | cut
instead.
8.2 Diagnostic guide
A node will not boot
ssh head01 'sudo wwctl node status' # which stage did it reach?
ssh head01 'sudo journalctl -u dhcpd -n 30' # did it get a lease?
ssh head01 'sudo journalctl -u warewulfd -n 30'
ssh proxmox 'qm terminal 301' # serial console
Stage tells you where to look: no IPXE → DHCP/TFTP; stuck at KERNEL →
image download; reached RUNTIME OVERLAY but unreachable → networking or the
node died after boot (check RAM).
Node boots but has the wrong IP
Check the rendered NetworkManager connection:
sudo wwctl overlay show --render wwnode1 NetworkManager \
/etc/NetworkManager/system-connections/ww4-managed.ww
interface-name= must match a real interface on the node (pitfall 2).
IPA users do not resolve on a node
ssh head01 'sudo ssh root@wwnode1 "
cat /etc/resolv.conf # must point at 10.10.10.5
systemctl status sssd
klist -k # host principal present?
kinit -k host/wwnode1.hpc.local && klist
id hpcuser
journalctl -u sssd -n 40"'
Ordered checks: DNS → keytab → Kerberos → SSSD. A failure at any step makes everything after it fail; fix them in order.
Also confirm /etc/sssd/sssd.conf is mode 0600 — SSSD refuses to start
otherwise and the message is easy to miss.
/home will not mount
ssh idm1 'sudo exportfs -v' # is it exported to 10.10.10.0/24?
ssh head01 'sudo ssh root@wwnode1 "
getent hosts idm1.hpc.local # DNS working?
showmount -e 10.10.10.5
mount -a; dmesg | tail"'
Files showing as nobody:nobody means Domain = hpc.local is missing from
/etc/idmapd.conf somewhere.
Slurm shows nodes as down or unk
sinfo -R # reason
ssh head01 'sudo ssh root@wwnode1 "
systemctl status slurmd munge
journalctl -u slurmd -n 30"'
scontrol update NodeName=wwnode1 State=RESUME
Most common causes: munge.key mismatch or permissions (pitfall 3), and
/var/spool/slurm/d missing (pitfall 4).
FreeIPA will not start
ssh idm1 'sudo ipactl status'
ssh idm1 'sudo ipactl restart'
ssh idm1 'sudo journalctl -u dirsrv@HPC-LOCAL -n 50'
FreeIPA is sensitive to time skew and to its own hostname resolving correctly.
Check chronyd and that idm1.hpc.local still resolves to 10.10.10.5.