Files
werewulf/docs/08-troubleshooting.md
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

8.0 KiB

8. Troubleshooting

Section 8.1 lists the twelve failures actually hit while building this environment, each with its diagnosis and the fix that is now baked into the scripts. Section 8.2 is a general diagnostic guide.

8.1 Pitfalls encountered

1. 4 GB of node RAM is not enough

Symptom: node PXE-boots, downloads the image, then powers off silently. qm status reports stopped. Nothing in any server log.

Cause: with Root: initramfs the whole rootfs (~1.8 GB) is unpacked into RAM while the compressed image (~1.2 GB) is still held in memory. 4 GB is not enough headroom.

Fix: nodes run 8 GB. Only visible on the serial console — this is why capturing serial output early is worth the setup.

2. --netdev default strands nodes in the DHCP pool

Symptom: node boots fully and checks in to warewulfd, but sits on 10.10.10.100 instead of its static address. Head cannot reach it by name.

Cause: Warewulf writes interface-name=default into the NetworkManager connection. No interface is named default — the real one is eth0 — so NM never activates the connection and the DHCP lease stays.

Fix: --netdev eth0, the node's real interface name. Note the node-level value overrides the profile, so both may need correcting.

3. wwctl overlay import -p creates parents as 0500 root:root

Symptom: two separate failures. /etc becomes inaccessible to non-root users on the node; and munged refuses to start with "Failed to check keyfile: Permission denied".

Cause: the auto-created parent directories get restrictive ownership and mode. munged validates the directory containing its key, not just the key.

Fix: explicitly correct after every import -p:

wwctl overlay chmod hpc /etc 0755
wwctl overlay chown hpc /etc/munge munge:munge
wwctl overlay chmod hpc /etc/munge 0700

4. slurmd dies without /var/spool/slurm/d

Symptom: fatal: mkdir(/var/spool/slurm/d): No such file or directory.

Cause: the directory is runtime state and does not exist in the OCI base image.

Fix: created inside the image, not in an overlay, plus a tmpfiles.d entry so it is re-asserted at every boot.

5. cloud-init's authorized_keys locks root SSH to the nodes

Symptom: ssh root@wwnode1 from the head is refused, or prints "Please login as the user ilker rather than root".

Cause: cloud-init writes a forced-command wrapper into /root/.ssh/authorized_keys on the head. The ssh.authorized_keys overlay copies that file verbatim to every node.

Fix: strip the wrapper on the head and generate a dedicated cluster key pair for head→node access.

6. --kernelargs splits on commas

Symptom: console=ttyS0,115200 arrives at the kernel as two arguments — console=ttyS0 and a stray 115200.

Cause: wwctl treats the value as a comma-separated list.

Fix: drop the baud rate (console=ttyS0). Baud is meaningless on a QEMU serial socket anyway.

7. wwctl profile set --nettagadd silently does nothing

Symptom: command succeeds, exit code 0, no error — and nodes.conf is unchanged. Nodes end up with an empty /etc/resolv.conf.

Cause: unimplemented or broken in this version. Because the resolv overlay reads DNS from netdev tags (DNS1, DNSSEARCH), the nodes got no DNS at all — and without DNS neither SSSD nor NFS-by-name works.

Fix: write the tags into nodes.conf directly with python3-pyyaml. This is the most dangerous item on the list precisely because it reports success.

8. Node-level overlay lists append to the profile, not replace it

Symptom: wwctl node list -a shows every overlay twice.

Cause: Warewulf merges profile and node overlay lists.

Fix: give the node only what is unique to it (ipa.wwnodeN) and let the profile supply the rest.

9. cloud-init pins DNS in two places

Symptom: nmcli con mod … ipv4.dns 10.10.10.5 appears to work, but /etc/resolv.conf never changes.

Cause: cloud-init writes a static /etc/resolv.conf and sets dns = none in /etc/NetworkManager/conf.d/99-cloud-init.conf, so NM never manages the file.

Fix: set manage_resolv_conf: false for cloud-init, override NM with dns = default, delete the stale resolv.conf, restart NetworkManager.

10. usermod -m refuses while the user is logged in

Symptom: usermod: user ilker is currently used by process NNNN when moving the local admin's home out of /home before the NFS mount.

Cause: your own SSH session owns the account.

Fix: cp -a the directory and edit /etc/passwd with sed.

11. ipa user-show --raw returns lowercase attribute names

Symptom: install: invalid user '' while pre-creating home directories.

Cause: the attribute is uidnumber, not uidNumber. A case-sensitive awk pattern matched nothing and produced an empty uid.

Fix: match uidnumber / gidnumber, and use --all --raw.

12. Local accounts shadow IPA accounts

Symptom: the same username resolves to different uids on different hosts; NFS home ownership looks wrong.

Cause: nsswitch.conf resolves files before sss. cloud-init creates a local ilker (uid 1000) everywhere, and wwctl image syncuser had copied the local hpcuser into the node image.

Fix: no IPA user named ilker; local hpcuser deleted from both the head and the image. See 04-identity §4.5.

Bonus: tr | head kills a set -euo pipefail script

DM_PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 20) — tr receives SIGPIPE when head exits, pipefail propagates the non-zero status and set -e aborts the script. Use head -c 40 /dev/urandom | base64 | tr -dc … | cut instead.

8.2 Diagnostic guide

A node will not boot

ssh head01 'sudo wwctl node status'          # which stage did it reach?
ssh head01 'sudo journalctl -u dhcpd -n 30'  # did it get a lease?
ssh head01 'sudo journalctl -u warewulfd -n 30'
ssh proxmox 'qm terminal 301'                # serial console

Stage tells you where to look: no IPXE → DHCP/TFTP; stuck at KERNEL → image download; reached RUNTIME OVERLAY but unreachable → networking or the node died after boot (check RAM).

Node boots but has the wrong IP

Check the rendered NetworkManager connection:

sudo wwctl overlay show --render wwnode1 NetworkManager \
     /etc/NetworkManager/system-connections/ww4-managed.ww

interface-name= must match a real interface on the node (pitfall 2).

IPA users do not resolve on a node

ssh head01 'sudo ssh root@wwnode1 "
  cat /etc/resolv.conf          # must point at 10.10.10.5
  systemctl status sssd
  klist -k                      # host principal present?
  kinit -k host/wwnode1.hpc.local && klist
  id hpcuser
  journalctl -u sssd -n 40"'

Ordered checks: DNS → keytab → Kerberos → SSSD. A failure at any step makes everything after it fail; fix them in order.

Also confirm /etc/sssd/sssd.conf is mode 0600 — SSSD refuses to start otherwise and the message is easy to miss.

/home will not mount

ssh idm1 'sudo exportfs -v'                  # is it exported to 10.10.10.0/24?
ssh head01 'sudo ssh root@wwnode1 "
  getent hosts idm1.hpc.local                # DNS working?
  showmount -e 10.10.10.5
  mount -a; dmesg | tail"'

Files showing as nobody:nobody means Domain = hpc.local is missing from /etc/idmapd.conf somewhere.

Slurm shows nodes as down or unk

sinfo -R                                     # reason
ssh head01 'sudo ssh root@wwnode1 "
  systemctl status slurmd munge
  journalctl -u slurmd -n 30"'
scontrol update NodeName=wwnode1 State=RESUME

Most common causes: munge.key mismatch or permissions (pitfall 3), and /var/spool/slurm/d missing (pitfall 4).

FreeIPA will not start

ssh idm1 'sudo ipactl status'
ssh idm1 'sudo ipactl restart'
ssh idm1 'sudo journalctl -u dirsrv@HPC-LOCAL -n 50'

FreeIPA is sensitive to time skew and to its own hostname resolving correctly. Check chronyd and that idm1.hpc.local still resolves to 10.10.10.5.