Files
werewulf/docs/08-troubleshooting.md
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

233 lines
8.0 KiB
Markdown

# 8. Troubleshooting
Section 8.1 lists the twelve failures actually hit while building this
environment, each with its diagnosis and the fix that is now baked into the
scripts. Section 8.2 is a general diagnostic guide.
## 8.1 Pitfalls encountered
### 1. 4 GB of node RAM is not enough
**Symptom:** node PXE-boots, downloads the image, then powers off silently.
`qm status` reports `stopped`. Nothing in any server log.
**Cause:** with `Root: initramfs` the whole rootfs (~1.8 GB) is unpacked into
RAM while the compressed image (~1.2 GB) is still held in memory. 4 GB is not
enough headroom.
**Fix:** nodes run 8 GB. Only visible on the serial console — this is why
capturing serial output early is worth the setup.
### 2. `--netdev default` strands nodes in the DHCP pool
**Symptom:** node boots fully and checks in to `warewulfd`, but sits on
`10.10.10.100` instead of its static address. Head cannot reach it by name.
**Cause:** Warewulf writes `interface-name=default` into the NetworkManager
connection. No interface is named `default` — the real one is `eth0` — so NM
never activates the connection and the DHCP lease stays.
**Fix:** `--netdev eth0`, the node's real interface name. Note the node-level
value overrides the profile, so both may need correcting.
### 3. `wwctl overlay import -p` creates parents as `0500 root:root`
**Symptom:** two separate failures. `/etc` becomes inaccessible to non-root
users on the node; and `munged` refuses to start with
*"Failed to check keyfile: Permission denied"*.
**Cause:** the auto-created parent directories get restrictive ownership and
mode. `munged` validates the **directory** containing its key, not just the key.
**Fix:** explicitly correct after every `import -p`:
```bash
wwctl overlay chmod hpc /etc 0755
wwctl overlay chown hpc /etc/munge munge:munge
wwctl overlay chmod hpc /etc/munge 0700
```
### 4. `slurmd` dies without `/var/spool/slurm/d`
**Symptom:** `fatal: mkdir(/var/spool/slurm/d): No such file or directory`.
**Cause:** the directory is runtime state and does not exist in the OCI base
image.
**Fix:** created **inside the image**, not in an overlay, plus a
`tmpfiles.d` entry so it is re-asserted at every boot.
### 5. cloud-init's `authorized_keys` locks root SSH to the nodes
**Symptom:** `ssh root@wwnode1` from the head is refused, or prints
*"Please login as the user ilker rather than root"*.
**Cause:** cloud-init writes a forced-command wrapper into
`/root/.ssh/authorized_keys` on the head. The `ssh.authorized_keys` overlay
copies that file verbatim to every node.
**Fix:** strip the wrapper on the head and generate a dedicated cluster key
pair for head→node access.
### 6. `--kernelargs` splits on commas
**Symptom:** `console=ttyS0,115200` arrives at the kernel as two arguments —
`console=ttyS0` and a stray `115200`.
**Cause:** `wwctl` treats the value as a comma-separated list.
**Fix:** drop the baud rate (`console=ttyS0`). Baud is meaningless on a QEMU
serial socket anyway.
### 7. `wwctl profile set --nettagadd` silently does nothing
**Symptom:** command succeeds, exit code 0, no error — and `nodes.conf` is
unchanged. Nodes end up with an empty `/etc/resolv.conf`.
**Cause:** unimplemented or broken in this version. Because the `resolv`
overlay reads DNS from netdev tags (`DNS1`, `DNSSEARCH`), the nodes got no DNS
at all — and without DNS neither SSSD nor NFS-by-name works.
**Fix:** write the tags into `nodes.conf` directly with `python3-pyyaml`.
This is the most dangerous item on the list precisely because it reports
success.
### 8. Node-level overlay lists append to the profile, not replace it
**Symptom:** `wwctl node list -a` shows every overlay twice.
**Cause:** Warewulf **merges** profile and node overlay lists.
**Fix:** give the node only what is unique to it (`ipa.wwnodeN`) and let the
profile supply the rest.
### 9. cloud-init pins DNS in two places
**Symptom:** `nmcli con mod … ipv4.dns 10.10.10.5` appears to work, but
`/etc/resolv.conf` never changes.
**Cause:** cloud-init writes a static `/etc/resolv.conf` **and** sets
`dns = none` in `/etc/NetworkManager/conf.d/99-cloud-init.conf`, so NM never
manages the file.
**Fix:** set `manage_resolv_conf: false` for cloud-init, override NM with
`dns = default`, delete the stale `resolv.conf`, restart NetworkManager.
### 10. `usermod -m` refuses while the user is logged in
**Symptom:** `usermod: user ilker is currently used by process NNNN` when
moving the local admin's home out of `/home` before the NFS mount.
**Cause:** your own SSH session owns the account.
**Fix:** `cp -a` the directory and edit `/etc/passwd` with `sed`.
### 11. `ipa user-show --raw` returns lowercase attribute names
**Symptom:** `install: invalid user ''` while pre-creating home directories.
**Cause:** the attribute is `uidnumber`, not `uidNumber`. A case-sensitive
`awk` pattern matched nothing and produced an empty uid.
**Fix:** match `uidnumber` / `gidnumber`, and use `--all --raw`.
### 12. Local accounts shadow IPA accounts
**Symptom:** the same username resolves to different uids on different hosts;
NFS home ownership looks wrong.
**Cause:** `nsswitch.conf` resolves `files` before `sss`. cloud-init creates a
local `ilker` (uid 1000) everywhere, and `wwctl image syncuser` had copied the
local `hpcuser` into the node image.
**Fix:** no IPA user named `ilker`; local `hpcuser` deleted from both the head
and the image. See [04-identity §4.5](04-identity.md#why-there-is-no-ipa-user-named-ilker).
### Bonus: `tr | head` kills a `set -euo pipefail` script
`DM_PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 20)` — `tr` receives
SIGPIPE when `head` exits, `pipefail` propagates the non-zero status and `set
-e` aborts the script. Use `head -c 40 /dev/urandom | base64 | tr -dc … | cut`
instead.
## 8.2 Diagnostic guide
### A node will not boot
```bash
ssh head01 'sudo wwctl node status' # which stage did it reach?
ssh head01 'sudo journalctl -u dhcpd -n 30' # did it get a lease?
ssh head01 'sudo journalctl -u warewulfd -n 30'
ssh proxmox 'qm terminal 301' # serial console
```
Stage tells you where to look: no `IPXE` → DHCP/TFTP; stuck at `KERNEL` →
image download; reached `RUNTIME OVERLAY` but unreachable → networking or the
node died after boot (check RAM).
### Node boots but has the wrong IP
Check the rendered NetworkManager connection:
```bash
sudo wwctl overlay show --render wwnode1 NetworkManager \
/etc/NetworkManager/system-connections/ww4-managed.ww
```
`interface-name=` must match a real interface on the node (pitfall 2).
### IPA users do not resolve on a node
```bash
ssh head01 'sudo ssh root@wwnode1 "
cat /etc/resolv.conf # must point at 10.10.10.5
systemctl status sssd
klist -k # host principal present?
kinit -k host/wwnode1.hpc.local && klist
id hpcuser
journalctl -u sssd -n 40"'
```
Ordered checks: DNS → keytab → Kerberos → SSSD. A failure at any step makes
everything after it fail; fix them in order.
Also confirm `/etc/sssd/sssd.conf` is mode `0600` — SSSD refuses to start
otherwise and the message is easy to miss.
### `/home` will not mount
```bash
ssh idm1 'sudo exportfs -v' # is it exported to 10.10.10.0/24?
ssh head01 'sudo ssh root@wwnode1 "
getent hosts idm1.hpc.local # DNS working?
showmount -e 10.10.10.5
mount -a; dmesg | tail"'
```
Files showing as `nobody:nobody` means `Domain = hpc.local` is missing from
`/etc/idmapd.conf` somewhere.
### Slurm shows nodes as `down` or `unk`
```bash
sinfo -R # reason
ssh head01 'sudo ssh root@wwnode1 "
systemctl status slurmd munge
journalctl -u slurmd -n 30"'
scontrol update NodeName=wwnode1 State=RESUME
```
Most common causes: `munge.key` mismatch or permissions (pitfall 3), and
`/var/spool/slurm/d` missing (pitfall 4).
### FreeIPA will not start
```bash
ssh idm1 'sudo ipactl status'
ssh idm1 'sudo ipactl restart'
ssh idm1 'sudo journalctl -u dirsrv@HPC-LOCAL -n 50'
```
FreeIPA is sensitive to time skew and to its own hostname resolving correctly.
Check `chronyd` and that `idm1.hpc.local` still resolves to 10.10.10.5.