A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
233 lines
8.0 KiB
Markdown
233 lines
8.0 KiB
Markdown
# 8. Troubleshooting
|
|
|
|
Section 8.1 lists the twelve failures actually hit while building this
|
|
environment, each with its diagnosis and the fix that is now baked into the
|
|
scripts. Section 8.2 is a general diagnostic guide.
|
|
|
|
## 8.1 Pitfalls encountered
|
|
|
|
### 1. 4 GB of node RAM is not enough
|
|
|
|
**Symptom:** node PXE-boots, downloads the image, then powers off silently.
|
|
`qm status` reports `stopped`. Nothing in any server log.
|
|
|
|
**Cause:** with `Root: initramfs` the whole rootfs (~1.8 GB) is unpacked into
|
|
RAM while the compressed image (~1.2 GB) is still held in memory. 4 GB is not
|
|
enough headroom.
|
|
|
|
**Fix:** nodes run 8 GB. Only visible on the serial console — this is why
|
|
capturing serial output early is worth the setup.
|
|
|
|
### 2. `--netdev default` strands nodes in the DHCP pool
|
|
|
|
**Symptom:** node boots fully and checks in to `warewulfd`, but sits on
|
|
`10.10.10.100` instead of its static address. Head cannot reach it by name.
|
|
|
|
**Cause:** Warewulf writes `interface-name=default` into the NetworkManager
|
|
connection. No interface is named `default` — the real one is `eth0` — so NM
|
|
never activates the connection and the DHCP lease stays.
|
|
|
|
**Fix:** `--netdev eth0`, the node's real interface name. Note the node-level
|
|
value overrides the profile, so both may need correcting.
|
|
|
|
### 3. `wwctl overlay import -p` creates parents as `0500 root:root`
|
|
|
|
**Symptom:** two separate failures. `/etc` becomes inaccessible to non-root
|
|
users on the node; and `munged` refuses to start with
|
|
*"Failed to check keyfile: Permission denied"*.
|
|
|
|
**Cause:** the auto-created parent directories get restrictive ownership and
|
|
mode. `munged` validates the **directory** containing its key, not just the key.
|
|
|
|
**Fix:** explicitly correct after every `import -p`:
|
|
|
|
```bash
|
|
wwctl overlay chmod hpc /etc 0755
|
|
wwctl overlay chown hpc /etc/munge munge:munge
|
|
wwctl overlay chmod hpc /etc/munge 0700
|
|
```
|
|
|
|
### 4. `slurmd` dies without `/var/spool/slurm/d`
|
|
|
|
**Symptom:** `fatal: mkdir(/var/spool/slurm/d): No such file or directory`.
|
|
|
|
**Cause:** the directory is runtime state and does not exist in the OCI base
|
|
image.
|
|
|
|
**Fix:** created **inside the image**, not in an overlay, plus a
|
|
`tmpfiles.d` entry so it is re-asserted at every boot.
|
|
|
|
### 5. cloud-init's `authorized_keys` locks root SSH to the nodes
|
|
|
|
**Symptom:** `ssh root@wwnode1` from the head is refused, or prints
|
|
*"Please login as the user ilker rather than root"*.
|
|
|
|
**Cause:** cloud-init writes a forced-command wrapper into
|
|
`/root/.ssh/authorized_keys` on the head. The `ssh.authorized_keys` overlay
|
|
copies that file verbatim to every node.
|
|
|
|
**Fix:** strip the wrapper on the head and generate a dedicated cluster key
|
|
pair for head→node access.
|
|
|
|
### 6. `--kernelargs` splits on commas
|
|
|
|
**Symptom:** `console=ttyS0,115200` arrives at the kernel as two arguments —
|
|
`console=ttyS0` and a stray `115200`.
|
|
|
|
**Cause:** `wwctl` treats the value as a comma-separated list.
|
|
|
|
**Fix:** drop the baud rate (`console=ttyS0`). Baud is meaningless on a QEMU
|
|
serial socket anyway.
|
|
|
|
### 7. `wwctl profile set --nettagadd` silently does nothing
|
|
|
|
**Symptom:** command succeeds, exit code 0, no error — and `nodes.conf` is
|
|
unchanged. Nodes end up with an empty `/etc/resolv.conf`.
|
|
|
|
**Cause:** unimplemented or broken in this version. Because the `resolv`
|
|
overlay reads DNS from netdev tags (`DNS1`, `DNSSEARCH`), the nodes got no DNS
|
|
at all — and without DNS neither SSSD nor NFS-by-name works.
|
|
|
|
**Fix:** write the tags into `nodes.conf` directly with `python3-pyyaml`.
|
|
This is the most dangerous item on the list precisely because it reports
|
|
success.
|
|
|
|
### 8. Node-level overlay lists append to the profile, not replace it
|
|
|
|
**Symptom:** `wwctl node list -a` shows every overlay twice.
|
|
|
|
**Cause:** Warewulf **merges** profile and node overlay lists.
|
|
|
|
**Fix:** give the node only what is unique to it (`ipa.wwnodeN`) and let the
|
|
profile supply the rest.
|
|
|
|
### 9. cloud-init pins DNS in two places
|
|
|
|
**Symptom:** `nmcli con mod … ipv4.dns 10.10.10.5` appears to work, but
|
|
`/etc/resolv.conf` never changes.
|
|
|
|
**Cause:** cloud-init writes a static `/etc/resolv.conf` **and** sets
|
|
`dns = none` in `/etc/NetworkManager/conf.d/99-cloud-init.conf`, so NM never
|
|
manages the file.
|
|
|
|
**Fix:** set `manage_resolv_conf: false` for cloud-init, override NM with
|
|
`dns = default`, delete the stale `resolv.conf`, restart NetworkManager.
|
|
|
|
### 10. `usermod -m` refuses while the user is logged in
|
|
|
|
**Symptom:** `usermod: user ilker is currently used by process NNNN` when
|
|
moving the local admin's home out of `/home` before the NFS mount.
|
|
|
|
**Cause:** your own SSH session owns the account.
|
|
|
|
**Fix:** `cp -a` the directory and edit `/etc/passwd` with `sed`.
|
|
|
|
### 11. `ipa user-show --raw` returns lowercase attribute names
|
|
|
|
**Symptom:** `install: invalid user ''` while pre-creating home directories.
|
|
|
|
**Cause:** the attribute is `uidnumber`, not `uidNumber`. A case-sensitive
|
|
`awk` pattern matched nothing and produced an empty uid.
|
|
|
|
**Fix:** match `uidnumber` / `gidnumber`, and use `--all --raw`.
|
|
|
|
### 12. Local accounts shadow IPA accounts
|
|
|
|
**Symptom:** the same username resolves to different uids on different hosts;
|
|
NFS home ownership looks wrong.
|
|
|
|
**Cause:** `nsswitch.conf` resolves `files` before `sss`. cloud-init creates a
|
|
local `ilker` (uid 1000) everywhere, and `wwctl image syncuser` had copied the
|
|
local `hpcuser` into the node image.
|
|
|
|
**Fix:** no IPA user named `ilker`; local `hpcuser` deleted from both the head
|
|
and the image. See [04-identity §4.5](04-identity.md#why-there-is-no-ipa-user-named-ilker).
|
|
|
|
### Bonus: `tr | head` kills a `set -euo pipefail` script
|
|
|
|
`DM_PW=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 20)` — `tr` receives
|
|
SIGPIPE when `head` exits, `pipefail` propagates the non-zero status and `set
|
|
-e` aborts the script. Use `head -c 40 /dev/urandom | base64 | tr -dc … | cut`
|
|
instead.
|
|
|
|
## 8.2 Diagnostic guide
|
|
|
|
### A node will not boot
|
|
|
|
```bash
|
|
ssh head01 'sudo wwctl node status' # which stage did it reach?
|
|
ssh head01 'sudo journalctl -u dhcpd -n 30' # did it get a lease?
|
|
ssh head01 'sudo journalctl -u warewulfd -n 30'
|
|
ssh proxmox 'qm terminal 301' # serial console
|
|
```
|
|
|
|
Stage tells you where to look: no `IPXE` → DHCP/TFTP; stuck at `KERNEL` →
|
|
image download; reached `RUNTIME OVERLAY` but unreachable → networking or the
|
|
node died after boot (check RAM).
|
|
|
|
### Node boots but has the wrong IP
|
|
|
|
Check the rendered NetworkManager connection:
|
|
|
|
```bash
|
|
sudo wwctl overlay show --render wwnode1 NetworkManager \
|
|
/etc/NetworkManager/system-connections/ww4-managed.ww
|
|
```
|
|
|
|
`interface-name=` must match a real interface on the node (pitfall 2).
|
|
|
|
### IPA users do not resolve on a node
|
|
|
|
```bash
|
|
ssh head01 'sudo ssh root@wwnode1 "
|
|
cat /etc/resolv.conf # must point at 10.10.10.5
|
|
systemctl status sssd
|
|
klist -k # host principal present?
|
|
kinit -k host/wwnode1.hpc.local && klist
|
|
id hpcuser
|
|
journalctl -u sssd -n 40"'
|
|
```
|
|
|
|
Ordered checks: DNS → keytab → Kerberos → SSSD. A failure at any step makes
|
|
everything after it fail; fix them in order.
|
|
|
|
Also confirm `/etc/sssd/sssd.conf` is mode `0600` — SSSD refuses to start
|
|
otherwise and the message is easy to miss.
|
|
|
|
### `/home` will not mount
|
|
|
|
```bash
|
|
ssh idm1 'sudo exportfs -v' # is it exported to 10.10.10.0/24?
|
|
ssh head01 'sudo ssh root@wwnode1 "
|
|
getent hosts idm1.hpc.local # DNS working?
|
|
showmount -e 10.10.10.5
|
|
mount -a; dmesg | tail"'
|
|
```
|
|
|
|
Files showing as `nobody:nobody` means `Domain = hpc.local` is missing from
|
|
`/etc/idmapd.conf` somewhere.
|
|
|
|
### Slurm shows nodes as `down` or `unk`
|
|
|
|
```bash
|
|
sinfo -R # reason
|
|
ssh head01 'sudo ssh root@wwnode1 "
|
|
systemctl status slurmd munge
|
|
journalctl -u slurmd -n 30"'
|
|
scontrol update NodeName=wwnode1 State=RESUME
|
|
```
|
|
|
|
Most common causes: `munge.key` mismatch or permissions (pitfall 3), and
|
|
`/var/spool/slurm/d` missing (pitfall 4).
|
|
|
|
### FreeIPA will not start
|
|
|
|
```bash
|
|
ssh idm1 'sudo ipactl status'
|
|
ssh idm1 'sudo ipactl restart'
|
|
ssh idm1 'sudo journalctl -u dirsrv@HPC-LOCAL -n 50'
|
|
```
|
|
|
|
FreeIPA is sensitive to time skew and to its own hostname resolving correctly.
|
|
Check `chronyd` and that `idm1.hpc.local` still resolves to 10.10.10.5.
|