Files
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

158 lines
6.0 KiB
Markdown

# 3. Warewulf
Version **4.7.0-1.el9**, installed from the official GitHub RPM — there is no
Warewulf package in the Rocky or EPEL repositories.
## 3.1 Concepts
| Concept | Meaning |
|---|---|
| **Image** | The root filesystem nodes run. Derived from an OCI image, kept as a chroot under `/var/lib/warewulf/chroots/<name>/rootfs`, served as `provision/images/<name>.img.gz`. |
| **Overlay** | A set of files written on top of the image. Files ending in `.ww` are rendered as Go templates. |
| **System overlay** | Applied at boot, **before** `/sbin/init`. Network, hostname and keytab live here. |
| **Runtime overlay** | Pulled by `wwclient` every 60 s. For things that can change without a reboot (`/etc/hosts`, `authorized_keys`). |
| **Profile** | Shared node settings. The `default` profile is applied to every node automatically. |
## 3.2 The image in this build
```
IMAGE NAME rocky-9
Source docker://ghcr.io/warewulf/warewulf-rockylinux:9
Kernel 5.14.0-687.41.1.el9_8.x86_64
Size 1.8 GB (raw) / 1.2 GB (compressed)
```
Added on top of the base image:
- `kernel`, `kernel-modules`, `dracut`, `dracut-network` — PXE boot
- `warewulf-dracut` — the `wwinit` dracut module (builds the initramfs inside the image)
- `NetworkManager`, `openssh-server`, `nfs-utils`, `chrony`
- OpenHPC: `ohpc-base-compute`, `lmod-ohpc`, `slurm-slurmd-ohpc`, `munge`
- Identity: `sssd`, `sssd-ipa`, `krb5-workstation`, `oddjob-mkhomedir`, `authselect`
- `/var/spool/slurm/{d,ctld}` plus a `tmpfiles.d/slurm.conf` entry
### Modifying the image
```bash
ssh head01
sudo wwctl image exec rocky-9 -- /bin/bash # shell inside the image
dnf install -y <package>
exit # rebuilds automatically on exit
sudo wwctl image build rocky-9 # or rebuild explicitly
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset
```
`image exec` opens a chroot; on exit Warewulf resets `machine-id` and repacks
the image. **Nodes keep running the old one until they are rebooted.**
## 3.3 Overlay inventory
| Overlay | Type | Files | Contents |
|---|---|---|---|
| `hpc` | site, system | 6 | `slurm.conf`, `cgroup.conf`, `munge.key` |
| `ipa` | site, system | 7 | `krb5.conf`, `ipa/ca.crt`, `sssd.conf.ww`, `ipa/default.conf.ww` |
| `ipa.wwnode1`…`4` | site, system | 2 | That node's `/etc/krb5.keytab` only |
Assignment (from the `default` profile):
```
SystemOverlay: wwinit, wwclient, hosts, fstab, hostname, ssh.host_keys,
issue, resolv, udev.netname, systemd.netname, ifcfg,
ifupdown, NetworkManager, wicked, ignition, hpc, ipa
RuntimeOverlay: hosts, ssh.authorized_keys
```
At node level **only** `ipa.wwnodeN` is added. Warewulf **merges** the profile
and node lists rather than overriding — repeating the full list at node level
applies every overlay twice.
### Editing overlays
```bash
sudo wwctl overlay import -p -o <overlay> <source> <target-path>
sudo wwctl overlay edit <overlay> <path> # open in an editor
sudo wwctl overlay list <overlay> # contents
sudo wwctl overlay show --render <node> <overlay> <path> # render a template
sudo wwctl overlay build # rebuild everything
```
**The `import -p` trap:** parent directories it creates end up `0500 root:root`.
Left uncorrected this locks `/etc` away from every non-root user on the node.
After any `import -p`:
```bash
sudo wwctl overlay chmod <overlay> /etc 0755
```
### Templates
Files ending in `.ww` are rendered with Go `text/template`. The most useful
variable is `{{.Id}}` — the node name (`wwnode1`). From the `ipa` overlay:
```
ipa_hostname = {{.Id}}.hpc.local
```
Others include `{{.NetDevs}}`, `{{.Tags}}`, `{{.ImageName}}` and
`{{.Kernel.Args}}`. The stock `resolv` overlay uses `{{$netdev.Tags.DNS1}}`.
Templating is **text-only**. Binary files such as keytabs cannot be templated —
which is precisely why per-node overlays were needed.
## 3.4 Node management
```bash
sudo wwctl node list # summary
sudo wwctl node list -a wwnode1 # all fields
sudo wwctl node status # provisioning stage / last seen
sudo wwctl node add wwnode5 --ipaddr 10.10.10.15 --hwaddr BC:24:11:E0:0A:15 \
--netdev eth0 --discoverable=false
sudo wwctl node set --yes wwnode5 -O ipa.wwnode5
sudo wwctl node delete --yes wwnode5
```
`wwctl node status` stages run `IPXE` → `KERNEL` → `IMAGE` → `SYSTEM OVERLAY` →
`RUNTIME OVERLAY`. In the final stage `LASTSEEN` should stay under 60 s; a
steadily climbing value means the node has died.
## 3.5 Network configuration
A node's address does **not** persist from DHCP. The sequence is:
1. During PXE, a temporary lease from `10.10.10.100-199`
2. The system overlay writes
`NetworkManager/system-connections/warewulf-default.conf` with
`interface-name=eth0`, `mac-address=…`, `address=10.10.10.11/24`
3. NM activates the connection and the node moves to its static address
**`--netdev` must be the real interface name on the node** (`eth0`). Warewulf
writes it into `interface-name=`; give it a name that does not exist and NM
never activates the connection, leaving the node stuck in the DHCP pool.
DNS comes from the `resolv` overlay, which reads **netdev tags**:
```yaml
network devices:
default:
device: eth0
tags:
DNS1: 10.10.10.5
DNSSEARCH: hpc.local
```
These tags **cannot** be written with `wwctl profile set --nettagadd` (it fails
silently) — `nodes.conf` is edited directly instead.
## 3.6 File locations
| Path | Contents |
|---|---|
| `/etc/warewulf/warewulf.conf` | Network, DHCP, TFTP, NFS, paths |
| `/etc/warewulf/nodes.conf` | Node and profile definitions (YAML) |
| `/var/lib/warewulf/chroots/<img>/rootfs` | Image chroot |
| `/var/lib/warewulf/provision/images/` | Served images (`.img`, `.img.gz`) |
| `/var/lib/warewulf/overlays/<name>/rootfs` | Site overlay sources |
| `/var/lib/warewulf/provision/overlays/<node>/` | Per-node compiled overlays |
| `/usr/share/warewulf/overlays/` | Distribution-supplied overlays |
| `/var/lib/tftpboot/warewulf/` | iPXE bootloaders |