A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
6.1 KiB
1. Architecture
1.1 Overview
A three-tier HPC test environment on the Proxmox node hv01:
| Tier | Host | Role |
|---|---|---|
| Provisioning | head01 (VM 300) |
Warewulf, DHCP/TFTP/iPXE, Slurm controller, NAT gateway |
| Identity + storage | idm1 (VM 305) |
FreeIPA (LDAP/Kerberos/CA/DNS), NFS home server |
| Compute | wwnode1-4 (VM 301-304) |
Diskless, PXE-booted stateless nodes |
1.2 Network topology
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
│ │
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ Rocky 9.8 │ │ Rocky 9.8 │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ │ │ │
│ warewulfd dhcpd │ │ FreeIPA (hpc.local) │
│ tftp/iPXE slurmctld│ │ 389-ds · KDC · DNS │
│ munge NAT gw │ │ Dogtag CA · httpd │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
│ │
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
1.3 Why vmbr6 has no IP on the Proxmox side
Existing bridges such as vmbr3 and vmbr5 use the Proxmox host as gateway
and NAT router. vmbr6 deliberately breaks that pattern.
Warewulf's provisioning model requires owning the entire subnet — DHCP (with PXE options), TFTP, the iPXE chain, and node address management. If the Proxmox host also ran DHCP/DNS on that subnet, two authorities would collide.
So vmbr6 is a pure L2 switch: bridge-ports none, no IP, no NAT rules. The
subnet's gateway, DHCP server and TFTP server is head01. Nodes reach the
outside world through firewalld masquerade on head01.
Side benefit: cluster traffic never leaks onto vmbr0. The other 15 VMs on
hv01 (webserver, crm6, kolla, …) never see the PXE broadcasts.
1.4 Why idm1 is dual-homed
This is the most consequential design decision in the build.
The server's primary identity lives on vmbr0: 10.0.0.31, used for SSH and
administration. But in DNS:
idm1.hpc.local. A 10.10.10.5 ← service address (cluster fabric)
idm1-mgmt.hpc.local. A 10.0.0.31 ← management address
Why the obvious alternative was rejected: the nodes are isolated on
vmbr6. Their only route to 10.0.0.31 is head01's masquerade — at which
point all four nodes appear to the server as 10.0.0.30. Consequences:
- Kerberos: address mismatch between KDC and client, reverse (PTR) lookups resolving to the wrong name, service-ticket validation failures.
- NFS: export rules are written per client IP. With four nodes collapsed into one address, per-node authorization becomes impossible.
- Auditing: logs cannot distinguish which node did what.
A third option, not taken: disable NAT and route 10.10.10.0/24 instead.
Packets would then flow node → head01 → idm1(eth0) while replies return
idm1(eth1) → node directly — asymmetric routing, which is fragile under
rp_filter and connection tracking.
Dual-homing means every client — including head01, which also holds
10.10.10.1 — reaches IPA directly, without NAT, over a symmetric path.
1.5 The stateless node model
Nodes have no disk. In Proxmox they carry boot: order=net0 and no scsi0
at all. The boot chain:
1. iPXE ROM → DHCP (temporary lease from 10.10.10.100-199)
2. DHCP reply: next-server 10.10.10.1, filename ipxe-snponly-x86_64.efi
3. iPXE → http://10.10.10.1:9873/ipxe/<mac> (warewulfd)
4. Kernel downloaded (vmlinuz)
5. Compressed rootfs downloaded (rocky-9.img.gz, ~1.2 GB)
6. Unpacked into RAM as initramfs (~1.8 GB)
7. System overlay applied (network, hostname, SSSD, keytab, fstab…)
8. /sbin/init → systemd
9. wwclient pulls the runtime overlay every 60 s
The static IP does not come from DHCP — in step 6 the system overlay writes
a NetworkManager connection file and the node moves from 10.10.10.100 to
10.10.10.1x.
Two practical consequences:
- High RAM usage. The rootfs lives in memory, not on disk. 8 GB is required (see 08-troubleshooting).
- Nothing persists. Every reboot resets the node. Anything that must survive — the Kerberos keytab, SSSD configuration — has to arrive via an overlay. This constraint shaped the entire identity design (see 04-identity).
1.6 Data flows
| Flow | Path | Protocol |
|---|---|---|
| Node boot | node → head01 | DHCP, TFTP, HTTP :9873 |
| User/group lookup | node → idm1 | LDAP :389 (SSSD, TLS) |
| Authentication | node → idm1 | Kerberos :88, :464 |
| Name resolution | node → idm1 | DNS :53 |
| Home directories | node → idm1 | NFSv4 :2049 |
| OpenHPC modules | node → head01 | NFSv4 :2049 (ro) |
| Job dispatch | head01 → node | Slurm :6817/:6818 |
| MPI (rank↔rank) | node ↔ node | TCP, direct L2 |
| Node → internet | node → head01 → vmbr0 | NAT (masquerade) |