Files
werewulf/docs/01-architecture.md
T
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

6.1 KiB

1. Architecture

1.1 Overview

A three-tier HPC test environment on the Proxmox node hv01:

Tier Host Role
Provisioning head01 (VM 300) Warewulf, DHCP/TFTP/iPXE, Slurm controller, NAT gateway
Identity + storage idm1 (VM 305) FreeIPA (LDAP/Kerberos/CA/DNS), NFS home server
Compute wwnode1-4 (VM 301-304) Diskless, PXE-booted stateless nodes

1.2 Network topology

                              Internet
                                  │
                         vmbr0 (10.0.0.0/24)  ← management network
                    ┌─────────────┴─────────────┐
                    │                           │
        ┌───────────┴──────────┐    ┌───────────┴───────────┐
        │  head01   (VM 300)   │    │   idm1     (VM 305)   │
        │  Rocky 9.8           │    │   Rocky 9.8           │
        │  eth0 10.0.0.30      │    │   eth0 10.0.0.31      │ ← SSH / admin
        │                      │    │                       │
        │  warewulfd  dhcpd    │    │   FreeIPA (hpc.local) │
        │  tftp/iPXE  slurmctld│    │   389-ds · KDC · DNS  │
        │  munge      NAT gw   │    │   Dogtag CA · httpd   │
        │  NFS: /opt (ro)      │    │   NFS: /export/home   │
        └───────────┬──────────┘    └───────────┬───────────┘
        eth1 10.10.10.1                eth1 10.10.10.5
                    │                           │
                    └─────────────┬─────────────┘
                                  │
                       vmbr6 ← pure L2, NO IP on Proxmox
                                  │
        ┌───────────┬─────────────┼─────────────┬───────────┐
     wwnode1     wwnode2       wwnode3       wwnode4
    10.10.10.11 10.10.10.12   10.10.10.13   10.10.10.14

1.3 Why vmbr6 has no IP on the Proxmox side

Existing bridges such as vmbr3 and vmbr5 use the Proxmox host as gateway and NAT router. vmbr6 deliberately breaks that pattern.

Warewulf's provisioning model requires owning the entire subnet — DHCP (with PXE options), TFTP, the iPXE chain, and node address management. If the Proxmox host also ran DHCP/DNS on that subnet, two authorities would collide.

So vmbr6 is a pure L2 switch: bridge-ports none, no IP, no NAT rules. The subnet's gateway, DHCP server and TFTP server is head01. Nodes reach the outside world through firewalld masquerade on head01.

Side benefit: cluster traffic never leaks onto vmbr0. The other 15 VMs on hv01 (webserver, crm6, kolla, …) never see the PXE broadcasts.

1.4 Why idm1 is dual-homed

This is the most consequential design decision in the build.

The server's primary identity lives on vmbr0: 10.0.0.31, used for SSH and administration. But in DNS:

idm1.hpc.local.       A  10.10.10.5    ← service address (cluster fabric)
idm1-mgmt.hpc.local.  A  10.0.0.31     ← management address

Why the obvious alternative was rejected: the nodes are isolated on vmbr6. Their only route to 10.0.0.31 is head01's masquerade — at which point all four nodes appear to the server as 10.0.0.30. Consequences:

  • Kerberos: address mismatch between KDC and client, reverse (PTR) lookups resolving to the wrong name, service-ticket validation failures.
  • NFS: export rules are written per client IP. With four nodes collapsed into one address, per-node authorization becomes impossible.
  • Auditing: logs cannot distinguish which node did what.

A third option, not taken: disable NAT and route 10.10.10.0/24 instead. Packets would then flow node → head01 → idm1(eth0) while replies return idm1(eth1) → node directly — asymmetric routing, which is fragile under rp_filter and connection tracking.

Dual-homing means every client — including head01, which also holds 10.10.10.1 — reaches IPA directly, without NAT, over a symmetric path.

1.5 The stateless node model

Nodes have no disk. In Proxmox they carry boot: order=net0 and no scsi0 at all. The boot chain:

1. iPXE ROM  →  DHCP (temporary lease from 10.10.10.100-199)
2. DHCP reply: next-server 10.10.10.1, filename ipxe-snponly-x86_64.efi
3. iPXE  →  http://10.10.10.1:9873/ipxe/<mac>  (warewulfd)
4. Kernel downloaded (vmlinuz)
5. Compressed rootfs downloaded (rocky-9.img.gz, ~1.2 GB)
6. Unpacked into RAM as initramfs (~1.8 GB)
7. System overlay applied (network, hostname, SSSD, keytab, fstab…)
8. /sbin/init  →  systemd
9. wwclient pulls the runtime overlay every 60 s

The static IP does not come from DHCP — in step 6 the system overlay writes a NetworkManager connection file and the node moves from 10.10.10.100 to 10.10.10.1x.

Two practical consequences:

  1. High RAM usage. The rootfs lives in memory, not on disk. 8 GB is required (see 08-troubleshooting).
  2. Nothing persists. Every reboot resets the node. Anything that must survive — the Kerberos keytab, SSSD configuration — has to arrive via an overlay. This constraint shaped the entire identity design (see 04-identity).

1.6 Data flows

Flow Path Protocol
Node boot node → head01 DHCP, TFTP, HTTP :9873
User/group lookup node → idm1 LDAP :389 (SSSD, TLS)
Authentication node → idm1 Kerberos :88, :464
Name resolution node → idm1 DNS :53
Home directories node → idm1 NFSv4 :2049
OpenHPC modules node → head01 NFSv4 :2049 (ro)
Job dispatch head01 → node Slurm :6817/:6818
MPI (rank↔rank) node ↔ node TCP, direct L2
Node → internet node → head01 → vmbr0 NAT (masquerade)