Files
werewulf/docs/01-architecture.md
T
ilkerandClaude Opus 5.5 5b6e1ed3dd docs: section 02 installs hpcod from scratch; the lab build moves to 10
02 now walks an empty host through `hpcod setup`: what it needs, what the
TUI asks and does, the simulated Proxmox, reaching the portal, verify and
selftest, with timings measured on one run. The hand-built lab's script
order moves unchanged to 10-lab-installation.md. README, 01, 09 and the
handbook (contents, chapter 2 with two screenshots, new chapter 10) follow;
the PDF is regenerated, now 25 pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:08:32 +01:00

6.4 KiB
Raw Permalink Blame History

1. Architecture

Sections 01 and 03–09 document the reference lab: one cluster built by hand with the scripts in scripts/ (section 10), which is where every design decision in this repository was first proven. The HPC-on-Demand control plane builds clusters like it on demand; section 02 installs it and section 20 describes it.

1.1 Overview

A three-tier HPC test environment on the Proxmox node hv01:

Tier Host Role
Provisioning head01 (VM 300) Warewulf, DHCP/TFTP/iPXE, Slurm controller, NAT gateway
Identity + storage idm1 (VM 305) FreeIPA (LDAP/Kerberos/CA/DNS), NFS home server
Compute wwnode1-4 (VM 301-304) Diskless, PXE-booted stateless nodes

1.2 Network topology

                              Internet
                                  │
                         vmbr0 (10.0.0.0/24)  ← management network
                    ┌─────────────┴─────────────┐
                    │                           │
        ┌───────────┴──────────┐    ┌───────────┴───────────┐
        │  head01   (VM 300)   │    │   idm1     (VM 305)   │
        │  Rocky 9.8           │    │   Rocky 9.8           │
        │  eth0 10.0.0.30      │    │   eth0 10.0.0.31      │ ← SSH / admin
        │                      │    │                       │
        │  warewulfd  dhcpd    │    │   FreeIPA (hpc.local) │
        │  tftp/iPXE  slurmctld│    │   389-ds · KDC · DNS  │
        │  munge      NAT gw   │    │   Dogtag CA · httpd   │
        │  NFS: /opt (ro)      │    │   NFS: /export/home   │
        └───────────┬──────────┘    └───────────┬───────────┘
        eth1 10.10.10.1                eth1 10.10.10.5
                    │                           │
                    └─────────────┬─────────────┘
                                  │
                       vmbr6 ← pure L2, NO IP on Proxmox
                                  │
        ┌───────────┬─────────────┼─────────────┬───────────┐
     wwnode1     wwnode2       wwnode3       wwnode4
    10.10.10.11 10.10.10.12   10.10.10.13   10.10.10.14

1.3 Why vmbr6 has no IP on the Proxmox side

Existing bridges such as vmbr3 and vmbr5 use the Proxmox host as gateway and NAT router. vmbr6 deliberately breaks that pattern.

Warewulf's provisioning model requires owning the entire subnet — DHCP (with PXE options), TFTP, the iPXE chain, and node address management. If the Proxmox host also ran DHCP/DNS on that subnet, two authorities would collide.

So vmbr6 is a pure L2 switch: bridge-ports none, no IP, no NAT rules. The subnet's gateway, DHCP server and TFTP server is head01. Nodes reach the outside world through firewalld masquerade on head01.

Side benefit: cluster traffic never leaks onto vmbr0. The other 15 VMs on hv01 (webserver, crm6, kolla, …) never see the PXE broadcasts.

1.4 Why idm1 is dual-homed

This is the most consequential design decision in the build.

The server's primary identity lives on vmbr0: 10.0.0.31, used for SSH and administration. But in DNS:

idm1.hpc.local.       A  10.10.10.5    ← service address (cluster fabric)
idm1-mgmt.hpc.local.  A  10.0.0.31     ← management address

Why the obvious alternative was rejected: the nodes are isolated on vmbr6. Their only route to 10.0.0.31 is head01's masquerade — at which point all four nodes appear to the server as 10.0.0.30. Consequences:

  • Kerberos: address mismatch between KDC and client, reverse (PTR) lookups resolving to the wrong name, service-ticket validation failures.
  • NFS: export rules are written per client IP. With four nodes collapsed into one address, per-node authorization becomes impossible.
  • Auditing: logs cannot distinguish which node did what.

A third option, not taken: disable NAT and route 10.10.10.0/24 instead. Packets would then flow node → head01 → idm1(eth0) while replies return idm1(eth1) → node directly — asymmetric routing, which is fragile under rp_filter and connection tracking.

Dual-homing means every client — including head01, which also holds 10.10.10.1 — reaches IPA directly, without NAT, over a symmetric path.

1.5 The stateless node model

Nodes have no disk. In Proxmox they carry boot: order=net0 and no scsi0 at all. The boot chain:

1. iPXE ROM  →  DHCP (temporary lease from 10.10.10.100-199)
2. DHCP reply: next-server 10.10.10.1, filename ipxe-snponly-x86_64.efi
3. iPXE  →  http://10.10.10.1:9873/ipxe/<mac>  (warewulfd)
4. Kernel downloaded (vmlinuz)
5. Compressed rootfs downloaded (rocky-9.img.gz, ~1.2 GB)
6. Unpacked into RAM as initramfs (~1.8 GB)
7. System overlay applied (network, hostname, SSSD, keytab, fstab…)
8. /sbin/init  →  systemd
9. wwclient pulls the runtime overlay every 60 s

The static IP does not come from DHCP — in step 6 the system overlay writes a NetworkManager connection file and the node moves from 10.10.10.100 to 10.10.10.1x.

Two practical consequences:

  1. High RAM usage. The rootfs lives in memory, not on disk. 8 GB is required (see 08-troubleshooting).
  2. Nothing persists. Every reboot resets the node. Anything that must survive — the Kerberos keytab, SSSD configuration — has to arrive via an overlay. This constraint shaped the entire identity design (see 04-identity).

1.6 Data flows

Flow Path Protocol
Node boot node → head01 DHCP, TFTP, HTTP :9873
User/group lookup node → idm1 LDAP :389 (SSSD, TLS)
Authentication node → idm1 Kerberos :88, :464
Name resolution node → idm1 DNS :53
Home directories node → idm1 NFSv4 :2049
OpenHPC modules node → head01 NFSv4 :2049 (ro)
Job dispatch head01 → node Slurm :6817/:6818
MPI (rank↔rank) node ↔ node TCP, direct L2
Node → internet node → head01 → vmbr0 NAT (masquerade)