02 now walks an empty host through `hpcod setup`: what it needs, what the TUI asks and does, the simulated Proxmox, reaching the portal, verify and selftest, with timings measured on one run. The hand-built lab's script order moves unchanged to 10-lab-installation.md. README, 01, 09 and the handbook (contents, chapter 2 with two screenshots, new chapter 10) follow; the PDF is regenerated, now 25 pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6.4 KiB
1. Architecture
Sections 01 and 03–09 document the reference lab: one cluster built by
hand with the scripts in scripts/ (section 10), which is where every
design decision in this repository was first proven. The HPC-on-Demand
control plane builds clusters like it on demand; section 02 installs it
and section 20 describes it.
1.1 Overview
A three-tier HPC test environment on the Proxmox node hv01:
| Tier | Host | Role |
|---|---|---|
| Provisioning | head01 (VM 300) |
Warewulf, DHCP/TFTP/iPXE, Slurm controller, NAT gateway |
| Identity + storage | idm1 (VM 305) |
FreeIPA (LDAP/Kerberos/CA/DNS), NFS home server |
| Compute | wwnode1-4 (VM 301-304) |
Diskless, PXE-booted stateless nodes |
1.2 Network topology
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
│ │
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ Rocky 9.8 │ │ Rocky 9.8 │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ │ │ │
│ warewulfd dhcpd │ │ FreeIPA (hpc.local) │
│ tftp/iPXE slurmctld│ │ 389-ds · KDC · DNS │
│ munge NAT gw │ │ Dogtag CA · httpd │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
│ │
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
1.3 Why vmbr6 has no IP on the Proxmox side
Existing bridges such as vmbr3 and vmbr5 use the Proxmox host as gateway
and NAT router. vmbr6 deliberately breaks that pattern.
Warewulf's provisioning model requires owning the entire subnet — DHCP (with PXE options), TFTP, the iPXE chain, and node address management. If the Proxmox host also ran DHCP/DNS on that subnet, two authorities would collide.
So vmbr6 is a pure L2 switch: bridge-ports none, no IP, no NAT rules. The
subnet's gateway, DHCP server and TFTP server is head01. Nodes reach the
outside world through firewalld masquerade on head01.
Side benefit: cluster traffic never leaks onto vmbr0. The other 15 VMs on
hv01 (webserver, crm6, kolla, …) never see the PXE broadcasts.
1.4 Why idm1 is dual-homed
This is the most consequential design decision in the build.
The server's primary identity lives on vmbr0: 10.0.0.31, used for SSH and
administration. But in DNS:
idm1.hpc.local. A 10.10.10.5 ← service address (cluster fabric)
idm1-mgmt.hpc.local. A 10.0.0.31 ← management address
Why the obvious alternative was rejected: the nodes are isolated on
vmbr6. Their only route to 10.0.0.31 is head01's masquerade — at which
point all four nodes appear to the server as 10.0.0.30. Consequences:
- Kerberos: address mismatch between KDC and client, reverse (PTR) lookups resolving to the wrong name, service-ticket validation failures.
- NFS: export rules are written per client IP. With four nodes collapsed into one address, per-node authorization becomes impossible.
- Auditing: logs cannot distinguish which node did what.
A third option, not taken: disable NAT and route 10.10.10.0/24 instead.
Packets would then flow node → head01 → idm1(eth0) while replies return
idm1(eth1) → node directly — asymmetric routing, which is fragile under
rp_filter and connection tracking.
Dual-homing means every client — including head01, which also holds
10.10.10.1 — reaches IPA directly, without NAT, over a symmetric path.
1.5 The stateless node model
Nodes have no disk. In Proxmox they carry boot: order=net0 and no scsi0
at all. The boot chain:
1. iPXE ROM → DHCP (temporary lease from 10.10.10.100-199)
2. DHCP reply: next-server 10.10.10.1, filename ipxe-snponly-x86_64.efi
3. iPXE → http://10.10.10.1:9873/ipxe/<mac> (warewulfd)
4. Kernel downloaded (vmlinuz)
5. Compressed rootfs downloaded (rocky-9.img.gz, ~1.2 GB)
6. Unpacked into RAM as initramfs (~1.8 GB)
7. System overlay applied (network, hostname, SSSD, keytab, fstab…)
8. /sbin/init → systemd
9. wwclient pulls the runtime overlay every 60 s
The static IP does not come from DHCP — in step 6 the system overlay writes
a NetworkManager connection file and the node moves from 10.10.10.100 to
10.10.10.1x.
Two practical consequences:
- High RAM usage. The rootfs lives in memory, not on disk. 8 GB is required (see 08-troubleshooting).
- Nothing persists. Every reboot resets the node. Anything that must survive — the Kerberos keytab, SSSD configuration — has to arrive via an overlay. This constraint shaped the entire identity design (see 04-identity).
1.6 Data flows
| Flow | Path | Protocol |
|---|---|---|
| Node boot | node → head01 | DHCP, TFTP, HTTP :9873 |
| User/group lookup | node → idm1 | LDAP :389 (SSSD, TLS) |
| Authentication | node → idm1 | Kerberos :88, :464 |
| Name resolution | node → idm1 | DNS :53 |
| Home directories | node → idm1 | NFSv4 :2049 |
| OpenHPC modules | node → head01 | NFSv4 :2049 (ro) |
| Job dispatch | head01 → node | Slurm :6817/:6818 |
| MPI (rank↔rank) | node ↔ node | TCP, direct L2 |
| Node → internet | node → head01 → vmbr0 | NAT (masquerade) |