02 now walks an empty host through `hpcod setup`: what it needs, what the TUI asks and does, the simulated Proxmox, reaching the portal, verify and selftest, with timings measured on one run. The hand-built lab's script order moves unchanged to 10-lab-installation.md. README, 01, 09 and the handbook (contents, chapter 2 with two screenshots, new chapter 10) follow; the PDF is regenerated, now 25 pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
137 lines
6.4 KiB
Markdown
137 lines
6.4 KiB
Markdown
# 1. Architecture
|
||
|
||
Sections 01 and 03–09 document the reference lab: one cluster built by
|
||
hand with the scripts in `scripts/` (section 10), which is where every
|
||
design decision in this repository was first proven. The HPC-on-Demand
|
||
control plane builds clusters like it on demand; section 02 installs it
|
||
and section 20 describes it.
|
||
|
||
## 1.1 Overview
|
||
|
||
A three-tier HPC test environment on the Proxmox node `hv01`:
|
||
|
||
| Tier | Host | Role |
|
||
|---|---|---|
|
||
| Provisioning | `head01` (VM 300) | Warewulf, DHCP/TFTP/iPXE, Slurm controller, NAT gateway |
|
||
| Identity + storage | `idm1` (VM 305) | FreeIPA (LDAP/Kerberos/CA/DNS), NFS home server |
|
||
| Compute | `wwnode1-4` (VM 301-304) | Diskless, PXE-booted stateless nodes |
|
||
|
||
## 1.2 Network topology
|
||
|
||
```
|
||
Internet
|
||
│
|
||
vmbr0 (10.0.0.0/24) ← management network
|
||
┌─────────────┴─────────────┐
|
||
│ │
|
||
┌───────────┴──────────┐ ┌───────────┴───────────┐
|
||
│ head01 (VM 300) │ │ idm1 (VM 305) │
|
||
│ Rocky 9.8 │ │ Rocky 9.8 │
|
||
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
|
||
│ │ │ │
|
||
│ warewulfd dhcpd │ │ FreeIPA (hpc.local) │
|
||
│ tftp/iPXE slurmctld│ │ 389-ds · KDC · DNS │
|
||
│ munge NAT gw │ │ Dogtag CA · httpd │
|
||
│ NFS: /opt (ro) │ │ NFS: /export/home │
|
||
└───────────┬──────────┘ └───────────┬───────────┘
|
||
eth1 10.10.10.1 eth1 10.10.10.5
|
||
│ │
|
||
└─────────────┬─────────────┘
|
||
│
|
||
vmbr6 ← pure L2, NO IP on Proxmox
|
||
│
|
||
┌───────────┬─────────────┼─────────────┬───────────┐
|
||
wwnode1 wwnode2 wwnode3 wwnode4
|
||
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
|
||
```
|
||
|
||
## 1.3 Why `vmbr6` has no IP on the Proxmox side
|
||
|
||
Existing bridges such as `vmbr3` and `vmbr5` use the Proxmox host as gateway
|
||
and NAT router. `vmbr6` deliberately breaks that pattern.
|
||
|
||
Warewulf's provisioning model requires **owning the entire subnet** — DHCP
|
||
(with PXE options), TFTP, the iPXE chain, and node address management. If the
|
||
Proxmox host also ran DHCP/DNS on that subnet, two authorities would collide.
|
||
|
||
So `vmbr6` is a pure L2 switch: `bridge-ports none`, no IP, no NAT rules. The
|
||
subnet's gateway, DHCP server and TFTP server is `head01`. Nodes reach the
|
||
outside world through `firewalld` masquerade on `head01`.
|
||
|
||
Side benefit: cluster traffic never leaks onto `vmbr0`. The other 15 VMs on
|
||
`hv01` (webserver, crm6, kolla, …) never see the PXE broadcasts.
|
||
|
||
## 1.4 Why `idm1` is dual-homed
|
||
|
||
This is the most consequential design decision in the build.
|
||
|
||
The server's **primary identity lives on `vmbr0`**: 10.0.0.31, used for SSH and
|
||
administration. But in DNS:
|
||
|
||
```
|
||
idm1.hpc.local. A 10.10.10.5 ← service address (cluster fabric)
|
||
idm1-mgmt.hpc.local. A 10.0.0.31 ← management address
|
||
```
|
||
|
||
**Why the obvious alternative was rejected:** the nodes are isolated on
|
||
`vmbr6`. Their only route to `10.0.0.31` is `head01`'s masquerade — at which
|
||
point all four nodes appear to the server **as 10.0.0.30**. Consequences:
|
||
|
||
- **Kerberos:** address mismatch between KDC and client, reverse (PTR) lookups
|
||
resolving to the wrong name, service-ticket validation failures.
|
||
- **NFS:** export rules are written per client IP. With four nodes collapsed
|
||
into one address, per-node authorization becomes impossible.
|
||
- **Auditing:** logs cannot distinguish which node did what.
|
||
|
||
**A third option, not taken:** disable NAT and route `10.10.10.0/24` instead.
|
||
Packets would then flow `node → head01 → idm1(eth0)` while replies return
|
||
`idm1(eth1) → node` directly — **asymmetric routing**, which is fragile under
|
||
`rp_filter` and connection tracking.
|
||
|
||
Dual-homing means every client — including `head01`, which also holds
|
||
10.10.10.1 — reaches IPA directly, without NAT, over a symmetric path.
|
||
|
||
## 1.5 The stateless node model
|
||
|
||
Nodes have **no disk**. In Proxmox they carry `boot: order=net0` and no `scsi0`
|
||
at all. The boot chain:
|
||
|
||
```
|
||
1. iPXE ROM → DHCP (temporary lease from 10.10.10.100-199)
|
||
2. DHCP reply: next-server 10.10.10.1, filename ipxe-snponly-x86_64.efi
|
||
3. iPXE → http://10.10.10.1:9873/ipxe/<mac> (warewulfd)
|
||
4. Kernel downloaded (vmlinuz)
|
||
5. Compressed rootfs downloaded (rocky-9.img.gz, ~1.2 GB)
|
||
6. Unpacked into RAM as initramfs (~1.8 GB)
|
||
7. System overlay applied (network, hostname, SSSD, keytab, fstab…)
|
||
8. /sbin/init → systemd
|
||
9. wwclient pulls the runtime overlay every 60 s
|
||
```
|
||
|
||
The static IP does **not** come from DHCP — in step 6 the system overlay writes
|
||
a `NetworkManager` connection file and the node moves from 10.10.10.100 to
|
||
10.10.10.1x.
|
||
|
||
Two practical consequences:
|
||
|
||
1. **High RAM usage.** The rootfs lives in memory, not on disk. 8 GB is
|
||
required (see [08-troubleshooting](08-troubleshooting.md#1-4-gb-of-node-ram-is-not-enough)).
|
||
2. **Nothing persists.** Every reboot resets the node. Anything that must
|
||
survive — the Kerberos keytab, SSSD configuration — has to arrive via an
|
||
**overlay**. This constraint shaped the entire identity design
|
||
(see [04-identity](04-identity.md)).
|
||
|
||
## 1.6 Data flows
|
||
|
||
| Flow | Path | Protocol |
|
||
|---|---|---|
|
||
| Node boot | node → head01 | DHCP, TFTP, HTTP :9873 |
|
||
| User/group lookup | node → idm1 | LDAP :389 (SSSD, TLS) |
|
||
| Authentication | node → idm1 | Kerberos :88, :464 |
|
||
| Name resolution | node → idm1 | DNS :53 |
|
||
| Home directories | node → idm1 | NFSv4 :2049 |
|
||
| OpenHPC modules | node → head01 | NFSv4 :2049 (ro) |
|
||
| Job dispatch | head01 → node | Slurm :6817/:6818 |
|
||
| MPI (rank↔rank) | node ↔ node | TCP, direct L2 |
|
||
| Node → internet | node → head01 → vmbr0 | NAT (masquerade) |
|