Files
werewulf/docs/01-architecture.md
ilkerandClaude Opus 5.5 5b6e1ed3dd docs: section 02 installs hpcod from scratch; the lab build moves to 10
02 now walks an empty host through `hpcod setup`: what it needs, what the
TUI asks and does, the simulated Proxmox, reaching the portal, verify and
selftest, with timings measured on one run. The hand-built lab's script
order moves unchanged to 10-lab-installation.md. README, 01, 09 and the
handbook (contents, chapter 2 with two screenshots, new chapter 10) follow;
the PDF is regenerated, now 25 pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:08:32 +01:00

137 lines
6.4 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 1. Architecture
Sections 01 and 03–09 document the reference lab: one cluster built by
hand with the scripts in `scripts/` (section 10), which is where every
design decision in this repository was first proven. The HPC-on-Demand
control plane builds clusters like it on demand; section 02 installs it
and section 20 describes it.
## 1.1 Overview
A three-tier HPC test environment on the Proxmox node `hv01`:
| Tier | Host | Role |
|---|---|---|
| Provisioning | `head01` (VM 300) | Warewulf, DHCP/TFTP/iPXE, Slurm controller, NAT gateway |
| Identity + storage | `idm1` (VM 305) | FreeIPA (LDAP/Kerberos/CA/DNS), NFS home server |
| Compute | `wwnode1-4` (VM 301-304) | Diskless, PXE-booted stateless nodes |
## 1.2 Network topology
```
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
│ │
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ Rocky 9.8 │ │ Rocky 9.8 │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ │ │ │
│ warewulfd dhcpd │ │ FreeIPA (hpc.local) │
│ tftp/iPXE slurmctld│ │ 389-ds · KDC · DNS │
│ munge NAT gw │ │ Dogtag CA · httpd │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
│ │
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
```
## 1.3 Why `vmbr6` has no IP on the Proxmox side
Existing bridges such as `vmbr3` and `vmbr5` use the Proxmox host as gateway
and NAT router. `vmbr6` deliberately breaks that pattern.
Warewulf's provisioning model requires **owning the entire subnet** — DHCP
(with PXE options), TFTP, the iPXE chain, and node address management. If the
Proxmox host also ran DHCP/DNS on that subnet, two authorities would collide.
So `vmbr6` is a pure L2 switch: `bridge-ports none`, no IP, no NAT rules. The
subnet's gateway, DHCP server and TFTP server is `head01`. Nodes reach the
outside world through `firewalld` masquerade on `head01`.
Side benefit: cluster traffic never leaks onto `vmbr0`. The other 15 VMs on
`hv01` (webserver, crm6, kolla, …) never see the PXE broadcasts.
## 1.4 Why `idm1` is dual-homed
This is the most consequential design decision in the build.
The server's **primary identity lives on `vmbr0`**: 10.0.0.31, used for SSH and
administration. But in DNS:
```
idm1.hpc.local. A 10.10.10.5 ← service address (cluster fabric)
idm1-mgmt.hpc.local. A 10.0.0.31 ← management address
```
**Why the obvious alternative was rejected:** the nodes are isolated on
`vmbr6`. Their only route to `10.0.0.31` is `head01`'s masquerade — at which
point all four nodes appear to the server **as 10.0.0.30**. Consequences:
- **Kerberos:** address mismatch between KDC and client, reverse (PTR) lookups
resolving to the wrong name, service-ticket validation failures.
- **NFS:** export rules are written per client IP. With four nodes collapsed
into one address, per-node authorization becomes impossible.
- **Auditing:** logs cannot distinguish which node did what.
**A third option, not taken:** disable NAT and route `10.10.10.0/24` instead.
Packets would then flow `node → head01 → idm1(eth0)` while replies return
`idm1(eth1) → node` directly — **asymmetric routing**, which is fragile under
`rp_filter` and connection tracking.
Dual-homing means every client — including `head01`, which also holds
10.10.10.1 — reaches IPA directly, without NAT, over a symmetric path.
## 1.5 The stateless node model
Nodes have **no disk**. In Proxmox they carry `boot: order=net0` and no `scsi0`
at all. The boot chain:
```
1. iPXE ROM → DHCP (temporary lease from 10.10.10.100-199)
2. DHCP reply: next-server 10.10.10.1, filename ipxe-snponly-x86_64.efi
3. iPXE → http://10.10.10.1:9873/ipxe/<mac> (warewulfd)
4. Kernel downloaded (vmlinuz)
5. Compressed rootfs downloaded (rocky-9.img.gz, ~1.2 GB)
6. Unpacked into RAM as initramfs (~1.8 GB)
7. System overlay applied (network, hostname, SSSD, keytab, fstab…)
8. /sbin/init → systemd
9. wwclient pulls the runtime overlay every 60 s
```
The static IP does **not** come from DHCP — in step 6 the system overlay writes
a `NetworkManager` connection file and the node moves from 10.10.10.100 to
10.10.10.1x.
Two practical consequences:
1. **High RAM usage.** The rootfs lives in memory, not on disk. 8 GB is
required (see [08-troubleshooting](08-troubleshooting.md#1-4-gb-of-node-ram-is-not-enough)).
2. **Nothing persists.** Every reboot resets the node. Anything that must
survive — the Kerberos keytab, SSSD configuration — has to arrive via an
**overlay**. This constraint shaped the entire identity design
(see [04-identity](04-identity.md)).
## 1.6 Data flows
| Flow | Path | Protocol |
|---|---|---|
| Node boot | node → head01 | DHCP, TFTP, HTTP :9873 |
| User/group lookup | node → idm1 | LDAP :389 (SSSD, TLS) |
| Authentication | node → idm1 | Kerberos :88, :464 |
| Name resolution | node → idm1 | DNS :53 |
| Home directories | node → idm1 | NFSv4 :2049 |
| OpenHPC modules | node → head01 | NFSv4 :2049 (ro) |
| Job dispatch | head01 → node | Slurm :6817/:6818 |
| MPI (rank↔rank) | node ↔ node | TCP, direct L2 |
| Node → internet | node → head01 → vmbr0 | NAT (masquerade) |