Files
ilkerandClaude Opus 5.5 3297de5d50
ci / derle (push) Successful in 22s
Apache License 2.0
The repository had no license, so it was public but not open source:
nobody could use, change or redistribute it. LICENSE is the text from
apache.org, unmodified. NOTICE says that the IBM Plex fonts embedded in
the print handbook are under the SIL Open Font License, not Apache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:27:25 +01:00

175 lines
9.1 KiB
Markdown

# Warewulf HPC Test Cluster
A working Warewulf 4 HPC lab on Proxmox (`hv01`) with its own isolated subnet,
central identity (FreeIPA) and shared NFS home directories.
**Status: built and verified.** Four diskless nodes PXE-boot from the head
node, enrol into the IPA domain, mount NFS homes, register `idle` in Slurm, and
run an 8-rank MPI job across all four as a directory user.
```
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ Warewulf · DHCP │ │ FreeIPA (hpc.local) │
│ TFTP/iPXE · Slurm │ │ LDAP · Kerberos │
│ MUNGE · NAT gateway │ │ CA · DNS │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4 (VM 301-304)
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
/home → idm1:/export/home /opt → head01:/opt (ro)
```
## Documentation
A **25-page PDF edition** of the whole handbook: [docs/warewulf-cluster-handbook.pdf](docs/warewulf-cluster-handbook.pdf).
Regenerate it with `./scripts/build-pdf.sh` after editing `docs/handbook-print.html`.
| Document | Contents |
|---|---|
| [01 — Architecture](docs/01-architecture.md) | Topology, why `vmbr6` has no IP, why `idm1` is dual-homed, the stateless boot chain |
| [02 — Installation](docs/02-installation.md) | Installing hpcod from scratch: one binary, `sudo ./hpcod setup`, verify and a test cluster |
| [03 — Warewulf](docs/03-warewulf.md) | Images, overlays, templates, node management |
| [04 — Identity](docs/04-identity.md) | FreeIPA vs AD, enrolling stateless nodes, users and groups |
| [05 — Storage](docs/05-storage.md) | NFS layout, id mapping, security posture, path to Kerberised NFS |
| [06 — Slurm & MPI](docs/06-slurm-mpi.md) | Scheduler config, running jobs, what is not configured |
| [07 — Operations](docs/07-operations.md) | Runbook: access, power, health checks, common tasks, logs |
| [08 — Troubleshooting](docs/08-troubleshooting.md) | The 12 pitfalls hit during the build, plus a diagnostic guide |
| [09 — Reference](docs/09-reference.md) | Inventory, addressing, ports, versions, file locations |
| [10 — Building the lab](docs/10-lab-installation.md) | The reference lab built by hand: script order, what each does, the two manual steps, timings |
| [20 — HPC-on-Demand](docs/20-hpc-on-demand-architecture.md) | Design: multi-site control plane that provisions clusters like this one on demand |
| [21 — Portal layer](docs/21-portal-layer.md) | What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to |
| [22 — Spack & Chapar](docs/22-spack-chapar-integration.md) | Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first |
| [23 — Application lifecycle](docs/23-application-lifecycle.md) | What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them |
| [hpcod/](hpcod/) | Implementation: multi-provider discovery, placement, and the portal at hpc.example.org |
| [hpcod/DEPLOY.md](hpcod/DEPLOY.md) | Installing hpcod against your own Proxmox: `sudo ./hpcod setup`, from the binary alone |
| [CHANGELOG](CHANGELOG.md) | What is in each release, and what is knowingly not |
## Quick start
```bash
ssh head01 # head node
ssh idm1 # identity + home server
sinfo # cluster state
sudo wwctl node status # provisioning state
./scripts/A99-verify.sh # full end-to-end health check
```
Compute nodes live on the isolated subnet — reach them through the head node:
```bash
ssh head01
sudo ssh root@wwnode1
```
HPC-on-Demand portal — **<https://hpc.example.org>** (see [hpcod/](hpcod/)).
Users ask for a cluster there, and it is built, published and handed over.
To install hpcod somewhere else, download the release binary and run
`sudo ./hpcod setup` ([docs/02-installation.md](docs/02-installation.md)).
Power management:
```bash
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes
```
## License
[Apache License 2.0](LICENSE). The fonts embedded in the print handbook
are under the SIL Open Font License; see [NOTICE](NOTICE).
## Software
| Component | Version |
|---|---|
| Proxmox VE | 8.4.19 |
| Host / node OS | Rocky Linux 9.8 |
| Warewulf | 4.7.0 |
| OpenHPC | 3 (EL_9) |
| Slurm | 25.11.4 |
| OpenMPI | 5.0.7 (GCC 14.2) |
| FreeIPA | 4.13.1 |
| SSSD | 2.9.8 |
## Three decisions worth knowing
**`vmbr6` has no IP on Proxmox.** Warewulf needs to own the entire subnet —
DHCP with PXE options, TFTP, address management. A second authority on the same
segment would collide. The bridge is a pure L2 switch and `head01` is the
gateway. See [01-architecture §1.3](docs/01-architecture.md#13-why-vmbr6-has-no-ip-on-the-proxmox-side).
**`idm1` is dual-homed, and `idm1.hpc.local` resolves to the cluster side.**
Behind `head01`'s NAT all four nodes would appear as 10.0.0.30, which breaks
Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every
client a direct, symmetric path. See [01-architecture §1.4](docs/01-architecture.md#14-why-idm1-is-dual-homed).
**Stateless nodes cannot run `ipa-client-install`.** The rootfs is rebuilt in
RAM every boot, so enrolment happens once and the result is distributed through
Warewulf overlays — shared config as templates rendered per node, and a
**separate keytab per node** so `wwnode1` cannot impersonate `wwnode2`.
See [04-identity §4.3](docs/04-identity.md#43-the-hard-part--enrolling-stateless-nodes).
## Verified output
```
================= IPA SERVER (idm1) =================
IPA services running: 9
/export/home 10.10.10.0/24(rw,no_root_squash,sec=sys,...)
IPA users: 3
================= HEAD NODE (head01) =================
hostname : head01.hpc.local
DNS : 10.10.10.5
/home : idm1.hpc.local:/export/home
hpcuser : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
services : warewulfd=active dhcpd=active tftp.socket=active
slurmctld=active munge=active sssd=active
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
normal* up infinite 4 idle wwnode[1-4]
================= COMPUTE NODES =================
wwnode1: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode2: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode3: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode4: sssd=active slurmd=active munge=active home=nfs krb5=OK
================= MPI JOB (IPA user, NFS home) =================
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
```
`krb5=OK` means the node actually obtained a ticket from the KDC using its own
host keytab (`kinit -k host/wwnodeN.hpc.local`) — not merely that the file is
present.
Additionally verified by hand: an IPA user SSH'd into a stateless node with a
password, received a Kerberos TGT and landed in their NFS home; and a brand-new
IPA user's home was auto-created on the NFS share at first login.
## Known gaps
- **NFS is `sec=sys`**, not Kerberised. Host keytabs and the
`nfs/idm1.hpc.local` service principal already exist, so `sec=krb5p` is a
short step — but it changes the home-provisioning flow.
- **Slurm accounting is recorded but not enforced** — `sacct` and `sreport` work; `AccountingStorageEnforce` is unset, so no association is required to submit.
- **SELinux permissive** on head and nodes.
- **FreeIPA is not Active Directory.** It is the RHEL-native equivalent for
Linux clients; Windows joins or the AD schema would need a Samba AD DC or an
AD trust. See [04-identity §4.1](docs/04-identity.md#41-freeipa-is-not-active-directory).
Full list with rationale in [09-reference §9.9](docs/09-reference.md#99-deliberate-choices-and-known-gaps).