Files
werewulf/README.md
T
ilkerandClaude Opus 5.5 9bca642c1b
ci / derle (push) Successful in 25s
docs: an install sequence that works on a fresh host
The quick install said `sudo hpcod setup`, which is "command not found"
until setup has put itself in /usr/local/bin, and never said that a
release binary arrives without its execute bit and under its release
name. All four places now give the same sequence - download, check
SHA256SUMS, `install -m 0755 ... hpcod`, `sudo ./hpcod setup` - run as
written on the trial host from the v0.2.1 release files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:20:09 +01:00

170 lines
8.9 KiB
Markdown

# Warewulf HPC Test Cluster
A working Warewulf 4 HPC lab on Proxmox (`hv01`) with its own isolated subnet,
central identity (FreeIPA) and shared NFS home directories.
**Status: built and verified.** Four diskless nodes PXE-boot from the head
node, enrol into the IPA domain, mount NFS homes, register `idle` in Slurm, and
run an 8-rank MPI job across all four as a directory user.
```
Internet
│
vmbr0 (10.0.0.0/24) ← management network
┌─────────────┴─────────────┐
┌───────────┴──────────┐ ┌───────────┴───────────┐
│ head01 (VM 300) │ │ idm1 (VM 305) │
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
│ Warewulf · DHCP │ │ FreeIPA (hpc.local) │
│ TFTP/iPXE · Slurm │ │ LDAP · Kerberos │
│ MUNGE · NAT gateway │ │ CA · DNS │
│ NFS: /opt (ro) │ │ NFS: /export/home │
└───────────┬──────────┘ └───────────┬───────────┘
eth1 10.10.10.1 eth1 10.10.10.5
└─────────────┬─────────────┘
│
vmbr6 ← pure L2, NO IP on Proxmox
│
┌───────────┬─────────────┼─────────────┬───────────┐
wwnode1 wwnode2 wwnode3 wwnode4 (VM 301-304)
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
/home → idm1:/export/home /opt → head01:/opt (ro)
```
## Documentation
A **25-page PDF edition** of the whole handbook: [docs/warewulf-cluster-handbook.pdf](docs/warewulf-cluster-handbook.pdf).
Regenerate it with `./scripts/build-pdf.sh` after editing `docs/handbook-print.html`.
| Document | Contents |
|---|---|
| [01 — Architecture](docs/01-architecture.md) | Topology, why `vmbr6` has no IP, why `idm1` is dual-homed, the stateless boot chain |
| [02 — Installation](docs/02-installation.md) | Installing hpcod from scratch: one binary, `sudo ./hpcod setup`, verify and a test cluster |
| [03 — Warewulf](docs/03-warewulf.md) | Images, overlays, templates, node management |
| [04 — Identity](docs/04-identity.md) | FreeIPA vs AD, enrolling stateless nodes, users and groups |
| [05 — Storage](docs/05-storage.md) | NFS layout, id mapping, security posture, path to Kerberised NFS |
| [06 — Slurm & MPI](docs/06-slurm-mpi.md) | Scheduler config, running jobs, what is not configured |
| [07 — Operations](docs/07-operations.md) | Runbook: access, power, health checks, common tasks, logs |
| [08 — Troubleshooting](docs/08-troubleshooting.md) | The 12 pitfalls hit during the build, plus a diagnostic guide |
| [09 — Reference](docs/09-reference.md) | Inventory, addressing, ports, versions, file locations |
| [10 — Building the lab](docs/10-lab-installation.md) | The reference lab built by hand: script order, what each does, the two manual steps, timings |
| [20 — HPC-on-Demand](docs/20-hpc-on-demand-architecture.md) | Design: multi-site control plane that provisions clusters like this one on demand |
| [21 — Portal layer](docs/21-portal-layer.md) | What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to |
| [22 — Spack & Chapar](docs/22-spack-chapar-integration.md) | Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first |
| [23 — Application lifecycle](docs/23-application-lifecycle.md) | What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them |
| [hpcod/](hpcod/) | Implementation: multi-provider discovery, placement, and the portal at hpc.example.org |
| [hpcod/DEPLOY.md](hpcod/DEPLOY.md) | Installing hpcod against your own Proxmox: `sudo ./hpcod setup`, from the binary alone |
| [CHANGELOG](CHANGELOG.md) | What is in each release, and what is knowingly not |
## Quick start
```bash
ssh head01 # head node
ssh idm1 # identity + home server
sinfo # cluster state
sudo wwctl node status # provisioning state
./scripts/A99-verify.sh # full end-to-end health check
```
Compute nodes live on the isolated subnet — reach them through the head node:
```bash
ssh head01
sudo ssh root@wwnode1
```
HPC-on-Demand portal — **<https://hpc.example.org>** (see [hpcod/](hpcod/)).
Users ask for a cluster there, and it is built, published and handed over.
To install hpcod somewhere else, download the release binary and run
`sudo ./hpcod setup` ([docs/02-installation.md](docs/02-installation.md)).
Power management:
```bash
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes
```
## Software
| Component | Version |
|---|---|
| Proxmox VE | 8.4.19 |
| Host / node OS | Rocky Linux 9.8 |
| Warewulf | 4.7.0 |
| OpenHPC | 3 (EL_9) |
| Slurm | 25.11.4 |
| OpenMPI | 5.0.7 (GCC 14.2) |
| FreeIPA | 4.13.1 |
| SSSD | 2.9.8 |
## Three decisions worth knowing
**`vmbr6` has no IP on Proxmox.** Warewulf needs to own the entire subnet —
DHCP with PXE options, TFTP, address management. A second authority on the same
segment would collide. The bridge is a pure L2 switch and `head01` is the
gateway. See [01-architecture §1.3](docs/01-architecture.md#13-why-vmbr6-has-no-ip-on-the-proxmox-side).
**`idm1` is dual-homed, and `idm1.hpc.local` resolves to the cluster side.**
Behind `head01`'s NAT all four nodes would appear as 10.0.0.30, which breaks
Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every
client a direct, symmetric path. See [01-architecture §1.4](docs/01-architecture.md#14-why-idm1-is-dual-homed).
**Stateless nodes cannot run `ipa-client-install`.** The rootfs is rebuilt in
RAM every boot, so enrolment happens once and the result is distributed through
Warewulf overlays — shared config as templates rendered per node, and a
**separate keytab per node** so `wwnode1` cannot impersonate `wwnode2`.
See [04-identity §4.3](docs/04-identity.md#43-the-hard-part--enrolling-stateless-nodes).
## Verified output
```
================= IPA SERVER (idm1) =================
IPA services running: 9
/export/home 10.10.10.0/24(rw,no_root_squash,sec=sys,...)
IPA users: 3
================= HEAD NODE (head01) =================
hostname : head01.hpc.local
DNS : 10.10.10.5
/home : idm1.hpc.local:/export/home
hpcuser : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
services : warewulfd=active dhcpd=active tftp.socket=active
slurmctld=active munge=active sssd=active
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
normal* up infinite 4 idle wwnode[1-4]
================= COMPUTE NODES =================
wwnode1: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode2: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode3: sssd=active slurmd=active munge=active home=nfs krb5=OK
wwnode4: sssd=active slurmd=active munge=active home=nfs krb5=OK
================= MPI JOB (IPA user, NFS home) =================
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
```
`krb5=OK` means the node actually obtained a ticket from the KDC using its own
host keytab (`kinit -k host/wwnodeN.hpc.local`) — not merely that the file is
present.
Additionally verified by hand: an IPA user SSH'd into a stateless node with a
password, received a Kerberos TGT and landed in their NFS home; and a brand-new
IPA user's home was auto-created on the NFS share at first login.
## Known gaps
- **NFS is `sec=sys`**, not Kerberised. Host keytabs and the
`nfs/idm1.hpc.local` service principal already exist, so `sec=krb5p` is a
short step — but it changes the home-provisioning flow.
- **Slurm accounting is recorded but not enforced** — `sacct` and `sreport` work; `AccountingStorageEnforce` is unset, so no association is required to submit.
- **SELinux permissive** on head and nodes.
- **FreeIPA is not Active Directory.** It is the RHEL-native equivalent for
Linux clients; Windows joins or the AD schema would need a Samba AD DC or an
AD trust. See [04-identity §4.1](docs/04-identity.md#41-freeipa-is-not-active-directory).
Full list with rationale in [09-reference §9.9](docs/09-reference.md#99-deliberate-choices-and-known-gaps).