Files
werewulf/README.md
T
ilkerandClaude Opus 5.5 9bca642c1b
ci / derle (push) Successful in 25s
docs: an install sequence that works on a fresh host
The quick install said `sudo hpcod setup`, which is "command not found"
until setup has put itself in /usr/local/bin, and never said that a
release binary arrives without its execute bit and under its release
name. All four places now give the same sequence - download, check
SHA256SUMS, `install -m 0755 ... hpcod`, `sudo ./hpcod setup` - run as
written on the trial host from the v0.2.1 release files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:20:09 +01:00

8.9 KiB

Warewulf HPC Test Cluster

A working Warewulf 4 HPC lab on Proxmox (hv01) with its own isolated subnet, central identity (FreeIPA) and shared NFS home directories.

Status: built and verified. Four diskless nodes PXE-boot from the head node, enrol into the IPA domain, mount NFS homes, register idle in Slurm, and run an 8-rank MPI job across all four as a directory user.

                              Internet
                                  │
                         vmbr0 (10.0.0.0/24)  ← management network
                    ┌─────────────┴─────────────┐
        ┌───────────┴──────────┐    ┌───────────┴───────────┐
        │  head01   (VM 300)   │    │   idm1     (VM 305)   │
        │  eth0 10.0.0.30      │    │   eth0 10.0.0.31      │ ← SSH / admin
        │  Warewulf · DHCP     │    │   FreeIPA (hpc.local) │
        │  TFTP/iPXE · Slurm   │    │   LDAP · Kerberos     │
        │  MUNGE · NAT gateway │    │   CA · DNS            │
        │  NFS: /opt (ro)      │    │   NFS: /export/home   │
        └───────────┬──────────┘    └───────────┬───────────┘
        eth1 10.10.10.1                eth1 10.10.10.5
                    └─────────────┬─────────────┘
                                  │
                       vmbr6 ← pure L2, NO IP on Proxmox
                                  │
        ┌───────────┬─────────────┼─────────────┬───────────┐
     wwnode1     wwnode2       wwnode3       wwnode4   (VM 301-304)
    10.10.10.11 10.10.10.12   10.10.10.13   10.10.10.14
     diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
     /home → idm1:/export/home      /opt → head01:/opt (ro)

Documentation

A 25-page PDF edition of the whole handbook: docs/warewulf-cluster-handbook.pdf. Regenerate it with ./scripts/build-pdf.sh after editing docs/handbook-print.html.

Document Contents
01 — Architecture Topology, why vmbr6 has no IP, why idm1 is dual-homed, the stateless boot chain
02 — Installation Installing hpcod from scratch: one binary, sudo ./hpcod setup, verify and a test cluster
03 — Warewulf Images, overlays, templates, node management
04 — Identity FreeIPA vs AD, enrolling stateless nodes, users and groups
05 — Storage NFS layout, id mapping, security posture, path to Kerberised NFS
06 — Slurm & MPI Scheduler config, running jobs, what is not configured
07 — Operations Runbook: access, power, health checks, common tasks, logs
08 — Troubleshooting The 12 pitfalls hit during the build, plus a diagnostic guide
09 — Reference Inventory, addressing, ports, versions, file locations
10 — Building the lab The reference lab built by hand: script order, what each does, the two manual steps, timings
20 — HPC-on-Demand Design: multi-site control plane that provisions clusters like this one on demand
21 — Portal layer What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to
22 — Spack & Chapar Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first
23 — Application lifecycle What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them
hpcod/ Implementation: multi-provider discovery, placement, and the portal at hpc.example.org
hpcod/DEPLOY.md Installing hpcod against your own Proxmox: sudo ./hpcod setup, from the binary alone
CHANGELOG What is in each release, and what is knowingly not

Quick start

ssh head01                  # head node
ssh idm1                    # identity + home server

sinfo                       # cluster state
sudo wwctl node status      # provisioning state
./scripts/A99-verify.sh     # full end-to-end health check

Compute nodes live on the isolated subnet — reach them through the head node:

ssh head01
sudo ssh root@wwnode1

HPC-on-Demand portal — https://hpc.example.org (see hpcod/). Users ask for a cluster there, and it is built, published and handed over. To install hpcod somewhere else, download the release binary and run sudo ./hpcod setup (docs/02-installation.md).

Power management:

ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset    # reboot nodes

Software

Component Version
Proxmox VE 8.4.19
Host / node OS Rocky Linux 9.8
Warewulf 4.7.0
OpenHPC 3 (EL_9)
Slurm 25.11.4
OpenMPI 5.0.7 (GCC 14.2)
FreeIPA 4.13.1
SSSD 2.9.8

Three decisions worth knowing

vmbr6 has no IP on Proxmox. Warewulf needs to own the entire subnet — DHCP with PXE options, TFTP, address management. A second authority on the same segment would collide. The bridge is a pure L2 switch and head01 is the gateway. See 01-architecture §1.3.

idm1 is dual-homed, and idm1.hpc.local resolves to the cluster side. Behind head01's NAT all four nodes would appear as 10.0.0.30, which breaks Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every client a direct, symmetric path. See 01-architecture §1.4.

Stateless nodes cannot run ipa-client-install. The rootfs is rebuilt in RAM every boot, so enrolment happens once and the result is distributed through Warewulf overlays — shared config as templates rendered per node, and a separate keytab per node so wwnode1 cannot impersonate wwnode2. See 04-identity §4.3.

Verified output

================= IPA SERVER (idm1) =================
  IPA services running: 9
  /export/home  10.10.10.0/24(rw,no_root_squash,sec=sys,...)
  IPA users: 3

================= HEAD NODE (head01) =================
  hostname : head01.hpc.local
  DNS      : 10.10.10.5
  /home    : idm1.hpc.local:/export/home
  hpcuser  : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
  services : warewulfd=active dhcpd=active tftp.socket=active
             slurmctld=active munge=active sssd=active
    PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
    normal*      up   infinite      4   idle wwnode[1-4]

================= COMPUTE NODES =================
  wwnode1: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode2: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode3: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode4: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK

================= MPI JOB (IPA user, NFS home) =================
  Rank  0 /  8  host=wwnode1    mpi_proc=wwnode1
  Ring complete. start=42, returned=49, expected=49, hops=8  -> SUCCESS

krb5=OK means the node actually obtained a ticket from the KDC using its own host keytab (kinit -k host/wwnodeN.hpc.local) — not merely that the file is present.

Additionally verified by hand: an IPA user SSH'd into a stateless node with a password, received a Kerberos TGT and landed in their NFS home; and a brand-new IPA user's home was auto-created on the NFS share at first login.

Known gaps

  • NFS is sec=sys, not Kerberised. Host keytabs and the nfs/idm1.hpc.local service principal already exist, so sec=krb5p is a short step — but it changes the home-provisioning flow.
  • Slurm accounting is recorded but not enforced — sacct and sreport work; AccountingStorageEnforce is unset, so no association is required to submit.
  • SELinux permissive on head and nodes.
  • FreeIPA is not Active Directory. It is the RHEL-native equivalent for Linux clients; Windows joins or the AD schema would need a Samba AD DC or an AD trust. See 04-identity §4.1.

Full list with rationale in 09-reference §9.9.