Files
werewulf/README.md
T
ilkerandClaude Opus 5 a6d453ae7b hpcod: application images, and a catalogue generated from them
The catalogue published one image. Filling it is not nine more
Containerfiles: it is a question of who compiles the software, against
which MPI, and how a second cluster gets it without paying for the
first one's build. image/apps holds the answer to the first of those -
one driver over one directory per application, with three recipes
(somebody else's image, one we assemble, or Spack against our own MPI)
and the rule for choosing between them.

The catalogue is generated from the recipes rather than kept beside
them, because an entry in it is a promise that a head node can pull the
image, and a promise nobody checks is how the first generated catalogue
came to point busybox at a tag that had never existed - guessed from
the upstream version instead of read from the registry. The generator
now asks the registry, and an application whose image is not there is
reported rather than offered.

Recipes are intent and published.json is measurement, kept apart so
that recording the hash of a push does not rewrite the reasoning in the
recipe that produced it.

Two entries to start: busybox, which is the cheapest way to prove the
whole delivery path, and stress-ng, which is what somebody actually
runs first on a cluster they have just been handed.

docs/22 is the Spack half of the question, from reading Chapar: what it
would give us, the three levels of integration, and the PMIx version to
settle on a real head before a single image is built against it.

Two things about the build host, both found by trying: Apptainer is not
installed, and Docker is, which sets FORWARD DROP and leaves podman
unable to build anything that needs the network.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 10:25:37 +02:00

8.5 KiB

Warewulf HPC Test Cluster

A working Warewulf 4 HPC lab on Proxmox (hv01) with its own isolated subnet, central identity (FreeIPA) and shared NFS home directories.

Status: built and verified. Four diskless nodes PXE-boot from the head node, enrol into the IPA domain, mount NFS homes, register idle in Slurm, and run an 8-rank MPI job across all four as a directory user.

                              Internet
                                  │
                         vmbr0 (10.0.0.0/24)  ← management network
                    ┌─────────────┴─────────────┐
        ┌───────────┴──────────┐    ┌───────────┴───────────┐
        │  head01   (VM 300)   │    │   idm1     (VM 305)   │
        │  eth0 10.0.0.30      │    │   eth0 10.0.0.31      │ ← SSH / admin
        │  Warewulf · DHCP     │    │   FreeIPA (hpc.local) │
        │  TFTP/iPXE · Slurm   │    │   LDAP · Kerberos     │
        │  MUNGE · NAT gateway │    │   CA · DNS            │
        │  NFS: /opt (ro)      │    │   NFS: /export/home   │
        └───────────┬──────────┘    └───────────┬───────────┘
        eth1 10.10.10.1                eth1 10.10.10.5
                    └─────────────┬─────────────┘
                                  │
                       vmbr6 ← pure L2, NO IP on Proxmox
                                  │
        ┌───────────┬─────────────┼─────────────┬───────────┐
     wwnode1     wwnode2       wwnode3       wwnode4   (VM 301-304)
    10.10.10.11 10.10.10.12   10.10.10.13   10.10.10.14
     diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
     /home → idm1:/export/home      /opt → head01:/opt (ro)

Documentation

A 22-page PDF edition of the whole handbook: docs/warewulf-cluster-handbook.pdf. Regenerate it with ./scripts/build-pdf.sh after editing docs/handbook-print.html.

Document Contents
01 — Architecture Topology, why vmbr6 has no IP, why idm1 is dual-homed, the stateless boot chain
02 — Installation Building from scratch, what each script does, timings
03 — Warewulf Images, overlays, templates, node management
04 — Identity FreeIPA vs AD, enrolling stateless nodes, users and groups
05 — Storage NFS layout, id mapping, security posture, path to Kerberised NFS
06 — Slurm & MPI Scheduler config, running jobs, what is not configured
07 — Operations Runbook: access, power, health checks, common tasks, logs
08 — Troubleshooting The 12 pitfalls hit during the build, plus a diagnostic guide
09 — Reference Inventory, addressing, ports, versions, file locations
20 — HPC-on-Demand Design: multi-site control plane that provisions clusters like this one on demand
21 — Portal layer What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to
22 — Spack & Chapar Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first
23 — Application lifecycle What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them
hpcod/ Implementation: multi-provider discovery, placement, and the portal at hpc.example.org
CHANGELOG What is in each release, and what is knowingly not

Quick start

ssh head01                  # head node
ssh idm1                    # identity + home server

sinfo                       # cluster state
sudo wwctl node status      # provisioning state
./scripts/A99-verify.sh     # full end-to-end health check

Compute nodes live on the isolated subnet — reach them through the head node:

ssh head01
sudo ssh root@wwnode1

HPC-on-Demand portal — https://hpc.example.org (see hpcod/). Read-only: what the estate looks like, and where a request would be placed.

Power management:

ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset    # reboot nodes

Software

Component Version
Proxmox VE 8.4.19
Host / node OS Rocky Linux 9.8
Warewulf 4.7.0
OpenHPC 3 (EL_9)
Slurm 25.11.4
OpenMPI 5.0.7 (GCC 14.2)
FreeIPA 4.13.1
SSSD 2.9.8

Three decisions worth knowing

vmbr6 has no IP on Proxmox. Warewulf needs to own the entire subnet — DHCP with PXE options, TFTP, address management. A second authority on the same segment would collide. The bridge is a pure L2 switch and head01 is the gateway. See 01-architecture §1.3.

idm1 is dual-homed, and idm1.hpc.local resolves to the cluster side. Behind head01's NAT all four nodes would appear as 10.0.0.30, which breaks Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every client a direct, symmetric path. See 01-architecture §1.4.

Stateless nodes cannot run ipa-client-install. The rootfs is rebuilt in RAM every boot, so enrolment happens once and the result is distributed through Warewulf overlays — shared config as templates rendered per node, and a separate keytab per node so wwnode1 cannot impersonate wwnode2. See 04-identity §4.3.

Verified output

================= IPA SERVER (idm1) =================
  IPA services running: 9
  /export/home  10.10.10.0/24(rw,no_root_squash,sec=sys,...)
  IPA users: 3

================= HEAD NODE (head01) =================
  hostname : head01.hpc.local
  DNS      : 10.10.10.5
  /home    : idm1.hpc.local:/export/home
  hpcuser  : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
  services : warewulfd=active dhcpd=active tftp.socket=active
             slurmctld=active munge=active sssd=active
    PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
    normal*      up   infinite      4   idle wwnode[1-4]

================= COMPUTE NODES =================
  wwnode1: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode2: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode3: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode4: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK

================= MPI JOB (IPA user, NFS home) =================
  Rank  0 /  8  host=wwnode1    mpi_proc=wwnode1
  Ring complete. start=42, returned=49, expected=49, hops=8  -> SUCCESS

krb5=OK means the node actually obtained a ticket from the KDC using its own host keytab (kinit -k host/wwnodeN.hpc.local) — not merely that the file is present.

Additionally verified by hand: an IPA user SSH'd into a stateless node with a password, received a Kerberos TGT and landed in their NFS home; and a brand-new IPA user's home was auto-created on the NFS share at first login.

Known gaps

  • NFS is sec=sys, not Kerberised. Host keytabs and the nfs/idm1.hpc.local service principal already exist, so sec=krb5p is a short step — but it changes the home-provisioning flow.
  • Slurm accounting is recorded but not enforced — sacct and sreport work; AccountingStorageEnforce is unset, so no association is required to submit.
  • SELinux permissive on head and nodes.
  • FreeIPA is not Active Directory. It is the RHEL-native equivalent for Linux clients; Windows joins or the AD schema would need a Samba AD DC or an AD trust. See 04-identity §4.1.

Full list with rationale in 09-reference §9.9.