ilkerandClaude Opus 5.5 3297de5d50
ci / derle (push) Successful in 22s
Apache License 2.0
The repository had no license, so it was public but not open source:
nobody could use, change or redistribute it. LICENSE is the text from
apache.org, unmodified. NOTICE says that the IBM Plex fonts embedded in
the print handbook are under the SIL Open Font License, not Apache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:27:25 +01:00
2026-11-16 20:22:09 +01:00
2026-11-16 20:27:25 +01:00
2026-08-31 10:35:56 +02:00
2026-11-16 20:22:09 +01:00
2026-11-16 20:27:25 +01:00
2026-11-16 20:27:25 +01:00
2026-11-16 20:27:25 +01:00

Warewulf HPC Test Cluster

A working Warewulf 4 HPC lab on Proxmox (hv01) with its own isolated subnet, central identity (FreeIPA) and shared NFS home directories.

Status: built and verified. Four diskless nodes PXE-boot from the head node, enrol into the IPA domain, mount NFS homes, register idle in Slurm, and run an 8-rank MPI job across all four as a directory user.

                              Internet
                                  │
                         vmbr0 (10.0.0.0/24)  ← management network
                    ┌─────────────┴─────────────┐
        ┌───────────┴──────────┐    ┌───────────┴───────────┐
        │  head01   (VM 300)   │    │   idm1     (VM 305)   │
        │  eth0 10.0.0.30      │    │   eth0 10.0.0.31      │ ← SSH / admin
        │  Warewulf · DHCP     │    │   FreeIPA (hpc.local) │
        │  TFTP/iPXE · Slurm   │    │   LDAP · Kerberos     │
        │  MUNGE · NAT gateway │    │   CA · DNS            │
        │  NFS: /opt (ro)      │    │   NFS: /export/home   │
        └───────────┬──────────┘    └───────────┬───────────┘
        eth1 10.10.10.1                eth1 10.10.10.5
                    └─────────────┬─────────────┘
                                  │
                       vmbr6 ← pure L2, NO IP on Proxmox
                                  │
        ┌───────────┬─────────────┼─────────────┬───────────┐
     wwnode1     wwnode2       wwnode3       wwnode4   (VM 301-304)
    10.10.10.11 10.10.10.12   10.10.10.13   10.10.10.14
     diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
     /home → idm1:/export/home      /opt → head01:/opt (ro)

Documentation

A 25-page PDF edition of the whole handbook: docs/warewulf-cluster-handbook.pdf. Regenerate it with ./scripts/build-pdf.sh after editing docs/handbook-print.html.

Document Contents
01 — Architecture Topology, why vmbr6 has no IP, why idm1 is dual-homed, the stateless boot chain
02 — Installation Installing hpcod from scratch: one binary, sudo ./hpcod setup, verify and a test cluster
03 — Warewulf Images, overlays, templates, node management
04 — Identity FreeIPA vs AD, enrolling stateless nodes, users and groups
05 — Storage NFS layout, id mapping, security posture, path to Kerberised NFS
06 — Slurm & MPI Scheduler config, running jobs, what is not configured
07 — Operations Runbook: access, power, health checks, common tasks, logs
08 — Troubleshooting The 12 pitfalls hit during the build, plus a diagnostic guide
09 — Reference Inventory, addressing, ports, versions, file locations
10 — Building the lab The reference lab built by hand: script order, what each does, the two manual steps, timings
20 — HPC-on-Demand Design: multi-site control plane that provisions clusters like this one on demand
21 — Portal layer What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to
22 — Spack & Chapar Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first
23 — Application lifecycle What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them
hpcod/ Implementation: multi-provider discovery, placement, and the portal at hpc.example.org
hpcod/DEPLOY.md Installing hpcod against your own Proxmox: sudo ./hpcod setup, from the binary alone
CHANGELOG What is in each release, and what is knowingly not

Quick start

ssh head01                  # head node
ssh idm1                    # identity + home server

sinfo                       # cluster state
sudo wwctl node status      # provisioning state
./scripts/A99-verify.sh     # full end-to-end health check

Compute nodes live on the isolated subnet — reach them through the head node:

ssh head01
sudo ssh root@wwnode1

HPC-on-Demand portal — https://hpc.example.org (see hpcod/). Users ask for a cluster there, and it is built, published and handed over. To install hpcod somewhere else, download the release binary and run sudo ./hpcod setup (docs/02-installation.md).

Power management:

ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset    # reboot nodes

License

Apache License 2.0. The fonts embedded in the print handbook are under the SIL Open Font License; see NOTICE.

Software

Component Version
Proxmox VE 8.4.19
Host / node OS Rocky Linux 9.8
Warewulf 4.7.0
OpenHPC 3 (EL_9)
Slurm 25.11.4
OpenMPI 5.0.7 (GCC 14.2)
FreeIPA 4.13.1
SSSD 2.9.8

Three decisions worth knowing

vmbr6 has no IP on Proxmox. Warewulf needs to own the entire subnet — DHCP with PXE options, TFTP, address management. A second authority on the same segment would collide. The bridge is a pure L2 switch and head01 is the gateway. See 01-architecture §1.3.

idm1 is dual-homed, and idm1.hpc.local resolves to the cluster side. Behind head01's NAT all four nodes would appear as 10.0.0.30, which breaks Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every client a direct, symmetric path. See 01-architecture §1.4.

Stateless nodes cannot run ipa-client-install. The rootfs is rebuilt in RAM every boot, so enrolment happens once and the result is distributed through Warewulf overlays — shared config as templates rendered per node, and a separate keytab per node so wwnode1 cannot impersonate wwnode2. See 04-identity §4.3.

Verified output

================= IPA SERVER (idm1) =================
  IPA services running: 9
  /export/home  10.10.10.0/24(rw,no_root_squash,sec=sys,...)
  IPA users: 3

================= HEAD NODE (head01) =================
  hostname : head01.hpc.local
  DNS      : 10.10.10.5
  /home    : idm1.hpc.local:/export/home
  hpcuser  : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
  services : warewulfd=active dhcpd=active tftp.socket=active
             slurmctld=active munge=active sssd=active
    PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
    normal*      up   infinite      4   idle wwnode[1-4]

================= COMPUTE NODES =================
  wwnode1: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode2: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode3: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK
  wwnode4: sssd=active  slurmd=active  munge=active  home=nfs  krb5=OK

================= MPI JOB (IPA user, NFS home) =================
  Rank  0 /  8  host=wwnode1    mpi_proc=wwnode1
  Ring complete. start=42, returned=49, expected=49, hops=8  -> SUCCESS

krb5=OK means the node actually obtained a ticket from the KDC using its own host keytab (kinit -k host/wwnodeN.hpc.local) — not merely that the file is present.

Additionally verified by hand: an IPA user SSH'd into a stateless node with a password, received a Kerberos TGT and landed in their NFS home; and a brand-new IPA user's home was auto-created on the NFS share at first login.

Known gaps

  • NFS is sec=sys, not Kerberised. Host keytabs and the nfs/idm1.hpc.local service principal already exist, so sec=krb5p is a short step — but it changes the home-provisioning flow.
  • Slurm accounting is recorded but not enforced — sacct and sreport work; AccountingStorageEnforce is unset, so no association is required to submit.
  • SELinux permissive on head and nodes.
  • FreeIPA is not Active Directory. It is the RHEL-native equivalent for Linux clients; Windows joins or the AD schema would need a Samba AD DC or an AD trust. See 04-identity §4.1.

Full list with rationale in 09-reference §9.9.

S
Description
No description provided
Readme Apache-2.0
7.5 MiB
0 Stars 1 Watchers 0 Forks
v0.2.2
Latest
2026-11-16 20:22:43 +01:00
Languages
Go 77.1%
Shell 7.3%
HTML 4.6%
HCL 3%
Python 2.7%
Other 5.3%