Files
werewulf/docs/09-reference.md
T
ilker 6876e0c5bd Replace the Warewulf overlay UI with the HPC-on-Demand portal
hpc.example.org now serves the HPC-on-Demand portal, and the overlay UI is
gone - service, code, docs and the API it depended on.

The portal exposes phases 1 and 2 over HTTP: what the estate looks like,
which capabilities are schedulable and which are merely claimed, a dry-run
form, and the curated device map, plus a JSON API. It is entirely read-only
and provisions nothing.

It runs on the Caddy host bound to 127.0.0.1:9880, so the reverse proxy is the
only thing that can reach it - a smaller surface than the overlay UI had,
which listened on the management network. It refuses to bind off loopback
without a user file, for the same reason: the portal knows the shape and spare
capacity of the whole estate and holds read credentials for it. Discovery runs
on a two-minute timer rather than per request, since polling every backing
cloud on each page load would hammer their APIs and make the portal as slow as
its slowest site; pages report the inventory's age instead.

An unauthenticated /v1/ request now returns 401 rather than redirecting to the
login page. Sending a script an HTML form instead of a status code it can act
on is the kind of thing that wastes an afternoon.

Removing the overlay UI also disabled the Warewulf REST API on head01, which
had been enabled solely to serve it. Leaving root-equivalent access switched
on with no consumer would be a liability, so warewulf.conf and auth.conf were
restored from the pre-feature backups and the firewall rule dropped. The
cluster was re-verified afterwards: four nodes idle, an eight-rank MPI job
completing.

Also adds the phase 3 Terraform module for one tenant - isolated bridge, head
node and diskless compute nodes with deterministic MACs, since Warewulf
matches nodes by MAC and a regenerated one would orphan the registration. Not
yet applied against a real site.
2026-09-01 12:24:54 +02:00

8.7 KiB
Raw Permalink Blame History

9. Reference

9.1 Virtual machine inventory

VMID Name vCPU RAM Disk Boot onboot
300 head01 4 8 GB 100 GB (zfspool) scsi0 yes
301 wwnode1 2 8 GB none net0 (PXE) no
302 wwnode2 2 8 GB none net0 (PXE) no
303 wwnode3 2 8 GB none net0 (PXE) no
304 wwnode4 2 8 GB none net0 (PXE) no
305 idm1 4 8 GB 100 GB (zfspool) scsi0 yes

Compute nodes additionally carry balloon: 0 (ballooning would fight the tmpfs rootfs), serial0: socket and vga: serial0.

9.2 Network addressing

Host Interface Bridge MAC Address
head01 eth0 vmbr0 BC:24:11:E0:00:FF 10.0.0.30/24
head01 eth1 vmbr6 BC:24:11:E0:0A:01 10.10.10.1/24
idm1 eth0 vmbr0 BC:24:11:E0:00:31 10.0.0.31/24
idm1 eth1 vmbr6 BC:24:11:E0:0A:05 10.10.10.5/24
wwnode1 eth0 vmbr6 BC:24:11:E0:0A:11 10.10.10.11/24
wwnode2 eth0 vmbr6 BC:24:11:E0:0A:12 10.10.10.12/24
wwnode3 eth0 vmbr6 BC:24:11:E0:0A:13 10.10.10.13/24
wwnode4 eth0 vmbr6 BC:24:11:E0:0A:14 10.10.10.14/24

MAC convention: BC:24:11:E0:xx:yy — 00: prefix for management NICs, 0A: for cluster NICs, node index in the last octet.

DHCP pool: 10.10.10.100–199, transient PXE leases only. Gateway for 10.10.10.0/24: 10.10.10.1 (head01), NAT out via vmbr0.

Bridges

Bridge Address Role
vmbr0 10.0.0.10/24 (Proxmox) Management, pre-existing
vmbr6 none Cluster fabric, L2 only, created for this build

9.3 DNS names

Name Address Notes
idm1.hpc.local 10.10.10.5 IPA/NFS service address
idm1-mgmt.hpc.local 10.0.0.31 Management address
head01.hpc.local 10.10.10.1 Head node, cluster side
head01-mgmt.hpc.local 10.0.0.30 Head node, management side
wwnode1-4.hpc.local 10.10.10.11–14 Compute nodes
warewulf 10.10.10.1 /etc/hosts alias from the Warewulf hosts overlay

Reverse zone 10.10.10.in-addr.arpa holds matching PTR records.

9.4 Ports

Port Proto Host Service
67/68 UDP head01 DHCP (PXE)
69 UDP head01 TFTP (iPXE bootloaders)
9873 TCP head01 warewulfd provisioning HTTP
9880 TCP web-frontend HPC-on-Demand portal, loopback only
443 TCP web-frontend https://hpc.example.org → 127.0.0.1:9880 (Caddy, Let's Encrypt)
9874 TCP head01 warewulfd TLS
6817 TCP head01 slurmctld
6818 TCP nodes slurmd
6819 TCP head01 slurmdbd (accounting)
3306 TCP head01 MariaDB, localhost only
2049 TCP idm1, head01 NFSv4
53 TCP/UDP idm1 DNS (bind-dyndb-ldap)
88, 464 TCP/UDP idm1 Kerberos KDC, kpasswd
749 TCP idm1 kadmin
389, 636 TCP idm1 LDAP, LDAPS (389-ds)
80, 443 TCP idm1 IPA web UI / CA

Firewall posture: on both servers eth1 (cluster) is in the firewalld trusted zone; eth0 (management) is public with only the required services opened.

9.5 Software versions

Component Version
Proxmox VE 8.4.19
Host / node OS Rocky Linux 9.8 (Blue Onyx)
Node kernel 5.14.0-687.41.1.el9_8
Warewulf 4.7.0-1.el9
Node image source ghcr.io/warewulf/warewulf-rockylinux:9
OpenHPC 3 (EL_9)
Slurm 25.11.4 (slurm-ohpc)
OpenMPI 5.0.7 (openmpi5-gnu14-ohpc)
GCC 14.2.0 (gnu14-compilers-ohpc)
Lmod 9.2
MUNGE 0.5.13
FreeIPA 4.13.1-3.el9_8.2
389-ds-base 2.8.0-9.el9_8
krb5-server 1.21.1-10.el9_8
SSSD 2.9.8-4.el9_8.1

9.6 Identity objects

Object ID Notes
Realm HPC.LOCAL
Domain hpc.local
hpcusers (group) 721800003 All cluster users
admin 721800000 IPA administrator
hpcuser 721800005 Runs MPI test jobs
testuser 721800006 Login verification

Host principals: host/wwnode1.hpc.local … host/wwnode4.hpc.local, host/head01.hpc.local, host/idm1.hpc.local. Service principal nfs/idm1.hpc.local exists but is unused (sec=sys today).

Credentials: idm1:/root/.ipa-credentials, mode 0600. Not in this repo.

9.7 Key file locations

head01

Path Contents
/etc/warewulf/warewulf.conf Warewulf global config
/etc/warewulf/nodes.conf Node/profile definitions
/var/lib/warewulf/overlays/ Site overlay sources
/var/lib/warewulf/provision/images/ Served node images
/etc/slurm/slurm.conf Slurm config (mirrored into the hpc overlay)
/etc/munge/munge.key MUNGE key (mirrored into the hpc overlay)
/root/node-keytabs/ Per-node Kerberos keytabs
/root/.ssh/cluster head→node SSH key
/var/home/ilker Local admin home, moved out of /home

idm1

Path Contents
/root/.ipa-credentials All lab passwords
/root/cacert.p12 CA private key — irreplaceable
/root/node-keytabs/ Generated keytabs
/export/home/ NFS-exported user homes
/etc/exports NFS export definitions

Compute nodes (in RAM, from overlays)

Path Source overlay
/etc/krb5.conf, /etc/ipa/ca.crt ipa
/etc/sssd/sssd.conf, /etc/ipa/default.conf ipa (templated)
/etc/krb5.keytab ipa.wwnodeN
/etc/slurm/slurm.conf, /etc/munge/munge.key hpc
/etc/resolv.conf, /etc/fstab, NM connection stock Warewulf overlays

9.8 Repository layout

werewulf/
├── README.md                        index and quickstart
├── docs/
│   ├── 01-architecture.md           topology and design rationale
│   ├── 02-installation.md           build from scratch
│   ├── 03-warewulf.md               images, overlays, node management
│   ├── 04-identity.md               FreeIPA, SSSD, stateless enrolment
│   ├── 05-storage.md                NFS layout and security posture
│   ├── 06-slurm-mpi.md              scheduler and MPI
│   ├── 07-operations.md             runbook
│   ├── 08-troubleshooting.md        12 pitfalls + diagnostics
│   ├── 09-reference.md              this file
│   ├── handbook-print.html          print edition source (fonts embedded)
│   └── warewulf-cluster-handbook.pdf  22-page PDF edition
├── scripts/
│   ├── 00-proxmox-provision.sh      cluster tier
│   ├── 10-head-base.sh
│   ├── 20-warewulf-install.sh
│   ├── 30-warewulf-configure.sh
│   ├── 40-node-image.sh
│   ├── 50-nodes-register.sh
│   ├── 60-slurm-openhpc.sh
│   ├── 70-mpi-test.sh
│   ├── A00-proxmox-ipa-vm.sh        identity tier
│   ├── A10-ipa-server.sh
│   ├── A20-ipa-nfs-home.sh
│   ├── A30-head01-ipa.sh
│   ├── A40-ipa-node-hosts.sh
│   ├── A50-node-image-ipa.sh
│   ├── A60-node-overlays-ipa.sh
│   ├── A99-verify.sh                verification
│   ├── 99-cluster-ctl.sh            power management
│   ├── A85-slurm-accounting.sh      slurmdbd + MariaDB accounting
│   ├── build-pdf.sh                 regenerate the PDF edition
│   └── run-all.sh                   orchestrates the cluster tier
└── mpi/
    ├── hello_mpi.c                  ring test program
    ├── heat_mpi.c                   long-running benchmark
    ├── mpi-ring.sbatch              Slurm batch job
    └── heat-long.sbatch             long-running batch job
    ├── static/login.html            sign-in page
    └── Makefile                     cross-compile and deploy

9.9 Deliberate choices and known gaps

Choice Rationale Production alternative
SELinux permissive on head and nodes Simplifies PXE/NFS/overlay debugging enforcing; FreeIPA and Warewulf both support it
NFS sec=sys Kerberised NFS changes the home-provisioning flow sec=krb5p (groundwork already laid)
no_root_squash on /export/home Needed by pam_oddjob_mkhomedir root_squash + provisioning hook
No Slurm accounting slurmdbd + MariaDB not needed for a functional test slurm-slurmdbd-ohpc + MariaDB
Single IPA server Lab IPA replica for redundancy
Single slurmctld Lab Backup SlurmctldHost
Nodes onboot=0 Deliberate start ordering after idm1/head01 onboot=1 with ordering constraints
FreeIPA rather than AD Samba AD DC unsupported on RHEL 9 Samba AD DC on Debian, or an AD trust
No backups Test environment ipa-backup, /export/home snapshots