Files
werewulf/docs/09-reference.md
T
ilkerandClaude Opus 5.5 5b6e1ed3dd docs: section 02 installs hpcod from scratch; the lab build moves to 10
02 now walks an empty host through `hpcod setup`: what it needs, what the
TUI asks and does, the simulated Proxmox, reaching the portal, verify and
selftest, with timings measured on one run. The hand-built lab's script
order moves unchanged to 10-lab-installation.md. README, 01, 09 and the
handbook (contents, chapter 2 with two screenshots, new chapter 10) follow;
the PDF is regenerated, now 25 pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:08:32 +01:00

10 KiB
Raw Permalink Blame History

9. Reference

9.1 Virtual machine inventory

VMID Name vCPU RAM Disk Boot onboot
300 head01 4 8 GB 100 GB (zfspool) scsi0 yes
301 wwnode1 2 8 GB none net0 (PXE) no
302 wwnode2 2 8 GB none net0 (PXE) no
303 wwnode3 2 8 GB none net0 (PXE) no
304 wwnode4 2 8 GB none net0 (PXE) no
305 idm1 4 8 GB 100 GB (zfspool) scsi0 yes

Compute nodes additionally carry balloon: 0 (ballooning would fight the tmpfs rootfs), serial0: socket and vga: serial0.

9.2 Network addressing

Host Interface Bridge MAC Address
head01 eth0 vmbr0 BC:24:11:E0:00:FF 10.0.0.30/24
head01 eth1 vmbr6 BC:24:11:E0:0A:01 10.10.10.1/24
idm1 eth0 vmbr0 BC:24:11:E0:00:31 10.0.0.31/24
idm1 eth1 vmbr6 BC:24:11:E0:0A:05 10.10.10.5/24
wwnode1 eth0 vmbr6 BC:24:11:E0:0A:11 10.10.10.11/24
wwnode2 eth0 vmbr6 BC:24:11:E0:0A:12 10.10.10.12/24
wwnode3 eth0 vmbr6 BC:24:11:E0:0A:13 10.10.10.13/24
wwnode4 eth0 vmbr6 BC:24:11:E0:0A:14 10.10.10.14/24

MAC convention: BC:24:11:E0:xx:yy — 00: prefix for management NICs, 0A: for cluster NICs, node index in the last octet.

DHCP pool: 10.10.10.100–199, transient PXE leases only. Gateway for 10.10.10.0/24: 10.10.10.1 (head01), NAT out via vmbr0.

Bridges

Bridge Address Role
vmbr0 10.0.0.10/24 (Proxmox) Management, pre-existing
vmbr6 none Cluster fabric, L2 only, created for this build

9.3 DNS names

Name Address Notes
idm1.hpc.local 10.10.10.5 IPA/NFS service address
idm1-mgmt.hpc.local 10.0.0.31 Management address
head01.hpc.local 10.10.10.1 Head node, cluster side
head01-mgmt.hpc.local 10.0.0.30 Head node, management side
wwnode1-4.hpc.local 10.10.10.11–14 Compute nodes
warewulf 10.10.10.1 /etc/hosts alias from the Warewulf hosts overlay

Reverse zone 10.10.10.in-addr.arpa holds matching PTR records.

9.4 Ports

Port Proto Host Service
67/68 UDP head01 DHCP (PXE)
69 UDP head01 TFTP (iPXE bootloaders)
9873 TCP head01 warewulfd provisioning HTTP
9880 TCP control plane HPC-on-Demand portal, loopback only
443 TCP reverse proxy https://hpc.example.org → 127.0.0.1:9880 (Caddy, wildcard certificate)
443, 80 TCP control plane hpcod itself, when it serves HTTPS with its own CA (closed networks; 80 redirects and serves /ca.crt)
53 TCP/UDP control plane hpcod's DNS for the portal and *.CLUSTER_DOMAIN, when enabled
9874 TCP head01 warewulfd TLS
6817 TCP head01 slurmctld
6818 TCP nodes slurmd
6819 TCP head01 slurmdbd (accounting)
3306 TCP head01 MariaDB, localhost only
2049 TCP idm1, head01 NFSv4
53 TCP/UDP idm1 DNS (bind-dyndb-ldap)
88, 464 TCP/UDP idm1 Kerberos KDC, kpasswd
749 TCP idm1 kadmin
389, 636 TCP idm1 LDAP, LDAPS (389-ds)
80, 443 TCP idm1 IPA web UI / CA

Firewall posture: on both servers eth1 (cluster) is in the firewalld trusted zone; eth0 (management) is public with only the required services opened.

9.5 Software versions

Component Version
Proxmox VE 8.4.19
Host / node OS Rocky Linux 9.8 (Blue Onyx)
Node kernel 5.14.0-687.41.1.el9_8
Warewulf 4.7.0-1.el9
Node image source ghcr.io/warewulf/warewulf-rockylinux:9
OpenHPC 3 (EL_9)
Slurm 25.11.4 (slurm-ohpc)
OpenMPI 5.0.7 (openmpi5-gnu14-ohpc)
GCC 14.2.0 (gnu14-compilers-ohpc)
Lmod 9.2
MUNGE 0.5.13
FreeIPA 4.13.1-3.el9_8.2
389-ds-base 2.8.0-9.el9_8
krb5-server 1.21.1-10.el9_8
SSSD 2.9.8-4.el9_8.1

9.6 Identity objects

Object ID Notes
Realm HPC.LOCAL
Domain hpc.local
hpcusers (group) 721800003 All cluster users
admin 721800000 IPA administrator
hpcuser 721800005 Runs MPI test jobs
testuser 721800006 Login verification

Host principals: host/wwnode1.hpc.local … host/wwnode4.hpc.local, host/head01.hpc.local, host/idm1.hpc.local. Service principal nfs/idm1.hpc.local exists but is unused (sec=sys today).

Credentials: idm1:/root/.ipa-credentials, mode 0600. Not in this repo.

9.7 Key file locations

head01

Path Contents
/etc/warewulf/warewulf.conf Warewulf global config
/etc/warewulf/nodes.conf Node/profile definitions
/var/lib/warewulf/overlays/ Site overlay sources
/var/lib/warewulf/provision/images/ Served node images
/etc/slurm/slurm.conf Slurm config (mirrored into the hpc overlay)
/etc/munge/munge.key MUNGE key (mirrored into the hpc overlay)
/root/node-keytabs/ Per-node Kerberos keytabs
/root/.ssh/cluster head→node SSH key
/var/home/ilker Local admin home, moved out of /home

idm1

Path Contents
/root/.ipa-credentials All lab passwords
/root/cacert.p12 CA private key — irreplaceable
/root/node-keytabs/ Generated keytabs
/export/home/ NFS-exported user homes
/etc/exports NFS export definitions

Compute nodes (in RAM, from overlays)

Path Source overlay
/etc/krb5.conf, /etc/ipa/ca.crt ipa
/etc/sssd/sssd.conf, /etc/ipa/default.conf ipa (templated)
/etc/krb5.keytab ipa.wwnodeN
/etc/slurm/slurm.conf, /etc/munge/munge.key hpc
/etc/resolv.conf, /etc/fstab, NM connection stock Warewulf overlays

9.8 Repository layout

werewulf/
├── README.md                        index and quickstart
├── CHANGELOG.md                     releases
├── docs/
│   ├── 01-architecture.md           the reference lab: topology and rationale
│   ├── 02-installation.md           installing hpcod from scratch
│   ├── 03-warewulf.md               images, overlays, node management
│   ├── 04-identity.md               FreeIPA, SSSD, stateless enrolment
│   ├── 05-storage.md                NFS layout and security posture
│   ├── 06-slurm-mpi.md              scheduler and MPI
│   ├── 07-operations.md             runbook
│   ├── 08-troubleshooting.md        12 pitfalls + diagnostics
│   ├── 09-reference.md              this file
│   ├── 10-lab-installation.md       building the lab by hand
│   ├── 20-hpc-on-demand-architecture.md   the control plane
│   ├── 21-portal-layer.md           per-cluster Open OnDemand
│   ├── 22-spack-chapar-integration.md     building application images
│   ├── 23-application-lifecycle.md  catalogue states and retention
│   ├── handbook-print.html          print edition source (fonts embedded)
│   └── warewulf-cluster-handbook.pdf  PDF edition of 01–10
├── hpcod/                           the HPC-on-Demand control plane
│   ├── DEPLOY.md                    installing it: `hpcod setup`
│   ├── cmd/hpcod/                   the one binary: serve, setup, verify, selftest, apps …
│   ├── internal/                    placement, store, provisioning, portal, setup TUI
│   ├── ansible/                     head node roles, embedded in the binary
│   ├── terraform/                   the per-cluster module, embedded
│   ├── db/                          schema migrations, embedded
│   ├── config/apps.yaml             application catalogue, embedded
│   ├── image/                       node, desktop and application images
│   └── deploy/                      site.env.example, Caddyfile.example
├── scripts/
│   ├── 00-proxmox-provision.sh      lab: cluster tier
│   ├── 10-head-base.sh
│   ├── 20-warewulf-install.sh
│   ├── 30-warewulf-configure.sh
│   ├── 40-node-image.sh
│   ├── 50-nodes-register.sh
│   ├── 60-slurm-openhpc.sh
│   ├── 70-mpi-test.sh
│   ├── A00-proxmox-ipa-vm.sh        lab: identity tier
│   ├── A10-ipa-server.sh
│   ├── A20-ipa-nfs-home.sh
│   ├── A30-head01-ipa.sh
│   ├── A40-ipa-node-hosts.sh
│   ├── A50-node-image-ipa.sh
│   ├── A60-node-overlays-ipa.sh
│   ├── A85-slurm-accounting.sh      slurmdbd + MariaDB accounting
│   ├── A99-verify.sh                verification
│   ├── 99-cluster-ctl.sh            power management
│   ├── B30-cluster-inventory.py     hpcod: Ansible inventory from tofu output
│   ├── B40-publish-cluster.py       hpcod: publish a cluster portal by hand
│   ├── C10-publish-release.sh       release attachments
│   ├── build-pdf.sh                 regenerate the PDF edition
│   └── run-all.sh                   orchestrates the cluster tier
└── mpi/
    ├── hello_mpi.c                  ring test program
    ├── heat_mpi.c                   long-running benchmark
    ├── mpi-ring.sbatch              Slurm batch job
    └── heat-long.sbatch             long-running batch job

9.9 Deliberate choices and known gaps

Choice Rationale Production alternative
SELinux permissive on head and nodes Simplifies PXE/NFS/overlay debugging enforcing; FreeIPA and Warewulf both support it
NFS sec=sys Kerberised NFS changes the home-provisioning flow sec=krb5p (groundwork already laid)
no_root_squash on /export/home Needed by pam_oddjob_mkhomedir root_squash + provisioning hook
Slurm accounting recorded, not enforced A85 added slurmdbd; no association is required to submit (§6.6) AccountingStorageEnforce=associations,limits,qos
Single IPA server Lab IPA replica for redundancy
Single slurmctld Lab Backup SlurmctldHost
Nodes onboot=0 Deliberate start ordering after idm1/head01 onboot=1 with ordering constraints
FreeIPA rather than AD Samba AD DC unsupported on RHEL 9 Samba AD DC on Debian, or an AD trust
No backups Test environment ipa-backup, /export/home snapshots