9. Reference
9.1 Virtual machine inventory
| VMID |
Name |
vCPU |
RAM |
Disk |
Boot |
onboot |
| 300 |
head01 |
4 |
8 GB |
100 GB (zfspool) |
scsi0 |
yes |
| 301 |
wwnode1 |
2 |
8 GB |
none |
net0 (PXE) |
no |
| 302 |
wwnode2 |
2 |
8 GB |
none |
net0 (PXE) |
no |
| 303 |
wwnode3 |
2 |
8 GB |
none |
net0 (PXE) |
no |
| 304 |
wwnode4 |
2 |
8 GB |
none |
net0 (PXE) |
no |
| 305 |
idm1 |
4 |
8 GB |
100 GB (zfspool) |
scsi0 |
yes |
Compute nodes additionally carry balloon: 0 (ballooning would fight the
tmpfs rootfs), serial0: socket and vga: serial0.
9.2 Network addressing
| Host |
Interface |
Bridge |
MAC |
Address |
head01 |
eth0 |
vmbr0 |
BC:24:11:E0:00:FF |
10.0.0.30/24 |
head01 |
eth1 |
vmbr6 |
BC:24:11:E0:0A:01 |
10.10.10.1/24 |
idm1 |
eth0 |
vmbr0 |
BC:24:11:E0:00:31 |
10.0.0.31/24 |
idm1 |
eth1 |
vmbr6 |
BC:24:11:E0:0A:05 |
10.10.10.5/24 |
wwnode1 |
eth0 |
vmbr6 |
BC:24:11:E0:0A:11 |
10.10.10.11/24 |
wwnode2 |
eth0 |
vmbr6 |
BC:24:11:E0:0A:12 |
10.10.10.12/24 |
wwnode3 |
eth0 |
vmbr6 |
BC:24:11:E0:0A:13 |
10.10.10.13/24 |
wwnode4 |
eth0 |
vmbr6 |
BC:24:11:E0:0A:14 |
10.10.10.14/24 |
MAC convention: BC:24:11:E0:xx:yy — 00: prefix for management NICs,
0A: for cluster NICs, node index in the last octet.
DHCP pool: 10.10.10.100–199, transient PXE leases only.
Gateway for 10.10.10.0/24: 10.10.10.1 (head01), NAT out via vmbr0.
Bridges
| Bridge |
Address |
Role |
vmbr0 |
10.0.0.10/24 (Proxmox) |
Management, pre-existing |
vmbr6 |
none |
Cluster fabric, L2 only, created for this build |
9.3 DNS names
| Name |
Address |
Notes |
idm1.hpc.local |
10.10.10.5 |
IPA/NFS service address |
idm1-mgmt.hpc.local |
10.0.0.31 |
Management address |
head01.hpc.local |
10.10.10.1 |
Head node, cluster side |
head01-mgmt.hpc.local |
10.0.0.30 |
Head node, management side |
wwnode1-4.hpc.local |
10.10.10.11–14 |
Compute nodes |
warewulf |
10.10.10.1 |
/etc/hosts alias from the Warewulf hosts overlay |
Reverse zone 10.10.10.in-addr.arpa holds matching PTR records.
9.4 Ports
| Port |
Proto |
Host |
Service |
| 67/68 |
UDP |
head01 |
DHCP (PXE) |
| 69 |
UDP |
head01 |
TFTP (iPXE bootloaders) |
| 9873 |
TCP |
head01 |
warewulfd provisioning HTTP |
| 9880 |
TCP |
control plane |
HPC-on-Demand portal, loopback only |
| 443 |
TCP |
reverse proxy |
https://hpc.example.org → 127.0.0.1:9880 (Caddy, wildcard certificate) |
| 443, 80 |
TCP |
control plane |
hpcod itself, when it serves HTTPS with its own CA (closed networks; 80 redirects and serves /ca.crt) |
| 53 |
TCP/UDP |
control plane |
hpcod's DNS for the portal and *.CLUSTER_DOMAIN, when enabled |
| 9874 |
TCP |
head01 |
warewulfd TLS |
| 6817 |
TCP |
head01 |
slurmctld |
| 6818 |
TCP |
nodes |
slurmd |
| 6819 |
TCP |
head01 |
slurmdbd (accounting) |
| 3306 |
TCP |
head01 |
MariaDB, localhost only |
| 2049 |
TCP |
idm1, head01 |
NFSv4 |
| 53 |
TCP/UDP |
idm1 |
DNS (bind-dyndb-ldap) |
| 88, 464 |
TCP/UDP |
idm1 |
Kerberos KDC, kpasswd |
| 749 |
TCP |
idm1 |
kadmin |
| 389, 636 |
TCP |
idm1 |
LDAP, LDAPS (389-ds) |
| 80, 443 |
TCP |
idm1 |
IPA web UI / CA |
Firewall posture: on both servers eth1 (cluster) is in the firewalld
trusted zone; eth0 (management) is public with only the required services
opened.
9.5 Software versions
| Component |
Version |
| Proxmox VE |
8.4.19 |
| Host / node OS |
Rocky Linux 9.8 (Blue Onyx) |
| Node kernel |
5.14.0-687.41.1.el9_8 |
| Warewulf |
4.7.0-1.el9 |
| Node image source |
ghcr.io/warewulf/warewulf-rockylinux:9 |
| OpenHPC |
3 (EL_9) |
| Slurm |
25.11.4 (slurm-ohpc) |
| OpenMPI |
5.0.7 (openmpi5-gnu14-ohpc) |
| GCC |
14.2.0 (gnu14-compilers-ohpc) |
| Lmod |
9.2 |
| MUNGE |
0.5.13 |
| FreeIPA |
4.13.1-3.el9_8.2 |
| 389-ds-base |
2.8.0-9.el9_8 |
| krb5-server |
1.21.1-10.el9_8 |
| SSSD |
2.9.8-4.el9_8.1 |
9.6 Identity objects
| Object |
ID |
Notes |
| Realm |
HPC.LOCAL |
|
| Domain |
hpc.local |
|
hpcusers (group) |
721800003 |
All cluster users |
admin |
721800000 |
IPA administrator |
hpcuser |
721800005 |
Runs MPI test jobs |
testuser |
721800006 |
Login verification |
Host principals: host/wwnode1.hpc.local … host/wwnode4.hpc.local,
host/head01.hpc.local, host/idm1.hpc.local.
Service principal nfs/idm1.hpc.local exists but is unused (sec=sys today).
Credentials: idm1:/root/.ipa-credentials, mode 0600. Not in this repo.
9.7 Key file locations
head01
| Path |
Contents |
/etc/warewulf/warewulf.conf |
Warewulf global config |
/etc/warewulf/nodes.conf |
Node/profile definitions |
/var/lib/warewulf/overlays/ |
Site overlay sources |
/var/lib/warewulf/provision/images/ |
Served node images |
/etc/slurm/slurm.conf |
Slurm config (mirrored into the hpc overlay) |
/etc/munge/munge.key |
MUNGE key (mirrored into the hpc overlay) |
/root/node-keytabs/ |
Per-node Kerberos keytabs |
/root/.ssh/cluster |
head→node SSH key |
/var/home/ilker |
Local admin home, moved out of /home |
idm1
| Path |
Contents |
/root/.ipa-credentials |
All lab passwords |
/root/cacert.p12 |
CA private key — irreplaceable |
/root/node-keytabs/ |
Generated keytabs |
/export/home/ |
NFS-exported user homes |
/etc/exports |
NFS export definitions |
Compute nodes (in RAM, from overlays)
| Path |
Source overlay |
/etc/krb5.conf, /etc/ipa/ca.crt |
ipa |
/etc/sssd/sssd.conf, /etc/ipa/default.conf |
ipa (templated) |
/etc/krb5.keytab |
ipa.wwnodeN |
/etc/slurm/slurm.conf, /etc/munge/munge.key |
hpc |
/etc/resolv.conf, /etc/fstab, NM connection |
stock Warewulf overlays |
9.8 Repository layout
9.9 Deliberate choices and known gaps
| Choice |
Rationale |
Production alternative |
| SELinux permissive on head and nodes |
Simplifies PXE/NFS/overlay debugging |
enforcing; FreeIPA and Warewulf both support it |
NFS sec=sys |
Kerberised NFS changes the home-provisioning flow |
sec=krb5p (groundwork already laid) |
no_root_squash on /export/home |
Needed by pam_oddjob_mkhomedir |
root_squash + provisioning hook |
| Slurm accounting recorded, not enforced |
A85 added slurmdbd; no association is required to submit (§6.6) |
AccountingStorageEnforce=associations,limits,qos |
| Single IPA server |
Lab |
IPA replica for redundancy |
Single slurmctld |
Lab |
Backup SlurmctldHost |
Nodes onboot=0 |
Deliberate start ordering after idm1/head01 |
onboot=1 with ordering constraints |
| FreeIPA rather than AD |
Samba AD DC unsupported on RHEL 9 |
Samba AD DC on Debian, or an AD trust |
| No backups |
Test environment |
ipa-backup, /export/home snapshots |