02 now walks an empty host through `hpcod setup`: what it needs, what the TUI asks and does, the simulated Proxmox, reaching the portal, verify and selftest, with timings measured on one run. The hand-built lab's script order moves unchanged to 10-lab-installation.md. README, 01, 09 and the handbook (contents, chapter 2 with two screenshots, new chapter 10) follow; the PDF is regenerated, now 25 pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
233 lines
10 KiB
Markdown
233 lines
10 KiB
Markdown
# 9. Reference
|
||
|
||
## 9.1 Virtual machine inventory
|
||
|
||
| VMID | Name | vCPU | RAM | Disk | Boot | onboot |
|
||
|---|---|---|---|---|---|---|
|
||
| 300 | `head01` | 4 | 8 GB | 100 GB (zfspool) | `scsi0` | yes |
|
||
| 301 | `wwnode1` | 2 | 8 GB | none | `net0` (PXE) | no |
|
||
| 302 | `wwnode2` | 2 | 8 GB | none | `net0` (PXE) | no |
|
||
| 303 | `wwnode3` | 2 | 8 GB | none | `net0` (PXE) | no |
|
||
| 304 | `wwnode4` | 2 | 8 GB | none | `net0` (PXE) | no |
|
||
| 305 | `idm1` | 4 | 8 GB | 100 GB (zfspool) | `scsi0` | yes |
|
||
|
||
Compute nodes additionally carry `balloon: 0` (ballooning would fight the
|
||
tmpfs rootfs), `serial0: socket` and `vga: serial0`.
|
||
|
||
## 9.2 Network addressing
|
||
|
||
| Host | Interface | Bridge | MAC | Address |
|
||
|---|---|---|---|---|
|
||
| `head01` | eth0 | vmbr0 | `BC:24:11:E0:00:FF` | 10.0.0.30/24 |
|
||
| `head01` | eth1 | vmbr6 | `BC:24:11:E0:0A:01` | 10.10.10.1/24 |
|
||
| `idm1` | eth0 | vmbr0 | `BC:24:11:E0:00:31` | 10.0.0.31/24 |
|
||
| `idm1` | eth1 | vmbr6 | `BC:24:11:E0:0A:05` | 10.10.10.5/24 |
|
||
| `wwnode1` | eth0 | vmbr6 | `BC:24:11:E0:0A:11` | 10.10.10.11/24 |
|
||
| `wwnode2` | eth0 | vmbr6 | `BC:24:11:E0:0A:12` | 10.10.10.12/24 |
|
||
| `wwnode3` | eth0 | vmbr6 | `BC:24:11:E0:0A:13` | 10.10.10.13/24 |
|
||
| `wwnode4` | eth0 | vmbr6 | `BC:24:11:E0:0A:14` | 10.10.10.14/24 |
|
||
|
||
MAC convention: `BC:24:11:E0:xx:yy` — `00:` prefix for management NICs,
|
||
`0A:` for cluster NICs, node index in the last octet.
|
||
|
||
**DHCP pool:** 10.10.10.100–199, transient PXE leases only.
|
||
**Gateway for 10.10.10.0/24:** 10.10.10.1 (`head01`), NAT out via `vmbr0`.
|
||
|
||
### Bridges
|
||
|
||
| Bridge | Address | Role |
|
||
|---|---|---|
|
||
| `vmbr0` | 10.0.0.10/24 (Proxmox) | Management, pre-existing |
|
||
| `vmbr6` | none | Cluster fabric, L2 only, created for this build |
|
||
|
||
## 9.3 DNS names
|
||
|
||
| Name | Address | Notes |
|
||
|---|---|---|
|
||
| `idm1.hpc.local` | 10.10.10.5 | IPA/NFS **service** address |
|
||
| `idm1-mgmt.hpc.local` | 10.0.0.31 | Management address |
|
||
| `head01.hpc.local` | 10.10.10.1 | Head node, cluster side |
|
||
| `head01-mgmt.hpc.local` | 10.0.0.30 | Head node, management side |
|
||
| `wwnode1-4.hpc.local` | 10.10.10.11–14 | Compute nodes |
|
||
| `warewulf` | 10.10.10.1 | `/etc/hosts` alias from the Warewulf `hosts` overlay |
|
||
|
||
Reverse zone `10.10.10.in-addr.arpa` holds matching PTR records.
|
||
|
||
## 9.4 Ports
|
||
|
||
| Port | Proto | Host | Service |
|
||
|---|---|---|---|
|
||
| 67/68 | UDP | head01 | DHCP (PXE) |
|
||
| 69 | UDP | head01 | TFTP (iPXE bootloaders) |
|
||
| 9873 | TCP | head01 | `warewulfd` provisioning HTTP |
|
||
| 9880 | TCP | control plane | HPC-on-Demand portal, loopback only |
|
||
| 443 | TCP | reverse proxy | `https://hpc.example.org` → 127.0.0.1:9880 (Caddy, wildcard certificate) |
|
||
| 443, 80 | TCP | control plane | hpcod itself, when it serves HTTPS with its own CA (closed networks; 80 redirects and serves `/ca.crt`) |
|
||
| 53 | TCP/UDP | control plane | hpcod's DNS for the portal and `*.CLUSTER_DOMAIN`, when enabled |
|
||
| 9874 | TCP | head01 | `warewulfd` TLS |
|
||
| 6817 | TCP | head01 | `slurmctld` |
|
||
| 6818 | TCP | nodes | `slurmd` |
|
||
| 6819 | TCP | head01 | `slurmdbd` (accounting) |
|
||
| 3306 | TCP | head01 | MariaDB, localhost only |
|
||
| 2049 | TCP | idm1, head01 | NFSv4 |
|
||
| 53 | TCP/UDP | idm1 | DNS (bind-dyndb-ldap) |
|
||
| 88, 464 | TCP/UDP | idm1 | Kerberos KDC, kpasswd |
|
||
| 749 | TCP | idm1 | kadmin |
|
||
| 389, 636 | TCP | idm1 | LDAP, LDAPS (389-ds) |
|
||
| 80, 443 | TCP | idm1 | IPA web UI / CA |
|
||
|
||
Firewall posture: on both servers `eth1` (cluster) is in the firewalld
|
||
`trusted` zone; `eth0` (management) is `public` with only the required services
|
||
opened.
|
||
|
||
## 9.5 Software versions
|
||
|
||
| Component | Version |
|
||
|---|---|
|
||
| Proxmox VE | 8.4.19 |
|
||
| Host / node OS | Rocky Linux 9.8 (Blue Onyx) |
|
||
| Node kernel | 5.14.0-687.41.1.el9_8 |
|
||
| Warewulf | 4.7.0-1.el9 |
|
||
| Node image source | `ghcr.io/warewulf/warewulf-rockylinux:9` |
|
||
| OpenHPC | 3 (EL_9) |
|
||
| Slurm | 25.11.4 (`slurm-ohpc`) |
|
||
| OpenMPI | 5.0.7 (`openmpi5-gnu14-ohpc`) |
|
||
| GCC | 14.2.0 (`gnu14-compilers-ohpc`) |
|
||
| Lmod | 9.2 |
|
||
| MUNGE | 0.5.13 |
|
||
| FreeIPA | 4.13.1-3.el9_8.2 |
|
||
| 389-ds-base | 2.8.0-9.el9_8 |
|
||
| krb5-server | 1.21.1-10.el9_8 |
|
||
| SSSD | 2.9.8-4.el9_8.1 |
|
||
|
||
## 9.6 Identity objects
|
||
|
||
| Object | ID | Notes |
|
||
|---|---|---|
|
||
| Realm | `HPC.LOCAL` | |
|
||
| Domain | `hpc.local` | |
|
||
| `hpcusers` (group) | 721800003 | All cluster users |
|
||
| `admin` | 721800000 | IPA administrator |
|
||
| `hpcuser` | 721800005 | Runs MPI test jobs |
|
||
| `testuser` | 721800006 | Login verification |
|
||
|
||
Host principals: `host/wwnode1.hpc.local` … `host/wwnode4.hpc.local`,
|
||
`host/head01.hpc.local`, `host/idm1.hpc.local`.
|
||
Service principal `nfs/idm1.hpc.local` exists but is unused (`sec=sys` today).
|
||
|
||
**Credentials:** `idm1:/root/.ipa-credentials`, mode 0600. Not in this repo.
|
||
|
||
## 9.7 Key file locations
|
||
|
||
### head01
|
||
|
||
| Path | Contents |
|
||
|---|---|
|
||
| `/etc/warewulf/warewulf.conf` | Warewulf global config |
|
||
| `/etc/warewulf/nodes.conf` | Node/profile definitions |
|
||
| `/var/lib/warewulf/overlays/` | Site overlay sources |
|
||
| `/var/lib/warewulf/provision/images/` | Served node images |
|
||
| `/etc/slurm/slurm.conf` | Slurm config (mirrored into the `hpc` overlay) |
|
||
| `/etc/munge/munge.key` | MUNGE key (mirrored into the `hpc` overlay) |
|
||
| `/root/node-keytabs/` | Per-node Kerberos keytabs |
|
||
| `/root/.ssh/cluster` | head→node SSH key |
|
||
| `/var/home/ilker` | Local admin home, moved out of `/home` |
|
||
|
||
### idm1
|
||
|
||
| Path | Contents |
|
||
|---|---|
|
||
| `/root/.ipa-credentials` | All lab passwords |
|
||
| `/root/cacert.p12` | CA private key — irreplaceable |
|
||
| `/root/node-keytabs/` | Generated keytabs |
|
||
| `/export/home/` | NFS-exported user homes |
|
||
| `/etc/exports` | NFS export definitions |
|
||
|
||
### Compute nodes (in RAM, from overlays)
|
||
|
||
| Path | Source overlay |
|
||
|---|---|
|
||
| `/etc/krb5.conf`, `/etc/ipa/ca.crt` | `ipa` |
|
||
| `/etc/sssd/sssd.conf`, `/etc/ipa/default.conf` | `ipa` (templated) |
|
||
| `/etc/krb5.keytab` | `ipa.wwnodeN` |
|
||
| `/etc/slurm/slurm.conf`, `/etc/munge/munge.key` | `hpc` |
|
||
| `/etc/resolv.conf`, `/etc/fstab`, NM connection | stock Warewulf overlays |
|
||
|
||
## 9.8 Repository layout
|
||
|
||
```
|
||
werewulf/
|
||
├── README.md index and quickstart
|
||
├── CHANGELOG.md releases
|
||
├── docs/
|
||
│ ├── 01-architecture.md the reference lab: topology and rationale
|
||
│ ├── 02-installation.md installing hpcod from scratch
|
||
│ ├── 03-warewulf.md images, overlays, node management
|
||
│ ├── 04-identity.md FreeIPA, SSSD, stateless enrolment
|
||
│ ├── 05-storage.md NFS layout and security posture
|
||
│ ├── 06-slurm-mpi.md scheduler and MPI
|
||
│ ├── 07-operations.md runbook
|
||
│ ├── 08-troubleshooting.md 12 pitfalls + diagnostics
|
||
│ ├── 09-reference.md this file
|
||
│ ├── 10-lab-installation.md building the lab by hand
|
||
│ ├── 20-hpc-on-demand-architecture.md the control plane
|
||
│ ├── 21-portal-layer.md per-cluster Open OnDemand
|
||
│ ├── 22-spack-chapar-integration.md building application images
|
||
│ ├── 23-application-lifecycle.md catalogue states and retention
|
||
│ ├── handbook-print.html print edition source (fonts embedded)
|
||
│ └── warewulf-cluster-handbook.pdf PDF edition of 01–10
|
||
├── hpcod/ the HPC-on-Demand control plane
|
||
│ ├── DEPLOY.md installing it: `hpcod setup`
|
||
│ ├── cmd/hpcod/ the one binary: serve, setup, verify, selftest, apps …
|
||
│ ├── internal/ placement, store, provisioning, portal, setup TUI
|
||
│ ├── ansible/ head node roles, embedded in the binary
|
||
│ ├── terraform/ the per-cluster module, embedded
|
||
│ ├── db/ schema migrations, embedded
|
||
│ ├── config/apps.yaml application catalogue, embedded
|
||
│ ├── image/ node, desktop and application images
|
||
│ └── deploy/ site.env.example, Caddyfile.example
|
||
├── scripts/
|
||
│ ├── 00-proxmox-provision.sh lab: cluster tier
|
||
│ ├── 10-head-base.sh
|
||
│ ├── 20-warewulf-install.sh
|
||
│ ├── 30-warewulf-configure.sh
|
||
│ ├── 40-node-image.sh
|
||
│ ├── 50-nodes-register.sh
|
||
│ ├── 60-slurm-openhpc.sh
|
||
│ ├── 70-mpi-test.sh
|
||
│ ├── A00-proxmox-ipa-vm.sh lab: identity tier
|
||
│ ├── A10-ipa-server.sh
|
||
│ ├── A20-ipa-nfs-home.sh
|
||
│ ├── A30-head01-ipa.sh
|
||
│ ├── A40-ipa-node-hosts.sh
|
||
│ ├── A50-node-image-ipa.sh
|
||
│ ├── A60-node-overlays-ipa.sh
|
||
│ ├── A85-slurm-accounting.sh slurmdbd + MariaDB accounting
|
||
│ ├── A99-verify.sh verification
|
||
│ ├── 99-cluster-ctl.sh power management
|
||
│ ├── B30-cluster-inventory.py hpcod: Ansible inventory from tofu output
|
||
│ ├── B40-publish-cluster.py hpcod: publish a cluster portal by hand
|
||
│ ├── C10-publish-release.sh release attachments
|
||
│ ├── build-pdf.sh regenerate the PDF edition
|
||
│ └── run-all.sh orchestrates the cluster tier
|
||
└── mpi/
|
||
├── hello_mpi.c ring test program
|
||
├── heat_mpi.c long-running benchmark
|
||
├── mpi-ring.sbatch Slurm batch job
|
||
└── heat-long.sbatch long-running batch job
|
||
```
|
||
|
||
## 9.9 Deliberate choices and known gaps
|
||
|
||
| Choice | Rationale | Production alternative |
|
||
|---|---|---|
|
||
| SELinux permissive on head and nodes | Simplifies PXE/NFS/overlay debugging | `enforcing`; FreeIPA and Warewulf both support it |
|
||
| NFS `sec=sys` | Kerberised NFS changes the home-provisioning flow | `sec=krb5p` (groundwork already laid) |
|
||
| `no_root_squash` on `/export/home` | Needed by `pam_oddjob_mkhomedir` | `root_squash` + provisioning hook |
|
||
| Slurm accounting recorded, not enforced | `A85` added `slurmdbd`; no association is required to submit (§6.6) | `AccountingStorageEnforce=associations,limits,qos` |
|
||
| Single IPA server | Lab | IPA replica for redundancy |
|
||
| Single `slurmctld` | Lab | Backup `SlurmctldHost` |
|
||
| Nodes `onboot=0` | Deliberate start ordering after idm1/head01 | `onboot=1` with ordering constraints |
|
||
| FreeIPA rather than AD | Samba AD DC unsupported on RHEL 9 | Samba AD DC on Debian, or an AD trust |
|
||
| No backups | Test environment | `ipa-backup`, `/export/home` snapshots |
|