Files
werewulf/docs/09-reference.md
ilkerandClaude Opus 5.5 5b6e1ed3dd docs: section 02 installs hpcod from scratch; the lab build moves to 10
02 now walks an empty host through `hpcod setup`: what it needs, what the
TUI asks and does, the simulated Proxmox, reaching the portal, verify and
selftest, with timings measured on one run. The hand-built lab's script
order moves unchanged to 10-lab-installation.md. README, 01, 09 and the
handbook (contents, chapter 2 with two screenshots, new chapter 10) follow;
the PDF is regenerated, now 25 pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:08:32 +01:00

233 lines
10 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 9. Reference
## 9.1 Virtual machine inventory
| VMID | Name | vCPU | RAM | Disk | Boot | onboot |
|---|---|---|---|---|---|---|
| 300 | `head01` | 4 | 8 GB | 100 GB (zfspool) | `scsi0` | yes |
| 301 | `wwnode1` | 2 | 8 GB | none | `net0` (PXE) | no |
| 302 | `wwnode2` | 2 | 8 GB | none | `net0` (PXE) | no |
| 303 | `wwnode3` | 2 | 8 GB | none | `net0` (PXE) | no |
| 304 | `wwnode4` | 2 | 8 GB | none | `net0` (PXE) | no |
| 305 | `idm1` | 4 | 8 GB | 100 GB (zfspool) | `scsi0` | yes |
Compute nodes additionally carry `balloon: 0` (ballooning would fight the
tmpfs rootfs), `serial0: socket` and `vga: serial0`.
## 9.2 Network addressing
| Host | Interface | Bridge | MAC | Address |
|---|---|---|---|---|
| `head01` | eth0 | vmbr0 | `BC:24:11:E0:00:FF` | 10.0.0.30/24 |
| `head01` | eth1 | vmbr6 | `BC:24:11:E0:0A:01` | 10.10.10.1/24 |
| `idm1` | eth0 | vmbr0 | `BC:24:11:E0:00:31` | 10.0.0.31/24 |
| `idm1` | eth1 | vmbr6 | `BC:24:11:E0:0A:05` | 10.10.10.5/24 |
| `wwnode1` | eth0 | vmbr6 | `BC:24:11:E0:0A:11` | 10.10.10.11/24 |
| `wwnode2` | eth0 | vmbr6 | `BC:24:11:E0:0A:12` | 10.10.10.12/24 |
| `wwnode3` | eth0 | vmbr6 | `BC:24:11:E0:0A:13` | 10.10.10.13/24 |
| `wwnode4` | eth0 | vmbr6 | `BC:24:11:E0:0A:14` | 10.10.10.14/24 |
MAC convention: `BC:24:11:E0:xx:yy` — `00:` prefix for management NICs,
`0A:` for cluster NICs, node index in the last octet.
**DHCP pool:** 10.10.10.100–199, transient PXE leases only.
**Gateway for 10.10.10.0/24:** 10.10.10.1 (`head01`), NAT out via `vmbr0`.
### Bridges
| Bridge | Address | Role |
|---|---|---|
| `vmbr0` | 10.0.0.10/24 (Proxmox) | Management, pre-existing |
| `vmbr6` | none | Cluster fabric, L2 only, created for this build |
## 9.3 DNS names
| Name | Address | Notes |
|---|---|---|
| `idm1.hpc.local` | 10.10.10.5 | IPA/NFS **service** address |
| `idm1-mgmt.hpc.local` | 10.0.0.31 | Management address |
| `head01.hpc.local` | 10.10.10.1 | Head node, cluster side |
| `head01-mgmt.hpc.local` | 10.0.0.30 | Head node, management side |
| `wwnode1-4.hpc.local` | 10.10.10.11–14 | Compute nodes |
| `warewulf` | 10.10.10.1 | `/etc/hosts` alias from the Warewulf `hosts` overlay |
Reverse zone `10.10.10.in-addr.arpa` holds matching PTR records.
## 9.4 Ports
| Port | Proto | Host | Service |
|---|---|---|---|
| 67/68 | UDP | head01 | DHCP (PXE) |
| 69 | UDP | head01 | TFTP (iPXE bootloaders) |
| 9873 | TCP | head01 | `warewulfd` provisioning HTTP |
| 9880 | TCP | control plane | HPC-on-Demand portal, loopback only |
| 443 | TCP | reverse proxy | `https://hpc.example.org` → 127.0.0.1:9880 (Caddy, wildcard certificate) |
| 443, 80 | TCP | control plane | hpcod itself, when it serves HTTPS with its own CA (closed networks; 80 redirects and serves `/ca.crt`) |
| 53 | TCP/UDP | control plane | hpcod's DNS for the portal and `*.CLUSTER_DOMAIN`, when enabled |
| 9874 | TCP | head01 | `warewulfd` TLS |
| 6817 | TCP | head01 | `slurmctld` |
| 6818 | TCP | nodes | `slurmd` |
| 6819 | TCP | head01 | `slurmdbd` (accounting) |
| 3306 | TCP | head01 | MariaDB, localhost only |
| 2049 | TCP | idm1, head01 | NFSv4 |
| 53 | TCP/UDP | idm1 | DNS (bind-dyndb-ldap) |
| 88, 464 | TCP/UDP | idm1 | Kerberos KDC, kpasswd |
| 749 | TCP | idm1 | kadmin |
| 389, 636 | TCP | idm1 | LDAP, LDAPS (389-ds) |
| 80, 443 | TCP | idm1 | IPA web UI / CA |
Firewall posture: on both servers `eth1` (cluster) is in the firewalld
`trusted` zone; `eth0` (management) is `public` with only the required services
opened.
## 9.5 Software versions
| Component | Version |
|---|---|
| Proxmox VE | 8.4.19 |
| Host / node OS | Rocky Linux 9.8 (Blue Onyx) |
| Node kernel | 5.14.0-687.41.1.el9_8 |
| Warewulf | 4.7.0-1.el9 |
| Node image source | `ghcr.io/warewulf/warewulf-rockylinux:9` |
| OpenHPC | 3 (EL_9) |
| Slurm | 25.11.4 (`slurm-ohpc`) |
| OpenMPI | 5.0.7 (`openmpi5-gnu14-ohpc`) |
| GCC | 14.2.0 (`gnu14-compilers-ohpc`) |
| Lmod | 9.2 |
| MUNGE | 0.5.13 |
| FreeIPA | 4.13.1-3.el9_8.2 |
| 389-ds-base | 2.8.0-9.el9_8 |
| krb5-server | 1.21.1-10.el9_8 |
| SSSD | 2.9.8-4.el9_8.1 |
## 9.6 Identity objects
| Object | ID | Notes |
|---|---|---|
| Realm | `HPC.LOCAL` | |
| Domain | `hpc.local` | |
| `hpcusers` (group) | 721800003 | All cluster users |
| `admin` | 721800000 | IPA administrator |
| `hpcuser` | 721800005 | Runs MPI test jobs |
| `testuser` | 721800006 | Login verification |
Host principals: `host/wwnode1.hpc.local` … `host/wwnode4.hpc.local`,
`host/head01.hpc.local`, `host/idm1.hpc.local`.
Service principal `nfs/idm1.hpc.local` exists but is unused (`sec=sys` today).
**Credentials:** `idm1:/root/.ipa-credentials`, mode 0600. Not in this repo.
## 9.7 Key file locations
### head01
| Path | Contents |
|---|---|
| `/etc/warewulf/warewulf.conf` | Warewulf global config |
| `/etc/warewulf/nodes.conf` | Node/profile definitions |
| `/var/lib/warewulf/overlays/` | Site overlay sources |
| `/var/lib/warewulf/provision/images/` | Served node images |
| `/etc/slurm/slurm.conf` | Slurm config (mirrored into the `hpc` overlay) |
| `/etc/munge/munge.key` | MUNGE key (mirrored into the `hpc` overlay) |
| `/root/node-keytabs/` | Per-node Kerberos keytabs |
| `/root/.ssh/cluster` | head→node SSH key |
| `/var/home/ilker` | Local admin home, moved out of `/home` |
### idm1
| Path | Contents |
|---|---|
| `/root/.ipa-credentials` | All lab passwords |
| `/root/cacert.p12` | CA private key — irreplaceable |
| `/root/node-keytabs/` | Generated keytabs |
| `/export/home/` | NFS-exported user homes |
| `/etc/exports` | NFS export definitions |
### Compute nodes (in RAM, from overlays)
| Path | Source overlay |
|---|---|
| `/etc/krb5.conf`, `/etc/ipa/ca.crt` | `ipa` |
| `/etc/sssd/sssd.conf`, `/etc/ipa/default.conf` | `ipa` (templated) |
| `/etc/krb5.keytab` | `ipa.wwnodeN` |
| `/etc/slurm/slurm.conf`, `/etc/munge/munge.key` | `hpc` |
| `/etc/resolv.conf`, `/etc/fstab`, NM connection | stock Warewulf overlays |
## 9.8 Repository layout
```
werewulf/
├── README.md index and quickstart
├── CHANGELOG.md releases
├── docs/
│ ├── 01-architecture.md the reference lab: topology and rationale
│ ├── 02-installation.md installing hpcod from scratch
│ ├── 03-warewulf.md images, overlays, node management
│ ├── 04-identity.md FreeIPA, SSSD, stateless enrolment
│ ├── 05-storage.md NFS layout and security posture
│ ├── 06-slurm-mpi.md scheduler and MPI
│ ├── 07-operations.md runbook
│ ├── 08-troubleshooting.md 12 pitfalls + diagnostics
│ ├── 09-reference.md this file
│ ├── 10-lab-installation.md building the lab by hand
│ ├── 20-hpc-on-demand-architecture.md the control plane
│ ├── 21-portal-layer.md per-cluster Open OnDemand
│ ├── 22-spack-chapar-integration.md building application images
│ ├── 23-application-lifecycle.md catalogue states and retention
│ ├── handbook-print.html print edition source (fonts embedded)
│ └── warewulf-cluster-handbook.pdf PDF edition of 01–10
├── hpcod/ the HPC-on-Demand control plane
│ ├── DEPLOY.md installing it: `hpcod setup`
│ ├── cmd/hpcod/ the one binary: serve, setup, verify, selftest, apps …
│ ├── internal/ placement, store, provisioning, portal, setup TUI
│ ├── ansible/ head node roles, embedded in the binary
│ ├── terraform/ the per-cluster module, embedded
│ ├── db/ schema migrations, embedded
│ ├── config/apps.yaml application catalogue, embedded
│ ├── image/ node, desktop and application images
│ └── deploy/ site.env.example, Caddyfile.example
├── scripts/
│ ├── 00-proxmox-provision.sh lab: cluster tier
│ ├── 10-head-base.sh
│ ├── 20-warewulf-install.sh
│ ├── 30-warewulf-configure.sh
│ ├── 40-node-image.sh
│ ├── 50-nodes-register.sh
│ ├── 60-slurm-openhpc.sh
│ ├── 70-mpi-test.sh
│ ├── A00-proxmox-ipa-vm.sh lab: identity tier
│ ├── A10-ipa-server.sh
│ ├── A20-ipa-nfs-home.sh
│ ├── A30-head01-ipa.sh
│ ├── A40-ipa-node-hosts.sh
│ ├── A50-node-image-ipa.sh
│ ├── A60-node-overlays-ipa.sh
│ ├── A85-slurm-accounting.sh slurmdbd + MariaDB accounting
│ ├── A99-verify.sh verification
│ ├── 99-cluster-ctl.sh power management
│ ├── B30-cluster-inventory.py hpcod: Ansible inventory from tofu output
│ ├── B40-publish-cluster.py hpcod: publish a cluster portal by hand
│ ├── C10-publish-release.sh release attachments
│ ├── build-pdf.sh regenerate the PDF edition
│ └── run-all.sh orchestrates the cluster tier
└── mpi/
├── hello_mpi.c ring test program
├── heat_mpi.c long-running benchmark
├── mpi-ring.sbatch Slurm batch job
└── heat-long.sbatch long-running batch job
```
## 9.9 Deliberate choices and known gaps
| Choice | Rationale | Production alternative |
|---|---|---|
| SELinux permissive on head and nodes | Simplifies PXE/NFS/overlay debugging | `enforcing`; FreeIPA and Warewulf both support it |
| NFS `sec=sys` | Kerberised NFS changes the home-provisioning flow | `sec=krb5p` (groundwork already laid) |
| `no_root_squash` on `/export/home` | Needed by `pam_oddjob_mkhomedir` | `root_squash` + provisioning hook |
| Slurm accounting recorded, not enforced | `A85` added `slurmdbd`; no association is required to submit (§6.6) | `AccountingStorageEnforce=associations,limits,qos` |
| Single IPA server | Lab | IPA replica for redundancy |
| Single `slurmctld` | Lab | Backup `SlurmctldHost` |
| Nodes `onboot=0` | Deliberate start ordering after idm1/head01 | `onboot=1` with ordering constraints |
| FreeIPA rather than AD | Samba AD DC unsupported on RHEL 9 | Samba AD DC on Debian, or an AD trust |
| No backups | Test environment | `ipa-backup`, `/export/home` snapshots |