Files
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

117 lines
4.6 KiB
Markdown

# 5. Storage and Shared Filesystems
## 5.1 Layout
| Mount | Server | Export options | Client options | Purpose |
|---|---|---|---|---|
| `/home` | `idm1` (10.10.10.5) | `rw,sync,no_subtree_check,no_root_squash` | `defaults,nofail` | IPA user home directories |
| `/opt` | `head01` (10.10.10.1) | `ro,sync,no_root_squash` | `defaults,nofail,ro` | OpenHPC modules (`/opt/ohpc/pub`) |
Both are NFSv4, both exported only to `10.10.10.0/24` — the isolated cluster
fabric. Neither is reachable from `vmbr0`.
The split is deliberate: **identity owns user data, the provisioning server
owns software.** `/opt/ohpc` is built by the OpenHPC packages installed on
head01 and is read-only everywhere else; nothing about it belongs on the
identity server.
## 5.2 Client mounts
On the nodes the mounts come from the Warewulf `fstab` overlay, driven by the
`fstab` resource in `nodes.conf`:
```json
[{"file":"/home","mntops":"defaults,nofail",
"spec":"idm1.hpc.local:/export/home","vfstype":"nfs"},
{"file":"/opt","mntops":"defaults,nofail,ro",
"spec":"warewulf:/opt","vfstype":"nfs"}]
```
Note `/opt` uses the name `warewulf`, an alias for the head node that Warewulf
injects into `/etc/hosts` via the `hosts` overlay. `/home` uses
`idm1.hpc.local`, which requires working DNS — one more reason node DNS must
point at the IPA server.
Originally the default `/opt` entry carried `noauto`, so it never mounted. It
was changed to auto-mount because the OpenHPC modules must be present before a
job starts.
On `head01` the same `/home` mount is a plain `/etc/fstab` line:
```
idm1.hpc.local:/export/home /home nfs4 defaults,_netdev,nofail 0 0
```
## 5.3 Identity mapping
NFSv4 maps users by `user@domain` strings, so `/etc/idmapd.conf` must agree
across every host:
```ini
Domain = hpc.local
```
This is set on `idm1`, on `head01`, and inside the node image. A mismatch makes
files appear as `nobody:nobody` even when uid/gid are otherwise correct.
Because SSSD supplies the same uid/gid everywhere from IPA, ownership stays
consistent: `hpcuser` is 721800005 on the head and on all four nodes.
## 5.4 Security posture
The `/export/home` export uses `sec=sys` and `no_root_squash`. Both are
deliberate lab trade-offs, and both are choices you would revisit in
production.
**`sec=sys`** means the server trusts the uid the client asserts. There is no
cryptographic authentication of NFS traffic. It is acceptable here only because
`vmbr6` is a closed L2 segment with four known hosts.
**`no_root_squash`** lets root on a client write as root on the server. It is
needed so `pam_oddjob_mkhomedir` can create a home directory on first login.
The alternative — pre-creating every home — was rejected because it makes user
onboarding a two-step manual process.
### Path to Kerberised NFS
The groundwork is already in place: every node holds a host keytab, and the
`nfs/idm1.hpc.local` service principal was created during `A40`. Moving to
`sec=krb5p` requires:
1. Fetching a keytab for `nfs/idm1.hpc.local` on the server and enabling
`nfs-server` with `rpc.gssd`/`rpc.svcgssd`
2. Adding `nfs/wwnodeN.hpc.local` service principals, or relying on the host
principal via `rpc.gssd` machine credentials
3. Changing the export to `sec=krb5p` and the client mount options to match
4. Dropping `no_root_squash` — with Kerberos, `mkhomedir` needs a different
approach (IPA automember + a home-provisioning hook, or automount with
`krb5p` and a pre-created skeleton)
Step 4 is the reason this was not done in the first pass: it changes the user
onboarding flow, not just the mount options.
## 5.5 Capacity
| Volume | Backing | Size | Used |
|---|---|---|---|
| `idm1:/export/home` | `zfspool/vm-305-disk-0` | 100 GB | ~3 GB |
| `head01:/opt` | `zfspool/vm-300-disk-0` | 100 GB | ~15 GB (OpenHPC) |
| Node rootfs | RAM (tmpfs) | 8 GB per node | ~1.8 GB |
`zfspool` has roughly 5.4 TB free, so growth is not a near-term concern. The
binding constraint is node RAM, not disk: every gigabyte added to the node
image is a gigabyte taken from each node's usable memory.
## 5.6 Backup considerations
Nothing in this environment is currently backed up. If it mattered:
- **`idm1:/export/home`** — the only irreplaceable user data.
- **FreeIPA** — `ipa-backup` produces a full directory + CA dump. The CA
private key in `/root/cacert.p12` cannot be regenerated; losing it means
re-enrolling every host.
- **`head01:/etc/warewulf/`** — `nodes.conf` and `warewulf.conf` are the whole
cluster definition and are small enough to keep in version control.
- **Node images and overlays** — reproducible from the scripts; not worth
backing up.