ci / derle (push) Failing after 34s
How someone else installs hpcod: the binary, sudo hpcod setup, what it asks and what each step does, the simulated Proxmox, a closed network with its own CA and DNS, verify, selftest, and what to back up. The changelog says what was proven and what was not. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
167 lines
8.6 KiB
Markdown
167 lines
8.6 KiB
Markdown
# Warewulf HPC Test Cluster
|
|
|
|
A working Warewulf 4 HPC lab on Proxmox (`hv01`) with its own isolated subnet,
|
|
central identity (FreeIPA) and shared NFS home directories.
|
|
|
|
**Status: built and verified.** Four diskless nodes PXE-boot from the head
|
|
node, enrol into the IPA domain, mount NFS homes, register `idle` in Slurm, and
|
|
run an 8-rank MPI job across all four as a directory user.
|
|
|
|
```
|
|
Internet
|
|
│
|
|
vmbr0 (10.0.0.0/24) ← management network
|
|
┌─────────────┴─────────────┐
|
|
┌───────────┴──────────┐ ┌───────────┴───────────┐
|
|
│ head01 (VM 300) │ │ idm1 (VM 305) │
|
|
│ eth0 10.0.0.30 │ │ eth0 10.0.0.31 │ ← SSH / admin
|
|
│ Warewulf · DHCP │ │ FreeIPA (hpc.local) │
|
|
│ TFTP/iPXE · Slurm │ │ LDAP · Kerberos │
|
|
│ MUNGE · NAT gateway │ │ CA · DNS │
|
|
│ NFS: /opt (ro) │ │ NFS: /export/home │
|
|
└───────────┬──────────┘ └───────────┬───────────┘
|
|
eth1 10.10.10.1 eth1 10.10.10.5
|
|
└─────────────┬─────────────┘
|
|
│
|
|
vmbr6 ← pure L2, NO IP on Proxmox
|
|
│
|
|
┌───────────┬─────────────┼─────────────┬───────────┐
|
|
wwnode1 wwnode2 wwnode3 wwnode4 (VM 301-304)
|
|
10.10.10.11 10.10.10.12 10.10.10.13 10.10.10.14
|
|
diskless · PXE · rootfs in RAM · IPA-enrolled via SSSD
|
|
/home → idm1:/export/home /opt → head01:/opt (ro)
|
|
```
|
|
|
|
## Documentation
|
|
|
|
A **22-page PDF edition** of the whole handbook: [docs/warewulf-cluster-handbook.pdf](docs/warewulf-cluster-handbook.pdf).
|
|
Regenerate it with `./scripts/build-pdf.sh` after editing `docs/handbook-print.html`.
|
|
|
|
| Document | Contents |
|
|
|---|---|
|
|
| [01 — Architecture](docs/01-architecture.md) | Topology, why `vmbr6` has no IP, why `idm1` is dual-homed, the stateless boot chain |
|
|
| [02 — Installation](docs/02-installation.md) | Building from scratch, what each script does, timings |
|
|
| [03 — Warewulf](docs/03-warewulf.md) | Images, overlays, templates, node management |
|
|
| [04 — Identity](docs/04-identity.md) | FreeIPA vs AD, enrolling stateless nodes, users and groups |
|
|
| [05 — Storage](docs/05-storage.md) | NFS layout, id mapping, security posture, path to Kerberised NFS |
|
|
| [06 — Slurm & MPI](docs/06-slurm-mpi.md) | Scheduler config, running jobs, what is not configured |
|
|
| [07 — Operations](docs/07-operations.md) | Runbook: access, power, health checks, common tasks, logs |
|
|
| [08 — Troubleshooting](docs/08-troubleshooting.md) | The 12 pitfalls hit during the build, plus a diagnostic guide |
|
|
| [09 — Reference](docs/09-reference.md) | Inventory, addressing, ports, versions, file locations |
|
|
| [20 — HPC-on-Demand](docs/20-hpc-on-demand-architecture.md) | Design: multi-site control plane that provisions clusters like this one on demand |
|
|
| [21 — Portal layer](docs/21-portal-layer.md) | What a user gets once the cluster exists: the gap, what to adopt from end-user portals, what not to |
|
|
| [22 — Spack & Chapar](docs/22-spack-chapar-integration.md) | Who compiles the application catalogue: what Chapar gives us, the three levels of integration, and the PMIx question to settle first |
|
|
| [23 — Application lifecycle](docs/23-application-lifecycle.md) | What happens to an application after it is published: the four states, what each transition owes, and the four defects that block them |
|
|
| [hpcod/](hpcod/) | Implementation: multi-provider discovery, placement, and the portal at hpc.example.org |
|
|
| [hpcod/DEPLOY.md](hpcod/DEPLOY.md) | Installing hpcod against your own Proxmox: `sudo hpcod setup`, from the binary alone |
|
|
| [CHANGELOG](CHANGELOG.md) | What is in each release, and what is knowingly not |
|
|
|
|
## Quick start
|
|
|
|
```bash
|
|
ssh head01 # head node
|
|
ssh idm1 # identity + home server
|
|
|
|
sinfo # cluster state
|
|
sudo wwctl node status # provisioning state
|
|
./scripts/A99-verify.sh # full end-to-end health check
|
|
```
|
|
|
|
Compute nodes live on the isolated subnet — reach them through the head node:
|
|
|
|
```bash
|
|
ssh head01
|
|
sudo ssh root@wwnode1
|
|
```
|
|
|
|
HPC-on-Demand portal — **<https://hpc.example.org>** (see [hpcod/](hpcod/)).
|
|
Read-only: what the estate looks like, and where a request would be placed.
|
|
|
|
Power management:
|
|
|
|
```bash
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh status
|
|
ssh proxmox 'bash -s' < scripts/99-cluster-ctl.sh reset # reboot nodes
|
|
```
|
|
|
|
## Software
|
|
|
|
| Component | Version |
|
|
|---|---|
|
|
| Proxmox VE | 8.4.19 |
|
|
| Host / node OS | Rocky Linux 9.8 |
|
|
| Warewulf | 4.7.0 |
|
|
| OpenHPC | 3 (EL_9) |
|
|
| Slurm | 25.11.4 |
|
|
| OpenMPI | 5.0.7 (GCC 14.2) |
|
|
| FreeIPA | 4.13.1 |
|
|
| SSSD | 2.9.8 |
|
|
|
|
## Three decisions worth knowing
|
|
|
|
**`vmbr6` has no IP on Proxmox.** Warewulf needs to own the entire subnet —
|
|
DHCP with PXE options, TFTP, address management. A second authority on the same
|
|
segment would collide. The bridge is a pure L2 switch and `head01` is the
|
|
gateway. See [01-architecture §1.3](docs/01-architecture.md#13-why-vmbr6-has-no-ip-on-the-proxmox-side).
|
|
|
|
**`idm1` is dual-homed, and `idm1.hpc.local` resolves to the cluster side.**
|
|
Behind `head01`'s NAT all four nodes would appear as 10.0.0.30, which breaks
|
|
Kerberos and makes per-node NFS export rules impossible. Dual-homing gives every
|
|
client a direct, symmetric path. See [01-architecture §1.4](docs/01-architecture.md#14-why-idm1-is-dual-homed).
|
|
|
|
**Stateless nodes cannot run `ipa-client-install`.** The rootfs is rebuilt in
|
|
RAM every boot, so enrolment happens once and the result is distributed through
|
|
Warewulf overlays — shared config as templates rendered per node, and a
|
|
**separate keytab per node** so `wwnode1` cannot impersonate `wwnode2`.
|
|
See [04-identity §4.3](docs/04-identity.md#43-the-hard-part--enrolling-stateless-nodes).
|
|
|
|
## Verified output
|
|
|
|
```
|
|
================= IPA SERVER (idm1) =================
|
|
IPA services running: 9
|
|
/export/home 10.10.10.0/24(rw,no_root_squash,sec=sys,...)
|
|
IPA users: 3
|
|
|
|
================= HEAD NODE (head01) =================
|
|
hostname : head01.hpc.local
|
|
DNS : 10.10.10.5
|
|
/home : idm1.hpc.local:/export/home
|
|
hpcuser : uid=721800005(hpcuser) gid=721800005 groups=...,721800003(hpcusers)
|
|
services : warewulfd=active dhcpd=active tftp.socket=active
|
|
slurmctld=active munge=active sssd=active
|
|
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
|
|
normal* up infinite 4 idle wwnode[1-4]
|
|
|
|
================= COMPUTE NODES =================
|
|
wwnode1: sssd=active slurmd=active munge=active home=nfs krb5=OK
|
|
wwnode2: sssd=active slurmd=active munge=active home=nfs krb5=OK
|
|
wwnode3: sssd=active slurmd=active munge=active home=nfs krb5=OK
|
|
wwnode4: sssd=active slurmd=active munge=active home=nfs krb5=OK
|
|
|
|
================= MPI JOB (IPA user, NFS home) =================
|
|
Rank 0 / 8 host=wwnode1 mpi_proc=wwnode1
|
|
Ring complete. start=42, returned=49, expected=49, hops=8 -> SUCCESS
|
|
```
|
|
|
|
`krb5=OK` means the node actually obtained a ticket from the KDC using its own
|
|
host keytab (`kinit -k host/wwnodeN.hpc.local`) — not merely that the file is
|
|
present.
|
|
|
|
Additionally verified by hand: an IPA user SSH'd into a stateless node with a
|
|
password, received a Kerberos TGT and landed in their NFS home; and a brand-new
|
|
IPA user's home was auto-created on the NFS share at first login.
|
|
|
|
## Known gaps
|
|
|
|
- **NFS is `sec=sys`**, not Kerberised. Host keytabs and the
|
|
`nfs/idm1.hpc.local` service principal already exist, so `sec=krb5p` is a
|
|
short step — but it changes the home-provisioning flow.
|
|
- **Slurm accounting is recorded but not enforced** — `sacct` and `sreport` work; `AccountingStorageEnforce` is unset, so no association is required to submit.
|
|
- **SELinux permissive** on head and nodes.
|
|
- **FreeIPA is not Active Directory.** It is the RHEL-native equivalent for
|
|
Linux clients; Windows joins or the AD schema would need a Samba AD DC or an
|
|
AD trust. See [04-identity §4.1](docs/04-identity.md#41-freeipa-is-not-active-directory).
|
|
|
|
Full list with rationale in [09-reference §9.9](docs/09-reference.md#99-deliberate-choices-and-known-gaps).
|