Files
ilker 29bd39c135 Warewulf HPC test cluster: provisioning, identity, docs and overlay UI
A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.

Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
  hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
  Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
  server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
  fabric so Kerberos and NFS never traverse head01's masquerade.

Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.

Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.

Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.

Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.

No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
2026-08-29 09:07:51 +02:00

196 lines
6.8 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 4. Identity — FreeIPA and SSSD
## 4.1 FreeIPA is not Active Directory
The original request was for Active Directory. What was built is **FreeIPA**,
chosen deliberately after weighing the options:
| Option | Verdict |
|---|---|
| Samba 4 AD DC on Rocky 9 | Red Hat does **not** support Samba in AD DC mode. Only an experimental MIT-KDC build exists. Not viable. |
| Samba 4 AD DC on Debian | Fully supported AD DC (Heimdal KDC), genuine AD — but a second distro in an otherwise all-Rocky lab. |
| **FreeIPA on Rocky 9** | **Chosen.** The RHEL-native identity server: LDAP + Kerberos + CA + DNS. |
For Linux clients FreeIPA is functionally equivalent to AD: Kerberos realm,
LDAP directory, POSIX attributes, host principals, centralised sudo/HBAC.
**What it is not:** it does not implement the AD schema, does not serve Group
Policy and Windows machines cannot join it natively. If those are needed, the
options are a Samba AD DC or establishing a **trust** between FreeIPA and a
real AD forest — FreeIPA supports the latter, and it is the usual path in mixed
environments.
## 4.2 Server layout
```
Realm HPC.LOCAL
Domain hpc.local
Server FQDN idm1.hpc.local → 10.10.10.5 (cluster fabric)
Management idm1-mgmt.hpc.local → 10.0.0.31
CA Dogtag, self-signed root, valid until 2046
DNS integrated (bind-dyndb-ldap), forwards to 1.1.1.1
Reverse zone 10.10.10.in-addr.arpa
```
Components: `ipa-server 4.13.1`, `389-ds-base 2.8.0`, `krb5-server 1.21.1`,
`bind-dyndb-ldap 11.11`.
Nine services must be running (`ipactl status`): Directory Service, krb5kdc,
kadmin, named, httpd, ipa-custodia, pki-tomcatd, ipa-otpd, ipa-dnskeysyncd.
### DNS records created for the cluster
| Name | Address |
|---|---|
| `idm1` | 10.10.10.5 |
| `idm1-mgmt` | 10.0.0.31 |
| `head01` | 10.10.10.1 |
| `head01-mgmt` | 10.0.0.30 |
| `wwnode1`…`wwnode4` | 10.10.10.11–14 |
Matching PTR records exist in `10.10.10.in-addr.arpa`. Reverse DNS matters:
Kerberos service-ticket validation uses it.
## 4.3 The hard part — enrolling stateless nodes
Compute nodes are diskless; the rootfs is rebuilt in RAM on every boot.
`ipa-client-install` therefore **cannot** be used: it would mint a fresh host
record on every reboot and fill the directory with garbage.
The solution is to enrol once and distribute the result through Warewulf
overlays:
| What | Where | Why |
|---|---|---|
| `krb5.conf`, `ipa/ca.crt` | `ipa` overlay (shared) | Identical on every node |
| `sssd.conf`, `ipa/default.conf` | `ipa` overlay, **Warewulf template** (`.ww`) | `{{.Id}}` gives each node its own FQDN |
| `krb5.keytab` | `ipa.wwnodeN` overlay (per node) | Binary — cannot be templated |
### Why per-node keytabs
A single shared host principal would have been less work. It was rejected
because it would let `wwnode1` impersonate `wwnode2` — any node could request
service tickets as any other. Per-node keytabs preserve the security property
that host identity is actually distinct.
The cost is four extra overlays and a keytab-generation step. In a larger
cluster this would be scripted over the node list.
### The `ipa_hostname` trick
The rendered `sssd.conf` contains:
```ini
ipa_hostname = {{.Id}}.hpc.local → ipa_hostname = wwnode1.hpc.local
```
This lets the node's **system hostname stay short** (`wwnode1`) while SSSD
still finds the right Kerberos principal (`host/wwnode1.hpc.local@HPC.LOCAL`).
That matters because Slurm's `NodeName` values are short — changing the system
hostname to an FQDN would have meant reworking `slurm.conf`.
### Static DNS, no dynamic updates
Nodes run with `dyndns_update = False`. Their DNS records are defined
statically in IPA; there is no reason for a stateless node to rewrite DNS on
every boot, and doing so would generate constant churn.
## 4.4 Rendered node configuration
`/etc/sssd/sssd.conf` on `wwnode1` (mode 0600, or SSSD refuses to start):
```ini
[domain/hpc.local]
id_provider = ipa
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
ipa_domain = hpc.local
ipa_server = idm1.hpc.local
ipa_hostname = wwnode1.hpc.local
ldap_tls_cacert = /etc/ipa/ca.crt
cache_credentials = True
krb5_store_password_if_offline = True
dyndns_update = False
[sssd]
services = nss, pam, ssh, sudo
domains = hpc.local
[nss]
homedir_substring = /home
```
`/etc/krb5.conf` is a hand-written minimal version. Note what is **absent**:
the `pkinit_anchors` and `pkinit_pool` lines that `ipa-client-install` normally
adds. Those point at `/var/lib/ipa-client/pki/…`, which only exists after a
real client install — leaving them in would break Kerberos on the nodes.
`auto_fast_armor = true` is kept and works, because it uses the host keytab.
## 4.5 Users and groups
| Object | ID | Notes |
|---|---|---|
| `hpcusers` (group) | 721800003 | All cluster users |
| `hpcuser` | 721800005 | Runs the MPI test jobs |
| `testuser` | 721800006 | Login verification |
| `admin` | 721800000 | IPA administrator |
### Why there is no IPA user named `ilker`
cloud-init creates a **local** `ilker` (uid 1000) on every VM. `nsswitch.conf`
resolves `files` before `sss`, so an IPA user with the same name would resolve
to uid 1000 on head01 and idm1 but to 721800xxx on the nodes — the same
username with two different uids depending on where you land, which silently
corrupts NFS home ownership.
For the same reason the local `hpcuser` was **deleted** from both head01 and
the node image (`wwctl image syncuser` had copied it there), so the IPA account
is the only one.
The local `ilker` home was moved from `/home/ilker` to `/var/home/ilker`
so that the NFS `/home` mount does not hide it.
### Adding a user
```bash
ssh idm1
sudo bash -c '. /root/.ipa-credentials
echo $IPA_ADMIN_PASSWORD | kinit admin
ipa user-add ayse --first=Ayse --last=Yilmaz \
--homedir=/home/ayse --shell=/bin/bash --password
ipa group-add-member hpcusers --users=ayse'
```
The home directory is created automatically on first login by
`pam_oddjob_mkhomedir`, on the NFS share, on both head and nodes. No
pre-creation is needed — this was verified explicitly.
## 4.6 Credentials
Everything lives in `idm1:/root/.ipa-credentials`, mode 0600: the IPA admin
password, the Directory Manager password, and the generated test-user
passwords. **No password is stored in this repository.**
```bash
ssh idm1
sudo cat /root/.ipa-credentials
sudo bash -c '. /root/.ipa-credentials; echo $IPA_ADMIN_PASSWORD | kinit admin'
sudo ipa user-find
```
## 4.7 Web UI access
The certificate is issued for `idm1.hpc.local`, and that name resolves to
10.10.10.5 — unreachable from the laptop. Use an SSH tunnel:
```bash
ssh -L 8443:10.10.10.5:443 head01
# add "127.0.0.1 idm1.hpc.local" to /etc/hosts, then browse:
# https://idm1.hpc.local:8443
```
Going straight to `https://10.0.0.31` reaches the server but produces a
certificate name mismatch, and IPA's web UI redirects to its canonical
hostname regardless.