A complete, verified HPC lab on Proxmox (hv01) with its own isolated subnet,
central identity and a web console for overlay management.
Infrastructure
- vmbr6: L2-only cluster fabric, deliberately without an address on the
hypervisor so Warewulf owns DHCP/TFTP/PXE for the whole subnet.
- head01 (VM 300): Warewulf 4.7.0, dnsmasq-free DHCP/TFTP/iPXE chain,
Slurm controller, MUNGE, NAT gateway, /opt exported read-only.
- wwnode1-4 (VM 301-304): diskless, PXE-booted, rootfs unpacked into RAM.
- idm1 (VM 305): FreeIPA (LDAP/Kerberos/Dogtag CA/DNS) plus the NFS home
server. Dual-homed on purpose: idm1.hpc.local resolves to the cluster
fabric so Kerberos and NFS never traverse head01's masquerade.
Stack
Rocky Linux 9.8, OpenHPC 3, Slurm 25.11.4, OpenMPI 5.0.7, FreeIPA 4.13.1.
Stateless domain join
Nodes are rebuilt in RAM every boot, so ipa-client-install cannot be used.
Enrolment happens once and is distributed through Warewulf overlays: shared
config as templates rendered per node via {{.Id}}, and a separate keytab per
node so wwnode1 cannot impersonate wwnode2.
Overlay web UI (webui/)
A Go service on the head node serving a browser interface over the Warewulf
REST API. It exists as a server rather than a static page because the API is
root-equivalent and bound to loopback; the service holds the credentials and
proxies same-origin, so the browser never sees one. Published behind Caddy
with bcrypt logins, server-side sessions and per-user/per-IP lockout.
Documentation
README plus docs/01-10 covering architecture, installation, Warewulf,
identity, storage, Slurm/MPI, operations, troubleshooting and reference,
with a 22-page PDF edition. Section 08 records twelve concrete failures hit
during the build, including a wwctl flag that fails silently.
No credentials are stored here. Every secret lives on the servers; the
documentation refers only to its path.
196 lines
6.8 KiB
Markdown
196 lines
6.8 KiB
Markdown
# 4. Identity — FreeIPA and SSSD
|
||
|
||
## 4.1 FreeIPA is not Active Directory
|
||
|
||
The original request was for Active Directory. What was built is **FreeIPA**,
|
||
chosen deliberately after weighing the options:
|
||
|
||
| Option | Verdict |
|
||
|---|---|
|
||
| Samba 4 AD DC on Rocky 9 | Red Hat does **not** support Samba in AD DC mode. Only an experimental MIT-KDC build exists. Not viable. |
|
||
| Samba 4 AD DC on Debian | Fully supported AD DC (Heimdal KDC), genuine AD — but a second distro in an otherwise all-Rocky lab. |
|
||
| **FreeIPA on Rocky 9** | **Chosen.** The RHEL-native identity server: LDAP + Kerberos + CA + DNS. |
|
||
|
||
For Linux clients FreeIPA is functionally equivalent to AD: Kerberos realm,
|
||
LDAP directory, POSIX attributes, host principals, centralised sudo/HBAC.
|
||
|
||
**What it is not:** it does not implement the AD schema, does not serve Group
|
||
Policy and Windows machines cannot join it natively. If those are needed, the
|
||
options are a Samba AD DC or establishing a **trust** between FreeIPA and a
|
||
real AD forest — FreeIPA supports the latter, and it is the usual path in mixed
|
||
environments.
|
||
|
||
## 4.2 Server layout
|
||
|
||
```
|
||
Realm HPC.LOCAL
|
||
Domain hpc.local
|
||
Server FQDN idm1.hpc.local → 10.10.10.5 (cluster fabric)
|
||
Management idm1-mgmt.hpc.local → 10.0.0.31
|
||
CA Dogtag, self-signed root, valid until 2046
|
||
DNS integrated (bind-dyndb-ldap), forwards to 1.1.1.1
|
||
Reverse zone 10.10.10.in-addr.arpa
|
||
```
|
||
|
||
Components: `ipa-server 4.13.1`, `389-ds-base 2.8.0`, `krb5-server 1.21.1`,
|
||
`bind-dyndb-ldap 11.11`.
|
||
|
||
Nine services must be running (`ipactl status`): Directory Service, krb5kdc,
|
||
kadmin, named, httpd, ipa-custodia, pki-tomcatd, ipa-otpd, ipa-dnskeysyncd.
|
||
|
||
### DNS records created for the cluster
|
||
|
||
| Name | Address |
|
||
|---|---|
|
||
| `idm1` | 10.10.10.5 |
|
||
| `idm1-mgmt` | 10.0.0.31 |
|
||
| `head01` | 10.10.10.1 |
|
||
| `head01-mgmt` | 10.0.0.30 |
|
||
| `wwnode1`…`wwnode4` | 10.10.10.11–14 |
|
||
|
||
Matching PTR records exist in `10.10.10.in-addr.arpa`. Reverse DNS matters:
|
||
Kerberos service-ticket validation uses it.
|
||
|
||
## 4.3 The hard part — enrolling stateless nodes
|
||
|
||
Compute nodes are diskless; the rootfs is rebuilt in RAM on every boot.
|
||
`ipa-client-install` therefore **cannot** be used: it would mint a fresh host
|
||
record on every reboot and fill the directory with garbage.
|
||
|
||
The solution is to enrol once and distribute the result through Warewulf
|
||
overlays:
|
||
|
||
| What | Where | Why |
|
||
|---|---|---|
|
||
| `krb5.conf`, `ipa/ca.crt` | `ipa` overlay (shared) | Identical on every node |
|
||
| `sssd.conf`, `ipa/default.conf` | `ipa` overlay, **Warewulf template** (`.ww`) | `{{.Id}}` gives each node its own FQDN |
|
||
| `krb5.keytab` | `ipa.wwnodeN` overlay (per node) | Binary — cannot be templated |
|
||
|
||
### Why per-node keytabs
|
||
|
||
A single shared host principal would have been less work. It was rejected
|
||
because it would let `wwnode1` impersonate `wwnode2` — any node could request
|
||
service tickets as any other. Per-node keytabs preserve the security property
|
||
that host identity is actually distinct.
|
||
|
||
The cost is four extra overlays and a keytab-generation step. In a larger
|
||
cluster this would be scripted over the node list.
|
||
|
||
### The `ipa_hostname` trick
|
||
|
||
The rendered `sssd.conf` contains:
|
||
|
||
```ini
|
||
ipa_hostname = {{.Id}}.hpc.local → ipa_hostname = wwnode1.hpc.local
|
||
```
|
||
|
||
This lets the node's **system hostname stay short** (`wwnode1`) while SSSD
|
||
still finds the right Kerberos principal (`host/wwnode1.hpc.local@HPC.LOCAL`).
|
||
That matters because Slurm's `NodeName` values are short — changing the system
|
||
hostname to an FQDN would have meant reworking `slurm.conf`.
|
||
|
||
### Static DNS, no dynamic updates
|
||
|
||
Nodes run with `dyndns_update = False`. Their DNS records are defined
|
||
statically in IPA; there is no reason for a stateless node to rewrite DNS on
|
||
every boot, and doing so would generate constant churn.
|
||
|
||
## 4.4 Rendered node configuration
|
||
|
||
`/etc/sssd/sssd.conf` on `wwnode1` (mode 0600, or SSSD refuses to start):
|
||
|
||
```ini
|
||
[domain/hpc.local]
|
||
id_provider = ipa
|
||
auth_provider = ipa
|
||
chpass_provider = ipa
|
||
access_provider = ipa
|
||
ipa_domain = hpc.local
|
||
ipa_server = idm1.hpc.local
|
||
ipa_hostname = wwnode1.hpc.local
|
||
ldap_tls_cacert = /etc/ipa/ca.crt
|
||
cache_credentials = True
|
||
krb5_store_password_if_offline = True
|
||
dyndns_update = False
|
||
|
||
[sssd]
|
||
services = nss, pam, ssh, sudo
|
||
domains = hpc.local
|
||
|
||
[nss]
|
||
homedir_substring = /home
|
||
```
|
||
|
||
`/etc/krb5.conf` is a hand-written minimal version. Note what is **absent**:
|
||
the `pkinit_anchors` and `pkinit_pool` lines that `ipa-client-install` normally
|
||
adds. Those point at `/var/lib/ipa-client/pki/…`, which only exists after a
|
||
real client install — leaving them in would break Kerberos on the nodes.
|
||
`auto_fast_armor = true` is kept and works, because it uses the host keytab.
|
||
|
||
## 4.5 Users and groups
|
||
|
||
| Object | ID | Notes |
|
||
|---|---|---|
|
||
| `hpcusers` (group) | 721800003 | All cluster users |
|
||
| `hpcuser` | 721800005 | Runs the MPI test jobs |
|
||
| `testuser` | 721800006 | Login verification |
|
||
| `admin` | 721800000 | IPA administrator |
|
||
|
||
### Why there is no IPA user named `ilker`
|
||
|
||
cloud-init creates a **local** `ilker` (uid 1000) on every VM. `nsswitch.conf`
|
||
resolves `files` before `sss`, so an IPA user with the same name would resolve
|
||
to uid 1000 on head01 and idm1 but to 721800xxx on the nodes — the same
|
||
username with two different uids depending on where you land, which silently
|
||
corrupts NFS home ownership.
|
||
|
||
For the same reason the local `hpcuser` was **deleted** from both head01 and
|
||
the node image (`wwctl image syncuser` had copied it there), so the IPA account
|
||
is the only one.
|
||
|
||
The local `ilker` home was moved from `/home/ilker` to `/var/home/ilker`
|
||
so that the NFS `/home` mount does not hide it.
|
||
|
||
### Adding a user
|
||
|
||
```bash
|
||
ssh idm1
|
||
sudo bash -c '. /root/.ipa-credentials
|
||
echo $IPA_ADMIN_PASSWORD | kinit admin
|
||
ipa user-add ayse --first=Ayse --last=Yilmaz \
|
||
--homedir=/home/ayse --shell=/bin/bash --password
|
||
ipa group-add-member hpcusers --users=ayse'
|
||
```
|
||
|
||
The home directory is created automatically on first login by
|
||
`pam_oddjob_mkhomedir`, on the NFS share, on both head and nodes. No
|
||
pre-creation is needed — this was verified explicitly.
|
||
|
||
## 4.6 Credentials
|
||
|
||
Everything lives in `idm1:/root/.ipa-credentials`, mode 0600: the IPA admin
|
||
password, the Directory Manager password, and the generated test-user
|
||
passwords. **No password is stored in this repository.**
|
||
|
||
```bash
|
||
ssh idm1
|
||
sudo cat /root/.ipa-credentials
|
||
sudo bash -c '. /root/.ipa-credentials; echo $IPA_ADMIN_PASSWORD | kinit admin'
|
||
sudo ipa user-find
|
||
```
|
||
|
||
## 4.7 Web UI access
|
||
|
||
The certificate is issued for `idm1.hpc.local`, and that name resolves to
|
||
10.10.10.5 — unreachable from the laptop. Use an SSH tunnel:
|
||
|
||
```bash
|
||
ssh -L 8443:10.10.10.5:443 head01
|
||
# add "127.0.0.1 idm1.hpc.local" to /etc/hosts, then browse:
|
||
# https://idm1.hpc.local:8443
|
||
```
|
||
|
||
Going straight to `https://10.0.0.31` reaches the server but produces a
|
||
certificate name mismatch, and IPA's web UI redirects to its canonical
|
||
hostname regardless.
|