Files
werewulf/docs/21-portal-layer.md
ilkerandClaude Opus 5.5 861aceb283 docs: bring sections 01-23 in line with hpcod setup and what is built
- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for
  installing the control plane.
- 09: repository layout now includes hpcod/, docs 20-23 and the B/C
  scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is
  recorded, not absent (matches 6.6). Handbook PDF regenerated.
- 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run
  from the control plane; Terraform state per cluster, not in Postgres;
  no reaper yet; usage per project.
- 21: c<NNN> portals, publishing wired into provisioning, closed-network
  mode.
- 22: three published images; chosen apps now reach the cluster.
- 23: setup installs the catalogue; apps.local keeps a site's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:50:33 +01:00

239 lines
13 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 21. The portal layer — what to adopt, and what not to
Section 20 designs a control plane that *creates* clusters. This section is
about the layer above it: what a person actually uses once their cluster
exists. Today that answer is "SSH and `sbatch`", and this section works out
which parts of the established end-user-portal practice are worth adopting.
**Status: partly built.** Adoption 1 is done and published: a cluster's
portal answers at `c<NNN>.hpc.example.org`. Adoption 2 is done — the verify
role runs a real job rather than reading `sinfo` and hoping. Adoption 3 was
implemented and then deleted; 21.4 records why. Adoption 4 is still open.
---
## 21.1 Two layers, not one
It is easy to conflate these, and the conflation hides the gap.
```
┌──────────────────────────────────────────────┐
│ end-user portal submit a job, watch │ ← does not exist here
│ (Open OnDemand …) it, open a desktop │
├──────────────────────────────────────────────┤
│ hpcod control plane pick a site, build a │ ← built, §20
│ cluster, tear it down │
├──────────────────────────────────────────────┤
│ Proxmox sites hardware │
└──────────────────────────────────────────────┘
```
`hpcod` serves the person who *procures* a cluster. It answers "which site
has InfiniBand", "what would this cost me in nodes", "give me fourteen days".
Its surface is `/v1/sites`, `/v1/capacity`, `/v1/dry-run` and the pages at
`hpc.example.org`.
A provisioned tenant arrives with Slurm, MUNGE, OpenHPC, NFS `/home` and
sshd. Everything a scheduler needs, and nothing a user sees. The person who
*uses* the cluster has no surface at all.
That is the gap this section is about. It is not a defect in the control
plane — it is a layer nobody has built yet.
## 21.2 The source, and how to read it
The prompt for this analysis was NI SP's *How to choose an HPC portal: a
practical guide for IT teams*. It proposes five criteria: scheduler
integration depth, remote visualization and VDI, support and escalation,
release cadence and upgrade experience, and total cost of ownership.
It is vendor content — NI SP sells EF Portal, and three of the five criteria
(support SLA, release cadence, the invisible engineering cost of open source)
are the three where a commercial product outscores Open OnDemand by
construction. That does not make them wrong. It means the useful move is to
take the question underneath each one rather than the comparison on top of
it. "What is your support SLA" does not apply to software we write
ourselves; "what happens when the one person who understands this leaves"
is the same question and it does apply.
The guide also explicitly scopes out multi-cluster placement, cloud bursting,
chargeback and authentication. Those are precisely what section 20 is about,
so the framework under-weights the part of this system that already exists.
## 21.3 Where we stand against each criterion
| Criterion | Our position |
|---|---|
| **Scheduler integration depth** | We *install* Slurm 25.11 and OpenHPC 3; we expose no job interface. The guide's real test — can a submission form enforce resource policy so invalid requests are impossible — has an analogue one layer down that we do have: `/v1/dry-run` refuses a placement no site can satisfy, before a lease is committed. At the user layer: nothing. |
| **Remote visualization / VDI** | Absent, and not a simple addition. See 21.5. |
| **Support and escalation** | Restated as continuity, this is partly covered: sections 01–09 document the cluster, section 20 the control plane, and `CLAUDE.md` records the failure modes of the tooling itself. What is missing is an operational runbook for a *tenant* as opposed to the lab. |
| **Release cadence and upgrade** | The criterion that does not transfer, and the one this section got wrong first time round. It presumes a cluster too permanent to delete. Ours are not: a tenant is rebuilt from the template in 141s, so "upgrade" means destroy and reprovision. See 21.4 item 3. |
| **Total cost of ownership** | The criterion that lands hardest. We are not on the open-source path the guide warns about; we are further out, on a bespoke one. Cutting provisioning from ~900s to 134s took days of engineering, and that cost appears in no budget. |
## 21.4 What to adopt
Four items, in the order they pay off.
### 1. Ship a user-facing portal inside the tenant
Open OnDemand is the obvious candidate: no licence, Slurm-native, and it
wants exactly the machine we already have — the head node runs the Slurm
controller and exports `/home`, which is what OOD needs to launch
interactive sessions and browse files.
It should be baked into template 9100 like everything else. The pattern is
settled (§20, and the `selfconfig` role): software goes into the template,
tenant-specific configuration comes from the inventory cloud-init writes. Done
that way it costs a tenant nothing at provision time.
**Done.** A cluster's portal is at `c<NNN>.hpc.example.org`, behind a
Let's Encrypt wildcard certificate, asking for that tenant's own
credentials. What follows is how, because most of it lives outside this
repository and cannot be rebuilt from these files alone.
Reachability was the part worth settling first, and the obvious answer was
wrong. A path under the existing site — `hpc.example.org/c002/` — does not
work: OnDemand builds absolute URLs from the root (`/pun/`, `/public/`,
`/node/`) and documents no base-path setting, so it would serve a login page
and then send every link on it back to the control plane. It has to own a
hostname.
```
browser ──▶ Caddy ──▶ hpcod ──▶ tenant OnDemand
wildcard routes by its own login,
cert Host, pins on the management
the cert address only
```
- **DNS.** One wildcard record, `*.hpc.example.org`. Creating a tenant
touches DNS not at all, which is the point: no per-tenant record, nothing
to propagate, nothing to wait for on the provisioning path.
- **Certificate.** A wildcard from certbot over DNS-01, using a hook that
drives the PowerDNS API. Issued once, renewed by certbot's own timer; a
deploy hook republishes it to Caddy and reloads. Renewal is verified by
dry-run, not assumed — a `--manual` certificate cannot auto-renew without
an auth hook.
- **Caddy.** One static block for `*.hpc.example.org`. It never changes when
a tenant is created or destroyed, so provisioning needs no write access to
the web server's configuration.
- **Routing.** `hpcod` resolves the subdomain to a tenant, because it is the
only component that knows which tenants exist. It sits outside the control
plane's own login: a tenant portal has its own accounts, and demanding a
control-plane credential first would be wrong.
- **Trust.** Each tenant signs its own certificate; the control plane pins
it as the sole root for that one tenant. Stricter than a shared internal
CA, because one tenant's certificate cannot authenticate another. A tenant
whose certificate is missing is left unpublished rather than proxied to
blindly.
- **Publishing.** Wired into provisioning: when a build finishes, hpcod
waits for the head's portal, writes the registry and copies the
certificate itself; releasing the cluster withdraws it.
`scripts/B40-publish-cluster.py` does the same by hand, for a head built
outside the control plane.
- **Closed networks.** Where there is no public DNS or Let's Encrypt,
hpcod can serve HTTPS itself with a CA constrained to the site's domains,
and answer DNS for the portal and `*.CLUSTER_DOMAIN`. `hpcod setup` asks
which; DEPLOY.md has the details.
The portal itself listens on the management address only, never on the
tenant fabric. Compute nodes have no business reaching it.
Two things found on the way, both pre-existing and neither about OnDemand.
The `.xyz` delegation for `example.org` points at `ns1.example.net`/`ns2.example.net`,
not at the `ns1.example.org` in the zone's own NS records — so every check
made against `ns1` had been reading half the answer. And the zone had no
`SOA-EDIT-API`, so writes through the API never bumped the serial and the
secondary never transferred them; it had been stale for some time. The
wildcard certificate could not be issued until both were understood.
### 2. Turn the guide's trial checklist into an acceptance test
Its five trial items — submit with advanced resource flags, have an
unfamiliar user submit unaided, launch an interactive session and time it,
open a support ticket, confirm admin visibility across jobs and node health —
are, with the support item dropped, a description of what a freshly
provisioned tenant should be able to do.
We already run a `verify` role at the end of every provision, and it checks
services and `sinfo`. Extending it to run an actual job proves the cluster
at the level users care about. This is the cheapest item on the list and the
one that would have caught the most: several times during this work `sinfo`
reported `idle` nodes on a cluster that could not have run anything.
### 3. Nothing. The guide's upgrade criterion does not transfer
This item previously proposed building a rolling upgrade: drain each compute
node, reboot it into a new image, resume it, and separately find some way to
move a running tenant's head node forward. It was written, it worked, and it
was deleted. Recording why, because the reasoning is the useful part.
The guide asks whether upgrades can happen without draining the cluster. That
question assumes a permanent cluster — one that has run for years, holds
state nobody can recreate, and cannot be deleted. Under that assumption,
in-place upgrade is the only option and doing it without losing work is worth
paying for.
A tenant here provisions cold in 141 seconds and holds a lease. Its upgrade
path is to destroy it and provision a new one, which is faster than rolling,
leaves nothing half-migrated, and needs no machinery at all. The same answer
covers the head node: there is no gap to fill, because "a tenant provisioned
last month runs last month's template" is not a defect in an architecture
where tenants are meant to be short-lived.
The narrow case where rolling would matter is a long lease with live work and
user data in `/home`, where something has to change mid-lease and destroying
is not available. If that case turns out to be real, this is the place to
argue for it — with the lease length that makes it real, rather than by
importing a criterion written for a different kind of system.
This entry is kept as a correction, not an adoption. Taking the guide's five
criteria and working out which apply was the point of this section; this is
the one that does not, and it was implemented before that was noticed.
### 4. Record the engineering cost
The guide is right that this stays invisible. `allocation` already records
what hardware was held and for how long (§20.12), which is the raw material
for tenant-side cost. Nothing records what the platform costs to build and
run. A provisioning system whose true cost is unstated will be compared
against products whose price is on a web page, and lose the argument for the
wrong reason.
## 21.5 What not to adopt
**DCV, or any GPU-accelerated VDI, as a drop-in.** Two real conflicts, both
architectural:
- Compute nodes are diskless with the root filesystem in tmpfs. Image size is
RAM cost, directly: the current image is 937 MB unpacked and every
megabyte is paid in every node's memory and again in every PXE transfer. A
graphics stack is not a small addition to that.
- Our GPU model is whole-device passthrough. The selling point of DCV on
bare metal is GPU *sharing* without vGPU licences, which lands on the
fractional-GPU question section 20.12 already lists as open. Adopting the
tool does not answer the question; it just makes answering it urgent.
Interactive visualization is a legitimate requirement and may well be worth
both costs. It is a decision about what this platform is for, not a feature
to add quietly.
**The commercial-versus-open-source comparison.** It does not map. We are on
a third path, and the guide's TCO argument applies to us more sharply than to
either option it compares — while its remedy, buying a supported product,
does not apply at all.
## 21.6 What this raises
- **Does a tenant get a portal, or does the platform?** A portal per tenant
is isolated and multiplies the reachability problem by the number of
tenants. One portal for all tenants means a single front end that knows how
to reach many Slurm controllers — closer to what `hpcod` already is, and a
much larger piece of software.
- **Who are the users?** The guide assumes a stable population on a permanent
cluster. Ours arrive with a lease and leave with it. Identity is already
designed for both (§20.8, dedicated or shared FreeIPA); a portal has to
make the same choice, and should make the same one.
- **Does `POST /v1/clusters` come first?** It did: the endpoint and the
portal's create page now share one path, from placement to a published
cluster portal (§20.9).