Files
werewulf/docs/21-portal-layer.md
ilkerandClaude Opus 5.5 861aceb283 docs: bring sections 01-23 in line with hpcod setup and what is built
- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for
  installing the control plane.
- 09: repository layout now includes hpcod/, docs 20-23 and the B/C
  scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is
  recorded, not absent (matches 6.6). Handbook PDF regenerated.
- 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run
  from the control plane; Terraform state per cluster, not in Postgres;
  no reaper yet; usage per project.
- 21: c<NNN> portals, publishing wired into provisioning, closed-network
  mode.
- 22: three published images; chosen apps now reach the cluster.
- 23: setup installs the catalogue; apps.local keeps a site's own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:50:33 +01:00

13 KiB
Raw Permalink Blame History

21. The portal layer — what to adopt, and what not to

Section 20 designs a control plane that creates clusters. This section is about the layer above it: what a person actually uses once their cluster exists. Today that answer is "SSH and sbatch", and this section works out which parts of the established end-user-portal practice are worth adopting.

Status: partly built. Adoption 1 is done and published: a cluster's portal answers at c<NNN>.hpc.example.org. Adoption 2 is done — the verify role runs a real job rather than reading sinfo and hoping. Adoption 3 was implemented and then deleted; 21.4 records why. Adoption 4 is still open.


21.1 Two layers, not one

It is easy to conflate these, and the conflation hides the gap.

  ┌──────────────────────────────────────────────┐
  │  end-user portal       submit a job, watch    │  ← does not exist here
  │  (Open OnDemand …)     it, open a desktop     │
  ├──────────────────────────────────────────────┤
  │  hpcod control plane   pick a site, build a   │  ← built, §20
  │                        cluster, tear it down  │
  ├──────────────────────────────────────────────┤
  │  Proxmox sites         hardware               │
  └──────────────────────────────────────────────┘

hpcod serves the person who procures a cluster. It answers "which site has InfiniBand", "what would this cost me in nodes", "give me fourteen days". Its surface is /v1/sites, /v1/capacity, /v1/dry-run and the pages at hpc.example.org.

A provisioned tenant arrives with Slurm, MUNGE, OpenHPC, NFS /home and sshd. Everything a scheduler needs, and nothing a user sees. The person who uses the cluster has no surface at all.

That is the gap this section is about. It is not a defect in the control plane — it is a layer nobody has built yet.

21.2 The source, and how to read it

The prompt for this analysis was NI SP's How to choose an HPC portal: a practical guide for IT teams. It proposes five criteria: scheduler integration depth, remote visualization and VDI, support and escalation, release cadence and upgrade experience, and total cost of ownership.

It is vendor content — NI SP sells EF Portal, and three of the five criteria (support SLA, release cadence, the invisible engineering cost of open source) are the three where a commercial product outscores Open OnDemand by construction. That does not make them wrong. It means the useful move is to take the question underneath each one rather than the comparison on top of it. "What is your support SLA" does not apply to software we write ourselves; "what happens when the one person who understands this leaves" is the same question and it does apply.

The guide also explicitly scopes out multi-cluster placement, cloud bursting, chargeback and authentication. Those are precisely what section 20 is about, so the framework under-weights the part of this system that already exists.

21.3 Where we stand against each criterion

Criterion Our position
Scheduler integration depth We install Slurm 25.11 and OpenHPC 3; we expose no job interface. The guide's real test — can a submission form enforce resource policy so invalid requests are impossible — has an analogue one layer down that we do have: /v1/dry-run refuses a placement no site can satisfy, before a lease is committed. At the user layer: nothing.
Remote visualization / VDI Absent, and not a simple addition. See 21.5.
Support and escalation Restated as continuity, this is partly covered: sections 01–09 document the cluster, section 20 the control plane, and CLAUDE.md records the failure modes of the tooling itself. What is missing is an operational runbook for a tenant as opposed to the lab.
Release cadence and upgrade The criterion that does not transfer, and the one this section got wrong first time round. It presumes a cluster too permanent to delete. Ours are not: a tenant is rebuilt from the template in 141s, so "upgrade" means destroy and reprovision. See 21.4 item 3.
Total cost of ownership The criterion that lands hardest. We are not on the open-source path the guide warns about; we are further out, on a bespoke one. Cutting provisioning from ~900s to 134s took days of engineering, and that cost appears in no budget.

21.4 What to adopt

Four items, in the order they pay off.

1. Ship a user-facing portal inside the tenant

Open OnDemand is the obvious candidate: no licence, Slurm-native, and it wants exactly the machine we already have — the head node runs the Slurm controller and exports /home, which is what OOD needs to launch interactive sessions and browse files.

It should be baked into template 9100 like everything else. The pattern is settled (§20, and the selfconfig role): software goes into the template, tenant-specific configuration comes from the inventory cloud-init writes. Done that way it costs a tenant nothing at provision time.

Done. A cluster's portal is at c<NNN>.hpc.example.org, behind a Let's Encrypt wildcard certificate, asking for that tenant's own credentials. What follows is how, because most of it lives outside this repository and cannot be rebuilt from these files alone.

Reachability was the part worth settling first, and the obvious answer was wrong. A path under the existing site — hpc.example.org/c002/ — does not work: OnDemand builds absolute URLs from the root (/pun/, /public/, /node/) and documents no base-path setting, so it would serve a login page and then send every link on it back to the control plane. It has to own a hostname.

  browser ──▶ Caddy ──▶ hpcod ──▶ tenant OnDemand
             wildcard   routes by   its own login,
             cert       Host, pins  on the management
                        the cert    address only
  • DNS. One wildcard record, *.hpc.example.org. Creating a tenant touches DNS not at all, which is the point: no per-tenant record, nothing to propagate, nothing to wait for on the provisioning path.
  • Certificate. A wildcard from certbot over DNS-01, using a hook that drives the PowerDNS API. Issued once, renewed by certbot's own timer; a deploy hook republishes it to Caddy and reloads. Renewal is verified by dry-run, not assumed — a --manual certificate cannot auto-renew without an auth hook.
  • Caddy. One static block for *.hpc.example.org. It never changes when a tenant is created or destroyed, so provisioning needs no write access to the web server's configuration.
  • Routing. hpcod resolves the subdomain to a tenant, because it is the only component that knows which tenants exist. It sits outside the control plane's own login: a tenant portal has its own accounts, and demanding a control-plane credential first would be wrong.
  • Trust. Each tenant signs its own certificate; the control plane pins it as the sole root for that one tenant. Stricter than a shared internal CA, because one tenant's certificate cannot authenticate another. A tenant whose certificate is missing is left unpublished rather than proxied to blindly.
  • Publishing. Wired into provisioning: when a build finishes, hpcod waits for the head's portal, writes the registry and copies the certificate itself; releasing the cluster withdraws it. scripts/B40-publish-cluster.py does the same by hand, for a head built outside the control plane.
  • Closed networks. Where there is no public DNS or Let's Encrypt, hpcod can serve HTTPS itself with a CA constrained to the site's domains, and answer DNS for the portal and *.CLUSTER_DOMAIN. hpcod setup asks which; DEPLOY.md has the details.

The portal itself listens on the management address only, never on the tenant fabric. Compute nodes have no business reaching it.

Two things found on the way, both pre-existing and neither about OnDemand. The .xyz delegation for example.org points at ns1.example.net/ns2.example.net, not at the ns1.example.org in the zone's own NS records — so every check made against ns1 had been reading half the answer. And the zone had no SOA-EDIT-API, so writes through the API never bumped the serial and the secondary never transferred them; it had been stale for some time. The wildcard certificate could not be issued until both were understood.

2. Turn the guide's trial checklist into an acceptance test

Its five trial items — submit with advanced resource flags, have an unfamiliar user submit unaided, launch an interactive session and time it, open a support ticket, confirm admin visibility across jobs and node health — are, with the support item dropped, a description of what a freshly provisioned tenant should be able to do.

We already run a verify role at the end of every provision, and it checks services and sinfo. Extending it to run an actual job proves the cluster at the level users care about. This is the cheapest item on the list and the one that would have caught the most: several times during this work sinfo reported idle nodes on a cluster that could not have run anything.

3. Nothing. The guide's upgrade criterion does not transfer

This item previously proposed building a rolling upgrade: drain each compute node, reboot it into a new image, resume it, and separately find some way to move a running tenant's head node forward. It was written, it worked, and it was deleted. Recording why, because the reasoning is the useful part.

The guide asks whether upgrades can happen without draining the cluster. That question assumes a permanent cluster — one that has run for years, holds state nobody can recreate, and cannot be deleted. Under that assumption, in-place upgrade is the only option and doing it without losing work is worth paying for.

A tenant here provisions cold in 141 seconds and holds a lease. Its upgrade path is to destroy it and provision a new one, which is faster than rolling, leaves nothing half-migrated, and needs no machinery at all. The same answer covers the head node: there is no gap to fill, because "a tenant provisioned last month runs last month's template" is not a defect in an architecture where tenants are meant to be short-lived.

The narrow case where rolling would matter is a long lease with live work and user data in /home, where something has to change mid-lease and destroying is not available. If that case turns out to be real, this is the place to argue for it — with the lease length that makes it real, rather than by importing a criterion written for a different kind of system.

This entry is kept as a correction, not an adoption. Taking the guide's five criteria and working out which apply was the point of this section; this is the one that does not, and it was implemented before that was noticed.

4. Record the engineering cost

The guide is right that this stays invisible. allocation already records what hardware was held and for how long (§20.12), which is the raw material for tenant-side cost. Nothing records what the platform costs to build and run. A provisioning system whose true cost is unstated will be compared against products whose price is on a web page, and lose the argument for the wrong reason.

21.5 What not to adopt

DCV, or any GPU-accelerated VDI, as a drop-in. Two real conflicts, both architectural:

  • Compute nodes are diskless with the root filesystem in tmpfs. Image size is RAM cost, directly: the current image is 937 MB unpacked and every megabyte is paid in every node's memory and again in every PXE transfer. A graphics stack is not a small addition to that.
  • Our GPU model is whole-device passthrough. The selling point of DCV on bare metal is GPU sharing without vGPU licences, which lands on the fractional-GPU question section 20.12 already lists as open. Adopting the tool does not answer the question; it just makes answering it urgent.

Interactive visualization is a legitimate requirement and may well be worth both costs. It is a decision about what this platform is for, not a feature to add quietly.

The commercial-versus-open-source comparison. It does not map. We are on a third path, and the guide's TCO argument applies to us more sharply than to either option it compares — while its remedy, buying a supported product, does not apply at all.

21.6 What this raises

  • Does a tenant get a portal, or does the platform? A portal per tenant is isolated and multiplies the reachability problem by the number of tenants. One portal for all tenants means a single front end that knows how to reach many Slurm controllers — closer to what hpcod already is, and a much larger piece of software.
  • Who are the users? The guide assumes a stable population on a permanent cluster. Ours arrive with a lease and leave with it. Identity is already designed for both (§20.8, dedicated or shared FreeIPA); a portal has to make the same choice, and should make the same one.
  • Does POST /v1/clusters come first? It did: the endpoint and the portal's create page now share one path, from placement to a published cluster portal (§20.9).