- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for installing the control plane. - 09: repository layout now includes hpcod/, docs 20-23 and the B/C scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is recorded, not absent (matches 6.6). Handbook PDF regenerated. - 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run from the control plane; Terraform state per cluster, not in Postgres; no reaper yet; usage per project. - 21: c<NNN> portals, publishing wired into provisioning, closed-network mode. - 22: three published images; chosen apps now reach the cluster. - 23: setup installs the catalogue; apps.local keeps a site's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
13 KiB
21. The portal layer — what to adopt, and what not to
Section 20 designs a control plane that creates clusters. This section is
about the layer above it: what a person actually uses once their cluster
exists. Today that answer is "SSH and sbatch", and this section works out
which parts of the established end-user-portal practice are worth adopting.
Status: partly built. Adoption 1 is done and published: a cluster's
portal answers at c<NNN>.hpc.example.org. Adoption 2 is done — the verify
role runs a real job rather than reading sinfo and hoping. Adoption 3 was
implemented and then deleted; 21.4 records why. Adoption 4 is still open.
21.1 Two layers, not one
It is easy to conflate these, and the conflation hides the gap.
┌──────────────────────────────────────────────┐
│ end-user portal submit a job, watch │ ← does not exist here
│ (Open OnDemand …) it, open a desktop │
├──────────────────────────────────────────────┤
│ hpcod control plane pick a site, build a │ ← built, §20
│ cluster, tear it down │
├──────────────────────────────────────────────┤
│ Proxmox sites hardware │
└──────────────────────────────────────────────┘
hpcod serves the person who procures a cluster. It answers "which site
has InfiniBand", "what would this cost me in nodes", "give me fourteen days".
Its surface is /v1/sites, /v1/capacity, /v1/dry-run and the pages at
hpc.example.org.
A provisioned tenant arrives with Slurm, MUNGE, OpenHPC, NFS /home and
sshd. Everything a scheduler needs, and nothing a user sees. The person who
uses the cluster has no surface at all.
That is the gap this section is about. It is not a defect in the control plane — it is a layer nobody has built yet.
21.2 The source, and how to read it
The prompt for this analysis was NI SP's How to choose an HPC portal: a practical guide for IT teams. It proposes five criteria: scheduler integration depth, remote visualization and VDI, support and escalation, release cadence and upgrade experience, and total cost of ownership.
It is vendor content — NI SP sells EF Portal, and three of the five criteria (support SLA, release cadence, the invisible engineering cost of open source) are the three where a commercial product outscores Open OnDemand by construction. That does not make them wrong. It means the useful move is to take the question underneath each one rather than the comparison on top of it. "What is your support SLA" does not apply to software we write ourselves; "what happens when the one person who understands this leaves" is the same question and it does apply.
The guide also explicitly scopes out multi-cluster placement, cloud bursting, chargeback and authentication. Those are precisely what section 20 is about, so the framework under-weights the part of this system that already exists.
21.3 Where we stand against each criterion
| Criterion | Our position |
|---|---|
| Scheduler integration depth | We install Slurm 25.11 and OpenHPC 3; we expose no job interface. The guide's real test — can a submission form enforce resource policy so invalid requests are impossible — has an analogue one layer down that we do have: /v1/dry-run refuses a placement no site can satisfy, before a lease is committed. At the user layer: nothing. |
| Remote visualization / VDI | Absent, and not a simple addition. See 21.5. |
| Support and escalation | Restated as continuity, this is partly covered: sections 01–09 document the cluster, section 20 the control plane, and CLAUDE.md records the failure modes of the tooling itself. What is missing is an operational runbook for a tenant as opposed to the lab. |
| Release cadence and upgrade | The criterion that does not transfer, and the one this section got wrong first time round. It presumes a cluster too permanent to delete. Ours are not: a tenant is rebuilt from the template in 141s, so "upgrade" means destroy and reprovision. See 21.4 item 3. |
| Total cost of ownership | The criterion that lands hardest. We are not on the open-source path the guide warns about; we are further out, on a bespoke one. Cutting provisioning from ~900s to 134s took days of engineering, and that cost appears in no budget. |
21.4 What to adopt
Four items, in the order they pay off.
1. Ship a user-facing portal inside the tenant
Open OnDemand is the obvious candidate: no licence, Slurm-native, and it
wants exactly the machine we already have — the head node runs the Slurm
controller and exports /home, which is what OOD needs to launch
interactive sessions and browse files.
It should be baked into template 9100 like everything else. The pattern is
settled (§20, and the selfconfig role): software goes into the template,
tenant-specific configuration comes from the inventory cloud-init writes. Done
that way it costs a tenant nothing at provision time.
Done. A cluster's portal is at c<NNN>.hpc.example.org, behind a
Let's Encrypt wildcard certificate, asking for that tenant's own
credentials. What follows is how, because most of it lives outside this
repository and cannot be rebuilt from these files alone.
Reachability was the part worth settling first, and the obvious answer was
wrong. A path under the existing site — hpc.example.org/c002/ — does not
work: OnDemand builds absolute URLs from the root (/pun/, /public/,
/node/) and documents no base-path setting, so it would serve a login page
and then send every link on it back to the control plane. It has to own a
hostname.
browser ──▶ Caddy ──▶ hpcod ──▶ tenant OnDemand
wildcard routes by its own login,
cert Host, pins on the management
the cert address only
- DNS. One wildcard record,
*.hpc.example.org. Creating a tenant touches DNS not at all, which is the point: no per-tenant record, nothing to propagate, nothing to wait for on the provisioning path. - Certificate. A wildcard from certbot over DNS-01, using a hook that
drives the PowerDNS API. Issued once, renewed by certbot's own timer; a
deploy hook republishes it to Caddy and reloads. Renewal is verified by
dry-run, not assumed — a
--manualcertificate cannot auto-renew without an auth hook. - Caddy. One static block for
*.hpc.example.org. It never changes when a tenant is created or destroyed, so provisioning needs no write access to the web server's configuration. - Routing.
hpcodresolves the subdomain to a tenant, because it is the only component that knows which tenants exist. It sits outside the control plane's own login: a tenant portal has its own accounts, and demanding a control-plane credential first would be wrong. - Trust. Each tenant signs its own certificate; the control plane pins it as the sole root for that one tenant. Stricter than a shared internal CA, because one tenant's certificate cannot authenticate another. A tenant whose certificate is missing is left unpublished rather than proxied to blindly.
- Publishing. Wired into provisioning: when a build finishes, hpcod
waits for the head's portal, writes the registry and copies the
certificate itself; releasing the cluster withdraws it.
scripts/B40-publish-cluster.pydoes the same by hand, for a head built outside the control plane. - Closed networks. Where there is no public DNS or Let's Encrypt,
hpcod can serve HTTPS itself with a CA constrained to the site's domains,
and answer DNS for the portal and
*.CLUSTER_DOMAIN.hpcod setupasks which; DEPLOY.md has the details.
The portal itself listens on the management address only, never on the tenant fabric. Compute nodes have no business reaching it.
Two things found on the way, both pre-existing and neither about OnDemand.
The .xyz delegation for example.org points at ns1.example.net/ns2.example.net,
not at the ns1.example.org in the zone's own NS records — so every check
made against ns1 had been reading half the answer. And the zone had no
SOA-EDIT-API, so writes through the API never bumped the serial and the
secondary never transferred them; it had been stale for some time. The
wildcard certificate could not be issued until both were understood.
2. Turn the guide's trial checklist into an acceptance test
Its five trial items — submit with advanced resource flags, have an unfamiliar user submit unaided, launch an interactive session and time it, open a support ticket, confirm admin visibility across jobs and node health — are, with the support item dropped, a description of what a freshly provisioned tenant should be able to do.
We already run a verify role at the end of every provision, and it checks
services and sinfo. Extending it to run an actual job proves the cluster
at the level users care about. This is the cheapest item on the list and the
one that would have caught the most: several times during this work sinfo
reported idle nodes on a cluster that could not have run anything.
3. Nothing. The guide's upgrade criterion does not transfer
This item previously proposed building a rolling upgrade: drain each compute node, reboot it into a new image, resume it, and separately find some way to move a running tenant's head node forward. It was written, it worked, and it was deleted. Recording why, because the reasoning is the useful part.
The guide asks whether upgrades can happen without draining the cluster. That question assumes a permanent cluster — one that has run for years, holds state nobody can recreate, and cannot be deleted. Under that assumption, in-place upgrade is the only option and doing it without losing work is worth paying for.
A tenant here provisions cold in 141 seconds and holds a lease. Its upgrade path is to destroy it and provision a new one, which is faster than rolling, leaves nothing half-migrated, and needs no machinery at all. The same answer covers the head node: there is no gap to fill, because "a tenant provisioned last month runs last month's template" is not a defect in an architecture where tenants are meant to be short-lived.
The narrow case where rolling would matter is a long lease with live work and
user data in /home, where something has to change mid-lease and destroying
is not available. If that case turns out to be real, this is the place to
argue for it — with the lease length that makes it real, rather than by
importing a criterion written for a different kind of system.
This entry is kept as a correction, not an adoption. Taking the guide's five criteria and working out which apply was the point of this section; this is the one that does not, and it was implemented before that was noticed.
4. Record the engineering cost
The guide is right that this stays invisible. allocation already records
what hardware was held and for how long (§20.12), which is the raw material
for tenant-side cost. Nothing records what the platform costs to build and
run. A provisioning system whose true cost is unstated will be compared
against products whose price is on a web page, and lose the argument for the
wrong reason.
21.5 What not to adopt
DCV, or any GPU-accelerated VDI, as a drop-in. Two real conflicts, both architectural:
- Compute nodes are diskless with the root filesystem in tmpfs. Image size is RAM cost, directly: the current image is 937 MB unpacked and every megabyte is paid in every node's memory and again in every PXE transfer. A graphics stack is not a small addition to that.
- Our GPU model is whole-device passthrough. The selling point of DCV on bare metal is GPU sharing without vGPU licences, which lands on the fractional-GPU question section 20.12 already lists as open. Adopting the tool does not answer the question; it just makes answering it urgent.
Interactive visualization is a legitimate requirement and may well be worth both costs. It is a decision about what this platform is for, not a feature to add quietly.
The commercial-versus-open-source comparison. It does not map. We are on a third path, and the guide's TCO argument applies to us more sharply than to either option it compares — while its remedy, buying a supported product, does not apply at all.
21.6 What this raises
- Does a tenant get a portal, or does the platform? A portal per tenant
is isolated and multiplies the reachability problem by the number of
tenants. One portal for all tenants means a single front end that knows how
to reach many Slurm controllers — closer to what
hpcodalready is, and a much larger piece of software. - Who are the users? The guide assumes a stable population on a permanent cluster. Ours arrive with a lease and leave with it. Identity is already designed for both (§20.8, dedicated or shared FreeIPA); a portal has to make the same choice, and should make the same one.
- Does
POST /v1/clusterscome first? It did: the endpoint and the portal's create page now share one path, from placement to a published cluster portal (§20.9).