- 01, 02: say the lab is built by hand and point to hpcod/DEPLOY.md for installing the control plane. - 09: repository layout now includes hpcod/, docs 20-23 and the B/C scripts; ports for hpcod's own HTTPS and DNS; Slurm accounting is recorded, not absent (matches 6.6). Handbook PDF regenerated. - 20: /v1/clusters exists; first-boot selfconfig instead of a playbook run from the control plane; Terraform state per cluster, not in Postgres; no reaper yet; usage per project. - 21: c<NNN> portals, publishing wired into provisioning, closed-network mode. - 22: three published images; chosen apps now reach the cluster. - 23: setup installs the catalogue; apps.local keeps a site's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
239 lines
13 KiB
Markdown
239 lines
13 KiB
Markdown
# 21. The portal layer — what to adopt, and what not to
|
||
|
||
Section 20 designs a control plane that *creates* clusters. This section is
|
||
about the layer above it: what a person actually uses once their cluster
|
||
exists. Today that answer is "SSH and `sbatch`", and this section works out
|
||
which parts of the established end-user-portal practice are worth adopting.
|
||
|
||
**Status: partly built.** Adoption 1 is done and published: a cluster's
|
||
portal answers at `c<NNN>.hpc.example.org`. Adoption 2 is done — the verify
|
||
role runs a real job rather than reading `sinfo` and hoping. Adoption 3 was
|
||
implemented and then deleted; 21.4 records why. Adoption 4 is still open.
|
||
|
||
---
|
||
|
||
## 21.1 Two layers, not one
|
||
|
||
It is easy to conflate these, and the conflation hides the gap.
|
||
|
||
```
|
||
┌──────────────────────────────────────────────┐
|
||
│ end-user portal submit a job, watch │ ← does not exist here
|
||
│ (Open OnDemand …) it, open a desktop │
|
||
├──────────────────────────────────────────────┤
|
||
│ hpcod control plane pick a site, build a │ ← built, §20
|
||
│ cluster, tear it down │
|
||
├──────────────────────────────────────────────┤
|
||
│ Proxmox sites hardware │
|
||
└──────────────────────────────────────────────┘
|
||
```
|
||
|
||
`hpcod` serves the person who *procures* a cluster. It answers "which site
|
||
has InfiniBand", "what would this cost me in nodes", "give me fourteen days".
|
||
Its surface is `/v1/sites`, `/v1/capacity`, `/v1/dry-run` and the pages at
|
||
`hpc.example.org`.
|
||
|
||
A provisioned tenant arrives with Slurm, MUNGE, OpenHPC, NFS `/home` and
|
||
sshd. Everything a scheduler needs, and nothing a user sees. The person who
|
||
*uses* the cluster has no surface at all.
|
||
|
||
That is the gap this section is about. It is not a defect in the control
|
||
plane — it is a layer nobody has built yet.
|
||
|
||
## 21.2 The source, and how to read it
|
||
|
||
The prompt for this analysis was NI SP's *How to choose an HPC portal: a
|
||
practical guide for IT teams*. It proposes five criteria: scheduler
|
||
integration depth, remote visualization and VDI, support and escalation,
|
||
release cadence and upgrade experience, and total cost of ownership.
|
||
|
||
It is vendor content — NI SP sells EF Portal, and three of the five criteria
|
||
(support SLA, release cadence, the invisible engineering cost of open source)
|
||
are the three where a commercial product outscores Open OnDemand by
|
||
construction. That does not make them wrong. It means the useful move is to
|
||
take the question underneath each one rather than the comparison on top of
|
||
it. "What is your support SLA" does not apply to software we write
|
||
ourselves; "what happens when the one person who understands this leaves"
|
||
is the same question and it does apply.
|
||
|
||
The guide also explicitly scopes out multi-cluster placement, cloud bursting,
|
||
chargeback and authentication. Those are precisely what section 20 is about,
|
||
so the framework under-weights the part of this system that already exists.
|
||
|
||
## 21.3 Where we stand against each criterion
|
||
|
||
| Criterion | Our position |
|
||
|---|---|
|
||
| **Scheduler integration depth** | We *install* Slurm 25.11 and OpenHPC 3; we expose no job interface. The guide's real test — can a submission form enforce resource policy so invalid requests are impossible — has an analogue one layer down that we do have: `/v1/dry-run` refuses a placement no site can satisfy, before a lease is committed. At the user layer: nothing. |
|
||
| **Remote visualization / VDI** | Absent, and not a simple addition. See 21.5. |
|
||
| **Support and escalation** | Restated as continuity, this is partly covered: sections 01–09 document the cluster, section 20 the control plane, and `CLAUDE.md` records the failure modes of the tooling itself. What is missing is an operational runbook for a *tenant* as opposed to the lab. |
|
||
| **Release cadence and upgrade** | The criterion that does not transfer, and the one this section got wrong first time round. It presumes a cluster too permanent to delete. Ours are not: a tenant is rebuilt from the template in 141s, so "upgrade" means destroy and reprovision. See 21.4 item 3. |
|
||
| **Total cost of ownership** | The criterion that lands hardest. We are not on the open-source path the guide warns about; we are further out, on a bespoke one. Cutting provisioning from ~900s to 134s took days of engineering, and that cost appears in no budget. |
|
||
|
||
## 21.4 What to adopt
|
||
|
||
Four items, in the order they pay off.
|
||
|
||
### 1. Ship a user-facing portal inside the tenant
|
||
|
||
Open OnDemand is the obvious candidate: no licence, Slurm-native, and it
|
||
wants exactly the machine we already have — the head node runs the Slurm
|
||
controller and exports `/home`, which is what OOD needs to launch
|
||
interactive sessions and browse files.
|
||
|
||
It should be baked into template 9100 like everything else. The pattern is
|
||
settled (§20, and the `selfconfig` role): software goes into the template,
|
||
tenant-specific configuration comes from the inventory cloud-init writes. Done
|
||
that way it costs a tenant nothing at provision time.
|
||
|
||
**Done.** A cluster's portal is at `c<NNN>.hpc.example.org`, behind a
|
||
Let's Encrypt wildcard certificate, asking for that tenant's own
|
||
credentials. What follows is how, because most of it lives outside this
|
||
repository and cannot be rebuilt from these files alone.
|
||
|
||
Reachability was the part worth settling first, and the obvious answer was
|
||
wrong. A path under the existing site — `hpc.example.org/c002/` — does not
|
||
work: OnDemand builds absolute URLs from the root (`/pun/`, `/public/`,
|
||
`/node/`) and documents no base-path setting, so it would serve a login page
|
||
and then send every link on it back to the control plane. It has to own a
|
||
hostname.
|
||
|
||
```
|
||
browser ──▶ Caddy ──▶ hpcod ──▶ tenant OnDemand
|
||
wildcard routes by its own login,
|
||
cert Host, pins on the management
|
||
the cert address only
|
||
```
|
||
|
||
- **DNS.** One wildcard record, `*.hpc.example.org`. Creating a tenant
|
||
touches DNS not at all, which is the point: no per-tenant record, nothing
|
||
to propagate, nothing to wait for on the provisioning path.
|
||
- **Certificate.** A wildcard from certbot over DNS-01, using a hook that
|
||
drives the PowerDNS API. Issued once, renewed by certbot's own timer; a
|
||
deploy hook republishes it to Caddy and reloads. Renewal is verified by
|
||
dry-run, not assumed — a `--manual` certificate cannot auto-renew without
|
||
an auth hook.
|
||
- **Caddy.** One static block for `*.hpc.example.org`. It never changes when
|
||
a tenant is created or destroyed, so provisioning needs no write access to
|
||
the web server's configuration.
|
||
- **Routing.** `hpcod` resolves the subdomain to a tenant, because it is the
|
||
only component that knows which tenants exist. It sits outside the control
|
||
plane's own login: a tenant portal has its own accounts, and demanding a
|
||
control-plane credential first would be wrong.
|
||
- **Trust.** Each tenant signs its own certificate; the control plane pins
|
||
it as the sole root for that one tenant. Stricter than a shared internal
|
||
CA, because one tenant's certificate cannot authenticate another. A tenant
|
||
whose certificate is missing is left unpublished rather than proxied to
|
||
blindly.
|
||
- **Publishing.** Wired into provisioning: when a build finishes, hpcod
|
||
waits for the head's portal, writes the registry and copies the
|
||
certificate itself; releasing the cluster withdraws it.
|
||
`scripts/B40-publish-cluster.py` does the same by hand, for a head built
|
||
outside the control plane.
|
||
- **Closed networks.** Where there is no public DNS or Let's Encrypt,
|
||
hpcod can serve HTTPS itself with a CA constrained to the site's domains,
|
||
and answer DNS for the portal and `*.CLUSTER_DOMAIN`. `hpcod setup` asks
|
||
which; DEPLOY.md has the details.
|
||
|
||
The portal itself listens on the management address only, never on the
|
||
tenant fabric. Compute nodes have no business reaching it.
|
||
|
||
Two things found on the way, both pre-existing and neither about OnDemand.
|
||
The `.xyz` delegation for `example.org` points at `ns1.example.net`/`ns2.example.net`,
|
||
not at the `ns1.example.org` in the zone's own NS records — so every check
|
||
made against `ns1` had been reading half the answer. And the zone had no
|
||
`SOA-EDIT-API`, so writes through the API never bumped the serial and the
|
||
secondary never transferred them; it had been stale for some time. The
|
||
wildcard certificate could not be issued until both were understood.
|
||
|
||
### 2. Turn the guide's trial checklist into an acceptance test
|
||
|
||
Its five trial items — submit with advanced resource flags, have an
|
||
unfamiliar user submit unaided, launch an interactive session and time it,
|
||
open a support ticket, confirm admin visibility across jobs and node health —
|
||
are, with the support item dropped, a description of what a freshly
|
||
provisioned tenant should be able to do.
|
||
|
||
We already run a `verify` role at the end of every provision, and it checks
|
||
services and `sinfo`. Extending it to run an actual job proves the cluster
|
||
at the level users care about. This is the cheapest item on the list and the
|
||
one that would have caught the most: several times during this work `sinfo`
|
||
reported `idle` nodes on a cluster that could not have run anything.
|
||
|
||
### 3. Nothing. The guide's upgrade criterion does not transfer
|
||
|
||
This item previously proposed building a rolling upgrade: drain each compute
|
||
node, reboot it into a new image, resume it, and separately find some way to
|
||
move a running tenant's head node forward. It was written, it worked, and it
|
||
was deleted. Recording why, because the reasoning is the useful part.
|
||
|
||
The guide asks whether upgrades can happen without draining the cluster. That
|
||
question assumes a permanent cluster — one that has run for years, holds
|
||
state nobody can recreate, and cannot be deleted. Under that assumption,
|
||
in-place upgrade is the only option and doing it without losing work is worth
|
||
paying for.
|
||
|
||
A tenant here provisions cold in 141 seconds and holds a lease. Its upgrade
|
||
path is to destroy it and provision a new one, which is faster than rolling,
|
||
leaves nothing half-migrated, and needs no machinery at all. The same answer
|
||
covers the head node: there is no gap to fill, because "a tenant provisioned
|
||
last month runs last month's template" is not a defect in an architecture
|
||
where tenants are meant to be short-lived.
|
||
|
||
The narrow case where rolling would matter is a long lease with live work and
|
||
user data in `/home`, where something has to change mid-lease and destroying
|
||
is not available. If that case turns out to be real, this is the place to
|
||
argue for it — with the lease length that makes it real, rather than by
|
||
importing a criterion written for a different kind of system.
|
||
|
||
This entry is kept as a correction, not an adoption. Taking the guide's five
|
||
criteria and working out which apply was the point of this section; this is
|
||
the one that does not, and it was implemented before that was noticed.
|
||
|
||
### 4. Record the engineering cost
|
||
|
||
The guide is right that this stays invisible. `allocation` already records
|
||
what hardware was held and for how long (§20.12), which is the raw material
|
||
for tenant-side cost. Nothing records what the platform costs to build and
|
||
run. A provisioning system whose true cost is unstated will be compared
|
||
against products whose price is on a web page, and lose the argument for the
|
||
wrong reason.
|
||
|
||
## 21.5 What not to adopt
|
||
|
||
**DCV, or any GPU-accelerated VDI, as a drop-in.** Two real conflicts, both
|
||
architectural:
|
||
|
||
- Compute nodes are diskless with the root filesystem in tmpfs. Image size is
|
||
RAM cost, directly: the current image is 937 MB unpacked and every
|
||
megabyte is paid in every node's memory and again in every PXE transfer. A
|
||
graphics stack is not a small addition to that.
|
||
- Our GPU model is whole-device passthrough. The selling point of DCV on
|
||
bare metal is GPU *sharing* without vGPU licences, which lands on the
|
||
fractional-GPU question section 20.12 already lists as open. Adopting the
|
||
tool does not answer the question; it just makes answering it urgent.
|
||
|
||
Interactive visualization is a legitimate requirement and may well be worth
|
||
both costs. It is a decision about what this platform is for, not a feature
|
||
to add quietly.
|
||
|
||
**The commercial-versus-open-source comparison.** It does not map. We are on
|
||
a third path, and the guide's TCO argument applies to us more sharply than to
|
||
either option it compares — while its remedy, buying a supported product,
|
||
does not apply at all.
|
||
|
||
## 21.6 What this raises
|
||
|
||
- **Does a tenant get a portal, or does the platform?** A portal per tenant
|
||
is isolated and multiplies the reachability problem by the number of
|
||
tenants. One portal for all tenants means a single front end that knows how
|
||
to reach many Slurm controllers — closer to what `hpcod` already is, and a
|
||
much larger piece of software.
|
||
- **Who are the users?** The guide assumes a stable population on a permanent
|
||
cluster. Ours arrive with a lease and leave with it. Identity is already
|
||
designed for both (§20.8, dedicated or shared FreeIPA); a portal has to
|
||
make the same choice, and should make the same one.
|
||
- **Does `POST /v1/clusters` come first?** It did: the endpoint and the
|
||
portal's create page now share one path, from placement to a published
|
||
cluster portal (§20.9).
|