One tenant may run several HPC clusters at the same time. The code did not allow for that: a single tenant_id drove the bridge, the subnet, the MAC prefix, the VM names, the vendor-data filename and the public hostname. A tenant asking for a second cluster while the first was running would have been handed the same bridge, the same subnet and the same MAC prefix - and Warewulf matches nodes by MAC, so the second cluster would have broken the first rather than failing cleanly. Everything that must not collide is now keyed on cluster_id, and tenant_id survives as what it always meant: who owns this. Bridges are vmbrC###, hosts c###-head and c###-node#, the portal is c###.hpc.example.org, and the Slurm ClusterName is the cluster's - which matters for the accounting work next, because records attributed to a shared name cannot be separated afterwards. The bound of 254 therefore applies to clusters running at one time, not to tenants. It comes from the addressing and nothing else: cluster_id is the third octet of 10.64.x.0/24 and one byte of the MAC. That is written down in 20.5a rather than left to be rediscovered. The registry keeps its shape but not its name: /etc/hpcod/clusters.yaml with a clusters: key, /etc/hpcod/clusters/ for the pinned certificates, -clusters and -cluster-domain on the portal, and the two scripts renamed to match. A half-finished rename would have left the most-read file still calling a cluster a tenant, which is the confusion this commit exists to remove. Verified on a cold provision: terraform 37s, self-configuration 43s, nodes 61s, 141s total; nodes named c002-node[1-2], ClusterName c002, bridge vmbrC002, and the portal answering 401 at c002.hpc.example.org. Not addressed here, and the more binding limit at the scale this is meant for: a cluster cannot span Proxmox nodes. Its bridge is bridge-ports none, a local L2 island, so two hypervisors would give two segments and PXE would not cross. Proxmox SDN is the way out and is already enabled on hv01; docs/20 anticipated it with vlan_tag in the allocation record.
13 KiB
21. The portal layer — what to adopt, and what not to
Section 20 designs a control plane that creates clusters. This section is
about the layer above it: what a person actually uses once their cluster
exists. Today that answer is "SSH and sbatch", and this section works out
which parts of the established end-user-portal practice are worth adopting.
Status: partly built. Adoption 1 is done and published: a tenant's
portal answers at t<NNN>.hpc.example.org. Adoption 2 is done — the verify
role runs a real job rather than reading sinfo and hoping. Adoption 3 was
implemented and then deleted; 21.4 records why. Adoption 4 is still open.
21.1 Two layers, not one
It is easy to conflate these, and the conflation hides the gap.
┌──────────────────────────────────────────────┐
│ end-user portal submit a job, watch │ ← does not exist here
│ (Open OnDemand …) it, open a desktop │
├──────────────────────────────────────────────┤
│ hpcod control plane pick a site, build a │ ← built, §20
│ cluster, tear it down │
├──────────────────────────────────────────────┤
│ Proxmox sites hardware │
└──────────────────────────────────────────────┘
hpcod serves the person who procures a cluster. It answers "which site
has InfiniBand", "what would this cost me in nodes", "give me fourteen days".
Its surface is /v1/sites, /v1/capacity, /v1/dry-run and the pages at
hpc.example.org.
A provisioned tenant arrives with Slurm, MUNGE, OpenHPC, NFS /home and
sshd. Everything a scheduler needs, and nothing a user sees. The person who
uses the cluster has no surface at all.
That is the gap this section is about. It is not a defect in the control plane — it is a layer nobody has built yet.
21.2 The source, and how to read it
The prompt for this analysis was NI SP's How to choose an HPC portal: a practical guide for IT teams. It proposes five criteria: scheduler integration depth, remote visualization and VDI, support and escalation, release cadence and upgrade experience, and total cost of ownership.
It is vendor content — NI SP sells EF Portal, and three of the five criteria (support SLA, release cadence, the invisible engineering cost of open source) are the three where a commercial product outscores Open OnDemand by construction. That does not make them wrong. It means the useful move is to take the question underneath each one rather than the comparison on top of it. "What is your support SLA" does not apply to software we write ourselves; "what happens when the one person who understands this leaves" is the same question and it does apply.
The guide also explicitly scopes out multi-cluster placement, cloud bursting, chargeback and authentication. Those are precisely what section 20 is about, so the framework under-weights the part of this system that already exists.
21.3 Where we stand against each criterion
| Criterion | Our position |
|---|---|
| Scheduler integration depth | We install Slurm 25.11 and OpenHPC 3; we expose no job interface. The guide's real test — can a submission form enforce resource policy so invalid requests are impossible — has an analogue one layer down that we do have: /v1/dry-run refuses a placement no site can satisfy, before a lease is committed. At the user layer: nothing. |
| Remote visualization / VDI | Absent, and not a simple addition. See 21.5. |
| Support and escalation | Restated as continuity, this is partly covered: sections 01–09 document the cluster, section 20 the control plane, and CLAUDE.md records the failure modes of the tooling itself. What is missing is an operational runbook for a tenant as opposed to the lab. |
| Release cadence and upgrade | The criterion that does not transfer, and the one this section got wrong first time round. It presumes a cluster too permanent to delete. Ours are not: a tenant is rebuilt from the template in 141s, so "upgrade" means destroy and reprovision. See 21.4 item 3. |
| Total cost of ownership | The criterion that lands hardest. We are not on the open-source path the guide warns about; we are further out, on a bespoke one. Cutting provisioning from ~900s to 134s took days of engineering, and that cost appears in no budget. |
21.4 What to adopt
Four items, in the order they pay off.
1. Ship a user-facing portal inside the tenant
Open OnDemand is the obvious candidate: no licence, Slurm-native, and it
wants exactly the machine we already have — the head node runs the Slurm
controller and exports /home, which is what OOD needs to launch
interactive sessions and browse files.
It should be baked into template 9100 like everything else. The pattern is
settled (§20, and the selfconfig role): software goes into the template,
tenant-specific configuration comes from the inventory cloud-init writes. Done
that way it costs a tenant nothing at provision time.
Done. A tenant's portal is at t<NNN>.hpc.example.org, behind a
Let's Encrypt wildcard certificate, asking for that tenant's own
credentials. What follows is how, because most of it lives outside this
repository and cannot be rebuilt from these files alone.
Reachability was the part worth settling first, and the obvious answer was
wrong. A path under the existing site — hpc.example.org/t002/ — does not
work: OnDemand builds absolute URLs from the root (/pun/, /public/,
/node/) and documents no base-path setting, so it would serve a login page
and then send every link on it back to the control plane. It has to own a
hostname.
browser ──▶ Caddy ──▶ hpcod ──▶ tenant OnDemand
wildcard routes by its own login,
cert Host, pins on the management
the cert address only
- DNS. One wildcard record,
*.hpc.example.org. Creating a tenant touches DNS not at all, which is the point: no per-tenant record, nothing to propagate, nothing to wait for on the provisioning path. - Certificate. A wildcard from certbot over DNS-01, using a hook that
drives the PowerDNS API. Issued once, renewed by certbot's own timer; a
deploy hook republishes it to Caddy and reloads. Renewal is verified by
dry-run, not assumed — a
--manualcertificate cannot auto-renew without an auth hook. - Caddy. One static block for
*.hpc.example.org. It never changes when a tenant is created or destroyed, so provisioning needs no write access to the web server's configuration. - Routing.
hpcodresolves the subdomain to a tenant, because it is the only component that knows which tenants exist. It sits outside the control plane's own login: a tenant portal has its own accounts, and demanding a control-plane credential first would be wrong. - Trust. Each tenant signs its own certificate; the control plane pins it as the sole root for that one tenant. Stricter than a shared internal CA, because one tenant's certificate cannot authenticate another. A tenant whose certificate is missing is left unpublished rather than proxied to blindly.
- Publishing.
scripts/B40-publish-cluster.pywrites the registry and copies the certificate;--removewithdraws. Not yet wired into provisioning — the natural home isPOST /v1/clusterswhen that exists (§20.9).
The portal itself listens on the management address only, never on the tenant fabric. Compute nodes have no business reaching it.
Two things found on the way, both pre-existing and neither about OnDemand.
The .xyz delegation for example.org points at ns1.example.net/ns2.example.net,
not at the ns1.example.org in the zone's own NS records — so every check
made against ns1 had been reading half the answer. And the zone had no
SOA-EDIT-API, so writes through the API never bumped the serial and the
secondary never transferred them; it had been stale for some time. The
wildcard certificate could not be issued until both were understood.
2. Turn the guide's trial checklist into an acceptance test
Its five trial items — submit with advanced resource flags, have an unfamiliar user submit unaided, launch an interactive session and time it, open a support ticket, confirm admin visibility across jobs and node health — are, with the support item dropped, a description of what a freshly provisioned tenant should be able to do.
We already run a verify role at the end of every provision, and it checks
services and sinfo. Extending it to run an actual job proves the cluster
at the level users care about. This is the cheapest item on the list and the
one that would have caught the most: several times during this work sinfo
reported idle nodes on a cluster that could not have run anything.
3. Nothing. The guide's upgrade criterion does not transfer
This item previously proposed building a rolling upgrade: drain each compute node, reboot it into a new image, resume it, and separately find some way to move a running tenant's head node forward. It was written, it worked, and it was deleted. Recording why, because the reasoning is the useful part.
The guide asks whether upgrades can happen without draining the cluster. That question assumes a permanent cluster — one that has run for years, holds state nobody can recreate, and cannot be deleted. Under that assumption, in-place upgrade is the only option and doing it without losing work is worth paying for.
A tenant here provisions cold in 141 seconds and holds a lease. Its upgrade path is to destroy it and provision a new one, which is faster than rolling, leaves nothing half-migrated, and needs no machinery at all. The same answer covers the head node: there is no gap to fill, because "a tenant provisioned last month runs last month's template" is not a defect in an architecture where tenants are meant to be short-lived.
The narrow case where rolling would matter is a long lease with live work and
user data in /home, where something has to change mid-lease and destroying
is not available. If that case turns out to be real, this is the place to
argue for it — with the lease length that makes it real, rather than by
importing a criterion written for a different kind of system.
This entry is kept as a correction, not an adoption. Taking the guide's five criteria and working out which apply was the point of this section; this is the one that does not, and it was implemented before that was noticed.
4. Record the engineering cost
The guide is right that this stays invisible. allocation already records
what hardware was held and for how long (§20.12), which is the raw material
for tenant-side cost. Nothing records what the platform costs to build and
run. A provisioning system whose true cost is unstated will be compared
against products whose price is on a web page, and lose the argument for the
wrong reason.
21.5 What not to adopt
DCV, or any GPU-accelerated VDI, as a drop-in. Two real conflicts, both architectural:
- Compute nodes are diskless with the root filesystem in tmpfs. Image size is RAM cost, directly: the current image is 937 MB unpacked and every megabyte is paid in every node's memory and again in every PXE transfer. A graphics stack is not a small addition to that.
- Our GPU model is whole-device passthrough. The selling point of DCV on bare metal is GPU sharing without vGPU licences, which lands on the fractional-GPU question section 20.12 already lists as open. Adopting the tool does not answer the question; it just makes answering it urgent.
Interactive visualization is a legitimate requirement and may well be worth both costs. It is a decision about what this platform is for, not a feature to add quietly.
The commercial-versus-open-source comparison. It does not map. We are on a third path, and the guide's TCO argument applies to us more sharply than to either option it compares — while its remedy, buying a supported product, does not apply at all.
21.6 What this raises
- Does a tenant get a portal, or does the platform? A portal per tenant
is isolated and multiplies the reachability problem by the number of
tenants. One portal for all tenants means a single front end that knows how
to reach many Slurm controllers — closer to what
hpcodalready is, and a much larger piece of software. - Who are the users? The guide assumes a stable population on a permanent cluster. Ours arrive with a lease and leave with it. Identity is already designed for both (§20.8, dedicated or shared FreeIPA); a portal has to make the same choice, and should make the same one.
- Does
POST /v1/clusterscome first? It is designed (§20.9) and not implemented — provisioning runs from Terraform plus the first-boot path. A user-facing portal that cannot request a cluster is half a product, and that endpoint is the smaller piece of work.