Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
11 KiB
Changelog
v0.2.2 — 2026-11-16
Clusters across the nodes of a Proxmox cluster
A Proxmox VE that is one node of a Proxmox cluster no longer means one
node. Setup lists the cluster's nodes and asks which ones clusters may use
(PVE_NODES); the rest are never touched. Compute nodes are spread over
the chosen nodes. Heads stay on PVE_NODE, where the template is, so no
shared storage is needed. With more than one node the SDN zone becomes
vxlan, with every node a peer and an MTU of 1450 that guests are given
through virtio; setup remakes a simple zone only while no cluster uses
it, and hpcod verify checks the nodes and the zone. Before this, discovery
already saw every node and placement could pick any of them, while the
template, the snippets and a simple zone only worked on one: a cluster
could land where it could not be built.
Run on the simulated Proxmox with the zone forced to vxlan: a two-node test cluster built and ran a job, both compute NICs came up at 1450, and a full 1450-byte frame passed between them while 1451 did not. Traffic between two real Proxmox nodes has not been tested; the lab has one.
Also
- Setup's step list keeps to its columns on narrow terminals, and the check says "1 warning", not "1 warnings".
- The documented install works on a fresh host: it now downloads the
release, checks
SHA256SUMS, sets the execute bit and runssudo ./hpcod setup. Before, it saidsudo hpcod setup, which is not found until setup has installed itself. - The documentation was brought in line with what is built: section 02 installs hpcod from scratch, and the hand-built lab moved to section 10.
v0.2.1 — 2026-11-16
A rebuild of v0.2.0 with Go 1.27.4, for GO-2026-6859: a race in net/http's MaxBytesReader that can crash an HTTP/1 server, which the portal is. The v0.2.0 binaries were built with 1.27.3. Nothing else changes for anyone running hpcod; x/crypto moves to v0.59.0 (GO-2026-5932, which hpcod does not call), and CI takes the latest Go patch instead of a cached one.
v0.2.0 — 2026-11-16
One binary installs the control plane
Until now an installation was this one. The deploy scripts and the Ansible defaults carried one installation's domain, hypervisor address, proxy host and login account. The Proxmox side, the database and the Terraform variables had been set up by hand. Deploying meant copying shell scripts and a tree of roles beside the binary, and keeping all of them in step.
Now the binary is the whole of it:
hpcod setupis the installer, for administrators only: root, in a terminal on the control plane host. It is a full-screen TUI with the stages of an install down the side. It asks first where clusters are built, then signs in to Proxmox before asking anything else, with root's password (used once, never stored) or tokens you made yourself. Nodes and datastores are picked from what Proxmox has. Every field is checked as it is typed, including against the others. Then it installs, showing each step's output live, checks the result, and offers a test cluster.hpcod setup -yesruns the same steps unattended, from a site file.make deploycopies the binary anddeploy/site.envto the host and runs it there. Nothing else is sent.- The binary carries the tree. The Ansible roles, the Terraform module,
the schema and the application catalogue are embedded (
embed.go). A host always runs the roles and schema of the binary it runs. A test checks that nothing local leaks into the embedded tree. hpcod verifychecks each fact by using it: the tokens against the API, the template by asking Proxmox, the database as the portal connects, HTTPS against the installation's CA, DNS as a user's resolver would ask. It reports READY, or which step fixes what failed.hpcod selftestbuilds the smallest real cluster by the portal's own path (placement, allocation, the portal's builder). It waits for Slurm, runssrunacross the nodes, checks the portal is published, and takes the cluster down. A failed test leaves the cluster up and prints how to remove it.
Closed networks
For an installation with no public domain, DNS or certificate:
- Its own CA.
SERVE_MODE=directmakes hpcod serve HTTPS itself on 443, with a CA setup creates once. The CA is name-constrained to the installation's domains, so trusting it trusts nothing else. The portal issues its own server certificate from it in memory and renews it before it expires. Users fetch the CA athttp://<portal>/ca.crt. - Its own DNS.
SERVE_DNS=trueanswers for the portal and every cluster's name, and forwards everything else. With no forwarders it refuses, so it is never an open resolver.
A simulated Proxmox, to try it
PVE_MODE=sim runs Proxmox VE in QEMU/KVM on the control plane host itself,
on a private bridge with NAT. Setup installs Proxmox VE into a Debian 13
cloud image and fills in every Proxmox and network value. Clusters run
inside it, so the host needs nested virtualization. It is for trying hpcod,
not for tenants.
What was proven, and what was not
- End to end, on a simulated Proxmox (a control plane VM on a real
Proxmox host, with Proxmox VE 9 inside it):
- the install, from the single binary;
- the Proxmox step by the root-password path: SSH key, both tokens, the SDN zone, the content types, the cloud image;
- the head template (47 tasks, none failed);
hpcod verifyREADY, with HTTPS from its own CA and its own DNS;hpcod selftest:c001built in about ten minutes, its compute node idle in Slurm,srun hostnameanswered from it, then destroyed and released.
- On a clean Debian 13 host: the install twice in a row, with nothing changed the second time. The generated password signs in, and the TUI was driven through a terminal.
- Not yet run:
- setup against a bare-metal Proxmox host;
- the token path, which was tested only against a fake Proxmox API;
- a fresh simulated Proxmox with the last two fixes to it (resolv.conf, and a clean power-down). Both were verified on the running simulation only.
Changes an existing installation should know about
- The Proxmox token moves from
site.auto.tfvarstosecrets.auto.tfvars. Setup moves it the first time and keeps a backup. ci_userand the portal's-ssh-userno longer default to a personal account. Setup writes both fromCI_USER.ansible/group_vars/all.ymlis nowgroup_vars/all/main.yml. The site's values are in a generatedsite.ymlbeside it, and the playbooks refuse to run without it. The head template only picks them up when it is rebuilt (-rebuild-template). Until then, setup and verify say it is stale.- The catalogue now comes from inside the binary. Publishing an application
means rebuilding and deploying the binary. A site that keeps its own
catalogue creates
/etc/hpcod/apps.local. - Removed:
scripts/B10,B20andC20. Their work is the setup steps./var/lib/hpcod/scriptson an existing host is no longer used.
v0.1.0 — 2026-09-21
First tagged release. The repository holds two things: the Warewulf HPC lab
documented in sections 01–09, and HPC-on-Demand (hpcod/), a control
plane that provisions clusters of that kind on demand. This release is about
the second.
The chain that works end to end
Someone signs up on the public site, an administrator approves them, and from
there the portal takes a request for a cluster through to a running desktop
with a job on it. Every step below was exercised on hv01, not inferred:
- Sign up, and wait. New accounts land in
pending; nobody gets in until an administrator approves them. Disabling an account takes effect on the next request, not when the session happens to expire. - Ask for hardware, not for a flavour. A request names capabilities — GPU, InfiniBand, RoCE, NVMe, cores, memory — or just names the software it wants and lets the application catalogue imply the hardware.
- Placement picks a site and a node. Devices already handed to another cluster are not offered again, and neither are devices reserved for a cluster that has been allocated but not yet built. Oversubscription is a policy an operator sets from the portal, with separate CPU and memory ratios; turning it off does not lose them.
- Allocation is atomic. Cluster ids, management addresses and PCI devices come from PostgreSQL under an advisory lock, so two requests racing for the last GPU produce one cluster and one clear refusal.
- Terraform builds it. An SDN vnet for the cluster fabric, a head node and compute VMs, with the placement carried straight through from the record. Teardown removes them in an order that actually works.
- Ansible finishes it. Warewulf 4.7, OpenHPC 3, Slurm 25.11, MUNGE,
Apptainer, shared
/opt/apps, and Open OnDemand on the head node. - The cluster publishes itself. It appears at
c<NNN>.<domain>, behind the control plane, with the portal's certificate pinned at the moment it was published. - One sign-in, not two. The cluster portal trusts the control plane over mTLS, so a session that may use a cluster reaches it without another password. A session that may not gets nothing, and a hostname for a cluster that does not exist says so instead of showing the front page.
- A desktop, a terminal, and a job. XFCE over TurboVNC in a container, a
shell on the head node, and
srunfrom inside either.
Measured
- Cold provision of a tenant cluster: ~134 s.
- Node image build: ~12 min, done once and shared, not per tenant.
Known limits
These are absent by decision or by circumstance, not by oversight:
- Leases are recorded but not enforced.
store.Expiredreturns what has run out; nothing acts on it yet. No reaper. - One Proxmox node. Placement discriminates between sites and between
hosts, and the Terraform module takes a per-VM host — but
hv01is a single node, so multi-site placement is tested against fixtures rather than metal. - No central Slurm accounting. There is no
slurmdbd, so jobs carry no cluster name andsaccthistory is per-cluster. - Whole-device GPU passthrough only. MIG and vGPU are out of scope.
/homeonly. No scratch, no parallel filesystem.- Nothing bills. The allocation record is the raw material for it and is written; no code reads it for that purpose.
- The API in §20.9 is partly implemented. The portal and CLI cover what is built; the REST surface does not yet.
Operating notes
hpcod versionnow reports the release tag alongside the revision, e.g.v0.1.0 (f6ea30dc1234). A binary built outside the Makefile prints the revision alone rather than guessing.- The control plane needs
$HPCOD_DB_DSNor-db; the CLI and the service resolve it the same way. make deploy-portalbuilds for Linux and installs it on the Caddy host.