Discovery already saw every node of a Proxmox cluster and placement could put a VM on any of them, but the template, the snippets and a simple SDN zone only worked on PVE_NODE, so a cluster could land where it could not be built. - Setup asks which nodes clusters may use (PVE_NODES; a multi-select in the TUI when Proxmox has more than one). Discovery offers only those. - Heads are placed on PVE_NODE only (head_node), where the template is, so no shared storage is needed; diskless compute nodes spread. - PVE_SDN_ZONE_TYPE (auto|simple|vxlan): more than one node makes the zone vxlan, every node a peer, MTU 1450. A zone of the wrong type is remade only while no cluster has a vnet in it. Guest fabric NICs take the bridge MTU through virtio (mtu = 1). - setup keeps nodes and head_node of its own site in sites.yaml, leaving the rest of the file as the operator wrote it. - verify checks the nodes are online and the zone matches. Run on the simulated Proxmox with the zone forced to vxlan: setup remade the zone, a two-node selftest passed, both compute NICs came up at 1450, and a DF ping of 1422 bytes passed between them while 1423 did not. Two real Proxmox nodes have not been tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
19 KiB
20. HPC-on-Demand — Architecture
A control plane that fronts several Proxmox clusters, picks one that satisfies a requested hardware profile (InfiniBand, GPU, NVMe, core count…), and provisions an HPC cluster on it of the kind documented in sections 01–10.
Status: built. The control plane, placement, the provisioning pipeline
and the /v1/clusters endpoints are implemented in hpcod/; a
cluster provisions cold from the template in ~134s. It ships as one binary
that installs itself (hpcod setup, see hpcod/DEPLOY.md).
Leases are recorded and shown, but nothing tears a cluster down when one
expires — the reaper in 20.2 is not built. Section 20.11 records the build
order that was followed.
20.1 What this has to do
user / portal / API client
│ "give me 8 nodes, 2 GPUs each, InfiniBand, 14 days"
▼
┌─────────────────────────────────────────────┐
│ HPC-on-Demand control plane │
│ │
│ inventory ──▶ placement ──▶ provisioner │
│ (discovery) (filter+score) (TF+Ansible) │
│ ▲ │ │
└───────┼────────────────────────────┼────────┘
│ Proxmox API (read) │ Terraform + SSH
┌───────┴──────┬──────────────┬──────┴────────┐
▼ ▼ ▼ ▼
site-a site-b site-c site-d
GPU + IB plain eth NVMe heavy mixed
(3 nodes) (12 nodes) (4 nodes) (6 nodes)
Each provisioned tenant is a self-contained cluster: its own bridge, its own subnet, its own Warewulf head node, its own Slurm. That is the structure this repository already builds by hand — the control plane's job is to do it repeatedly, on the right hardware, without collisions.
20.2 Components
| Component | Responsibility |
|---|---|
| API server | REST + auth, request validation, quota, lifecycle state machine |
| Inventory | Periodically polls every Proxmox cluster; stores nodes, capacity, capabilities |
| Placement | Turns a request into a chosen cluster + chosen nodes, or a clear refusal |
| IPAM | Allocates a bridge name and subnet per tenant; the collision-avoidance authority |
| Provisioner | Runs Terraform (VMs, vnet); the head configures itself on first boot |
| State store | PostgreSQL: tenants, clusters, allocations, applications, projects. Terraform state stays in one directory per cluster |
| Reaper | Enforces leases; tears clusters down on expiry. Not built: an expired lease is shown, not acted on |
Deliberately not a component: a scheduler across clusters. Slurm schedules inside a tenant; this system allocates whole clusters, not jobs. Conflating the two is how these systems become unmaintainable.
20.3 Capability discovery — and its real limits
Discovery reads the Proxmox API on each cluster:
| Source | Yields |
|---|---|
/nodes/{node}/status |
CPU model, socket/core count, total memory |
/nodes |
Per-node free memory, CPU load, online state |
/nodes/{node}/hardware/pci |
class, vendor, device, device_name, iommugroup |
/nodes/{node}/storage |
Storage types (zfspool, lvmthin, nfs), free space |
/cluster/status |
Node membership, quorum |
/cluster/resources |
Existing VM/CT load, so we do not oversubscribe |
Verified against the hv01 node in this lab:
VGA/GPU 0000:01:00.0 NVIDIA Corporation GM107GL [Quadro K620]
Ethernet 0000:07:00.0 Mellanox Technologies MT27710 [ConnectX-4 Lx]
Ethernet 0000:07:00.1 Mellanox Technologies MT27710 [ConnectX-4 Lx]
53 IOMMU groups; GPU alone in group 47 with only its own audio function
The finding that shapes the design
PCI class is not sufficient to detect InfiniBand. The ConnectX-4 Lx above
reports class 0x0200 (Ethernet), because the Lx variant is Ethernet-only.
A ConnectX-6 in IB mode reports 0x0207; the same card flipped to Ethernet mode
reports 0x0200. Class alone therefore produces both false negatives (IB-capable
card currently in Ethernet mode) and false positives (RoCE treated as IB).
So discovery is layered, most reliable first:
- PCI class — dependable for GPUs (
0x0300,0x0302) and NVMe (0x0108). - Curated
vendor:device→ capability map, shipped with the orchestrator and versioned like a hardware database. This is what distinguishes ConnectX-4 Lx (Ethernet, RoCE) from ConnectX-6 (IB-capable). - A tiny discovery agent on each Proxmox node, reporting what the API
cannot see:
/sys/class/infiniband/*(link layer, active speed), NVIDIA driver and MIG state, NUMA topology,vfio-pcibindings.
Layers 1 and 2 need nothing installed. Layer 3 is optional but is the only way to state "InfiniBand, 200 Gb/s, active" rather than "a card that could do IB".
Recommendation: treat auto-discovery as authoritative for capacity and
presence, and require layer 3 before a cluster may advertise infiniband as a
schedulable capability. Advertising IB on the strength of a PCI ID alone will
eventually hand someone a cluster whose fabric is not actually up.
Passthrough viability
A GPU is only offerable if its IOMMU group contains nothing else that matters.
On hv01 the K620 sits in group 47 with just its own HDA function — clean. The
discovery record therefore carries passthrough_safe: true|false derived from
group membership, not merely "a GPU exists".
20.4 Data model
proxmox_cluster(id, name, endpoint, token_ref, state, last_seen)
pve_node(id, cluster_id, name, cores, mem_bytes, online, alloc_cores, alloc_mem)
device(id, pve_node_id, pci_id, class, vendor, device, iommu_group,
capability, passthrough_safe) -- capability: gpu|ib|nvme|roce
capability_map(vendor, device, capability, notes) -- curated, versioned
tenant(id, owner, name, state, lease_expires_at, spec_json)
allocation(tenant_id, cluster_id, bridge, subnet, vlan_tag)
allocated_node(tenant_id, pve_node_id, role, vmid, cores, mem, devices[])
state on tenant is the lifecycle: requested → placed → provisioning → ready → expiring → deleting → deleted, plus failed. Every transition is
written before the side effect, so a crashed provisioner is resumable rather
than ambiguous.
Request
name: genomics-week12
lease: 14d
head: { cores: 4, memory_gb: 16, disk_gb: 200 }
compute:
count: 8
cores: 8
memory_gb: 64
require:
- capability: gpu
count: 2
vendor: nvidia
- capability: infiniband
min_speed_gbps: 100
prefer:
- capability: nvme
stack:
os: rocky9
scheduler: slurm
identity: freeipa # or: join-existing
require is a hard filter; prefer only affects scoring. Keeping them
separate is what lets the API answer "no cluster has 100 Gb IB" instead of
silently returning something slower.
20.5 Placement
Two phases, deliberately in this order:
Filter — eliminate the impossible. Cluster is healthy and quorate; every
require is satisfiable on nodes that are online; free cores/memory cover the
request with headroom; enough passthrough-safe devices are unallocated; a free
subnet exists.
Score — rank what remains:
score = w1 * capability_fit // penalise waste: do not burn an IB+GPU
// cluster on a request needing neither
+ w2 * free_capacity_ratio // prefer emptier clusters
+ w3 * packing_quality // fewer PVE nodes per tenant = less noise
- w4 * fragmentation_cost // avoid leaving unusable remainders
capability_fit is the one that matters and the one most such systems get
wrong. Scarce hardware must be reserved for requests that need it; a plain
CPU job placed on the only InfiniBand cluster is a scheduling failure even
though it "fits".
Selection is committed inside a transaction that increments the allocation counters, so two concurrent requests cannot both win the last GPU.
20.5a Tenants and clusters are not the same thing
A tenant is who owns a cluster. A cluster is one provisioned installation. One tenant may hold several at once, and the difference is not cosmetic: everything that must not collide - the bridge, the subnet, the MAC prefix, the VM names, the public hostname, the Slurm cluster name - is a property of the cluster, not of its owner.
This was conflated in the first implementation. A single tenant_id drove
all of them, so a tenant asking for a second cluster while the first was
running would have been handed the same bridge, the same subnet and the same
MAC prefix - and Warewulf matches nodes by MAC, so the second cluster would
have broken the first rather than failing cleanly.
cluster_id 1-254 addressing: bridge, subnet, MACs, names, hostname
tenant_id attribution: tags, accounting, billing
The bound of 254 is therefore on clusters that exist at the same time, not on tenants. A tenant running three clusters consumes three.
20.6 Tenant isolation
One Linux bridge and one /24 per tenant — the vmbr6 pattern from section 01,
multiplied:
| Tenant | Bridge | Subnet | Head node |
|---|---|---|---|
| 1 | vmbrT001 |
10.64.1.0/24 | 10.64.1.1 |
| 2 | vmbrT002 |
10.64.2.0/24 | 10.64.2.1 |
| n | vmbrT00n |
10.64.n.0/24 | 10.64.n.1 |
Properties carried over from the working build, and the reasons they matter:
- No IP on the Proxmox side. Warewulf must own DHCP/TFTP/PXE for its subnet; a second authority on the segment breaks PXE.
- The head node is the tenant's router, NATing out through the management network — so tenants cannot reach each other even if they guess addresses.
- PXE broadcasts stay inside the bridge, which is what makes running many Warewulf clusters on one hypervisor safe at all.
IPAM allocates from 10.64.0.0/10 (≈16k /24s), recorded in Postgres. The
bridge is created by Terraform and torn down with the tenant; a leaked bridge
is a leaked subnet, so deletion is verified rather than assumed.
That day arrived early, and the fabric is now an SDN vnet rather than a
bridge stanza written over SSH. The reason was not multi-node: appending to
/etc/network/interfaces on a hypervisor running other people's guests is a
poor way to work, and an early version of that code wrote a comment line
without its #, which would have taken the host's networking down at the
next reload. Proxmox owns the lifecycle now, and the module has no
provisioner blocks left.
Each cluster gets one proxmox_sdn_vnet named after it, in a zone shared by
the site and referenced rather than created — several Terraform states each
declaring the same zone would each believe they owned it. The vnet name is
the bridge a guest attaches to, so it is c002, not vmbrC002: Proxmox caps
vnet names at eight characters.
Permissions. Creating a vnet needs SDN.Allocate on the zone and can be
scoped to it. Applying needs it on /sdn and cannot be scoped at all,
because Proxmox applies the entire pending SDN configuration at once. Both
live in a separate role holding only the three SDN privileges, rather than
being added to the provisioner's role, which is bound at /. This is still
less than the mechanism it replaced: editing the hypervisor's network
configuration needed root SSH.
Several nodes. A simple zone gives every node its own isolated
bridge of the same name: correct for one hypervisor, and wrong the moment a
cluster spans two, because two nodes would get two unconnected L2 segments
and PXE would not cross. When a site uses more than one node of a Proxmox
cluster, setup makes the zone vxlan, with every node a peer; each vnet
carries the cluster id as its VNI. VXLAN costs 50 bytes a frame, so the
bridge is 1450, and the guests' NICs take the bridge's MTU through virtio
(mtu = 1). Without that a guest assumes 1500 and full-sized frames are
dropped at the bridge without an error.
Heads are not spread. The template lives on one node, and a clone cannot
cross nodes without shared storage, so placement puts every head on that
node (head_node) and spreads only the diskless compute nodes, over the
nodes the site lists (nodes). Both are site settings that setup keeps in
step with its answers.
The vxlan zone and the MTU path have been run end to end on one node,
where the bridge is still 1450 and an oversized frame would still be
dropped: a two-node cluster built, its nodes booted over PXE and ran
srun -N 2, both compute NICs came up at 1450, and between them a
1450-byte frame with DF set passed while 1451 did not. Traffic between two
Proxmox nodes has not been run: the lab has one.
20.7 Provisioning pipeline
Terraform for infrastructure, Ansible for configuration, with a hard boundary between them.
placement decided, cluster allocated in Postgres
│
├─ tofu apply one state directory per cluster
│ • SDN vnet c<NNN>
│ • head node VM (cloned from the template; cloud-init vendor data
│ carries the cluster's inventory)
│ • N compute VMs (diskless, boot order=net0, PCI passthrough if requested)
│
├─ the head configures itself on first boot
│ hpcod-firstboot (role selfconfig) runs site.yml from the copy the
│ template carries: warewulf, node image, slurm, warewulf_nodes,
│ apps, ondemand, verify — the install-tagged half already done
│
├─ wait for the head's portal; publish it at c<NNN>.<CLUSTER_DOMAIN>
│
└─ state = ready
Terraform (OpenTofu) uses the bpg/proxmox provider, which covers VMs,
cloud-init, PCI passthrough (hostpci) and SDN. Each cluster has its own
state directory under /var/lib/hpcod/terraform/envs, so a failure touches
one cluster. The control plane runs one apply per cluster at a time, which
is what stops two writers racing; the state is not in Postgres.
Ansible roles are ports of the scripts in scripts/, which are already
proven end to end. Most of the work runs once, when setup builds the head
node template; what is specific to one cluster runs on the head itself at
first boot. The control plane never runs a playbook against a tenant. Each
role keeps the comments explaining why — particularly the twelve traps in
section 08, every one of which will otherwise be rediscovered.
Two rules worth stating because they are usually violated:
- Terraform never configures software; Ansible never creates infrastructure. Mixing them is what makes these pipelines impossible to re-run.
- Both must be idempotent. Provisioning will be interrupted, and the recovery path must be "run it again", not manual repair.
MAC and VMID allocation
Warewulf matches nodes by MAC, so MACs must be deterministic and unique across
clusters. Derive them from the cluster id (§20.5a) and the node index, inside
the locally-administered range. VMIDs come from a per-cluster range recorded
in Postgres, never from nextid, which races.
20.8 Identity
Two modes, because both are legitimately wanted:
- Dedicated — a FreeIPA server per tenant, as built in section 04. Full isolation; costs one extra VM and ~8 minutes of provisioning.
- Shared — tenants enrol against a central IPA with per-tenant groups and HBAC rules restricting which hosts a group may log into.
Shared is cheaper and gives users one identity across clusters; dedicated is
the right answer when tenants must not see each other's user lists. The spec
field identity: freeipa | join-existing selects it.
Either way the stateless-node enrolment mechanism from section 04.3 applies
unchanged: per-node keytabs distributed through Warewulf overlays, because
ipa-client-install cannot run on a rootfs rebuilt every boot.
20.9 API surface
POST /v1/clusters submit a request → 202 + tenant id
GET /v1/clusters/{id} state, endpoints, progress
DELETE /v1/clusters/{id} tear down
POST /v1/clusters/{id}/extend extend the lease
GET /v1/capacity what could be satisfied right now
POST /v1/dry-run placement decision without provisioning
GET /v1/sites clusters, capabilities, utilisation
/v1/dry-run earns its place: it lets a user find out that no site has 100 Gb
InfiniBand before committing to a fourteen-day lease, and it makes the
placement logic testable in isolation.
Implemented so far: /v1/sites, /v1/capacity, /v1/dry-run, and
POST/GET /v1/clusters, GET/DELETE /v1/clusters/{id}. The portal's
create page takes the same path. A lease is extended from the portal's
cluster page; POST /v1/clusters/{id}/extend does not exist.
20.10 Failure modes to design for now
| Failure | Consequence if ignored | Mitigation |
|---|---|---|
| Provisioner dies mid-run | Orphaned VMs, leaked subnet | State written before side effects; resumable; reconciler sweeps orphans |
| Proxmox unreachable during discovery | Stale inventory, bad placement | last_seen timestamps; refuse placement on stale data |
| Two requests, one GPU | Double allocation | Allocation committed in a transaction |
| Tenant lease expires while jobs run | Angry users | Warn at T-24h; drain Slurm; then delete |
| Bridge not cleaned up | Subnet exhaustion | Deletion verified against Proxmox, not assumed |
| PCI ID map out of date | New card unusable, or IB wrongly claimed | Map is versioned and testable; layer-3 agent confirms |
| Terraform state corruption | Cannot delete a cluster | One state directory per cluster, one apply at a time; it is backed up with /var/lib/hpcod |
20.11 Suggested build order
Each phase should end in something demonstrable, on the hardware in this lab.
- Inventory + discovery against hv01. Read-only, no provisioning. Proves the capability model against real PCI data.
- Placement +
/v1/dry-run. Pure function over the inventory, heavily unit-tested with synthetic multi-site fixtures. No Proxmox writes yet. - Terraform module producing the bridge, head node and compute VMs for one tenant. Verified by hand-running Ansible afterwards.
- Ansible roles ported from
scripts/, ending in the same verificationA99-verify.shalready performs. - API server + lifecycle, including delete and the reconciler.
- Leases, quotas, multi-site, and a second Proxmox to prove placement actually discriminates.
Phases 1–2 carry nearly all the design risk and none of the destructive operations, which is why they come first.
20.12 Open questions
- Compute node RAM. The Warewulf model unpacks the rootfs into RAM (§01.5, §08.1). Every tenant node therefore needs ~2 GB before user workload. Does the request specify usable memory or total, and who explains the difference?
- GPU sharing. Passthrough is whole-device. Fractional GPUs need MIG or vGPU licensing — out of scope for a first version, but the data model should not assume one device equals one tenant forever.
- Image build time. ~12 minutes per node image. Tenants with identical stacks should share a prebuilt image; the cache key is the stack spec.
- Storage beyond
/home. Real HPC wants scratch. Is that per-tenant NFS, or does it come from an existing parallel filesystem the tenant mounts? - Billing. Projects and the
cluster_usageview now turn allocations into hours held per project. Nothing prices them.