Files
werewulf/docs/20-hpc-on-demand-architecture.md
ilkerandClaude Opus 5.5 090719fc74
ci / derle (push) Successful in 30s
hpcod: clusters across the nodes of a Proxmox cluster
Discovery already saw every node of a Proxmox cluster and placement could
put a VM on any of them, but the template, the snippets and a simple SDN
zone only worked on PVE_NODE, so a cluster could land where it could not
be built.

- Setup asks which nodes clusters may use (PVE_NODES; a multi-select in
  the TUI when Proxmox has more than one). Discovery offers only those.
- Heads are placed on PVE_NODE only (head_node), where the template is, so
  no shared storage is needed; diskless compute nodes spread.
- PVE_SDN_ZONE_TYPE (auto|simple|vxlan): more than one node makes the zone
  vxlan, every node a peer, MTU 1450. A zone of the wrong type is remade
  only while no cluster has a vnet in it. Guest fabric NICs take the
  bridge MTU through virtio (mtu = 1).
- setup keeps nodes and head_node of its own site in sites.yaml, leaving
  the rest of the file as the operator wrote it.
- verify checks the nodes are online and the zone matches.

Run on the simulated Proxmox with the zone forced to vxlan: setup remade
the zone, a two-node selftest passed, both compute NICs came up at 1450,
and a DF ping of 1422 bytes passed between them while 1423 did not. Two
real Proxmox nodes have not been tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:51:35 +01:00

19 KiB
Raw Permalink Blame History

20. HPC-on-Demand — Architecture

A control plane that fronts several Proxmox clusters, picks one that satisfies a requested hardware profile (InfiniBand, GPU, NVMe, core count…), and provisions an HPC cluster on it of the kind documented in sections 01–10.

Status: built. The control plane, placement, the provisioning pipeline and the /v1/clusters endpoints are implemented in hpcod/; a cluster provisions cold from the template in ~134s. It ships as one binary that installs itself (hpcod setup, see hpcod/DEPLOY.md). Leases are recorded and shown, but nothing tears a cluster down when one expires — the reaper in 20.2 is not built. Section 20.11 records the build order that was followed.


20.1 What this has to do

    user / portal / API client
              │  "give me 8 nodes, 2 GPUs each, InfiniBand, 14 days"
              ▼
    ┌─────────────────────────────────────────────┐
    │            HPC-on-Demand control plane      │
    │                                             │
    │  inventory ──▶ placement ──▶ provisioner    │
    │  (discovery)   (filter+score)  (TF+Ansible) │
    │       ▲                            │        │
    └───────┼────────────────────────────┼────────┘
            │ Proxmox API (read)         │ Terraform + SSH
    ┌───────┴──────┬──────────────┬──────┴────────┐
    ▼              ▼              ▼               ▼
 site-a         site-b         site-c         site-d
 GPU + IB       plain eth      NVMe heavy     mixed
 (3 nodes)      (12 nodes)     (4 nodes)      (6 nodes)

Each provisioned tenant is a self-contained cluster: its own bridge, its own subnet, its own Warewulf head node, its own Slurm. That is the structure this repository already builds by hand — the control plane's job is to do it repeatedly, on the right hardware, without collisions.

20.2 Components

Component Responsibility
API server REST + auth, request validation, quota, lifecycle state machine
Inventory Periodically polls every Proxmox cluster; stores nodes, capacity, capabilities
Placement Turns a request into a chosen cluster + chosen nodes, or a clear refusal
IPAM Allocates a bridge name and subnet per tenant; the collision-avoidance authority
Provisioner Runs Terraform (VMs, vnet); the head configures itself on first boot
State store PostgreSQL: tenants, clusters, allocations, applications, projects. Terraform state stays in one directory per cluster
Reaper Enforces leases; tears clusters down on expiry. Not built: an expired lease is shown, not acted on

Deliberately not a component: a scheduler across clusters. Slurm schedules inside a tenant; this system allocates whole clusters, not jobs. Conflating the two is how these systems become unmaintainable.

20.3 Capability discovery — and its real limits

Discovery reads the Proxmox API on each cluster:

Source Yields
/nodes/{node}/status CPU model, socket/core count, total memory
/nodes Per-node free memory, CPU load, online state
/nodes/{node}/hardware/pci class, vendor, device, device_name, iommugroup
/nodes/{node}/storage Storage types (zfspool, lvmthin, nfs), free space
/cluster/status Node membership, quorum
/cluster/resources Existing VM/CT load, so we do not oversubscribe

Verified against the hv01 node in this lab:

  VGA/GPU         0000:01:00.0  NVIDIA Corporation   GM107GL [Quadro K620]
  Ethernet        0000:07:00.0  Mellanox Technologies MT27710 [ConnectX-4 Lx]
  Ethernet        0000:07:00.1  Mellanox Technologies MT27710 [ConnectX-4 Lx]
  53 IOMMU groups; GPU alone in group 47 with only its own audio function

The finding that shapes the design

PCI class is not sufficient to detect InfiniBand. The ConnectX-4 Lx above reports class 0x0200 (Ethernet), because the Lx variant is Ethernet-only. A ConnectX-6 in IB mode reports 0x0207; the same card flipped to Ethernet mode reports 0x0200. Class alone therefore produces both false negatives (IB-capable card currently in Ethernet mode) and false positives (RoCE treated as IB).

So discovery is layered, most reliable first:

  1. PCI class — dependable for GPUs (0x0300, 0x0302) and NVMe (0x0108).
  2. Curated vendor:device → capability map, shipped with the orchestrator and versioned like a hardware database. This is what distinguishes ConnectX-4 Lx (Ethernet, RoCE) from ConnectX-6 (IB-capable).
  3. A tiny discovery agent on each Proxmox node, reporting what the API cannot see: /sys/class/infiniband/* (link layer, active speed), NVIDIA driver and MIG state, NUMA topology, vfio-pci bindings.

Layers 1 and 2 need nothing installed. Layer 3 is optional but is the only way to state "InfiniBand, 200 Gb/s, active" rather than "a card that could do IB".

Recommendation: treat auto-discovery as authoritative for capacity and presence, and require layer 3 before a cluster may advertise infiniband as a schedulable capability. Advertising IB on the strength of a PCI ID alone will eventually hand someone a cluster whose fabric is not actually up.

Passthrough viability

A GPU is only offerable if its IOMMU group contains nothing else that matters. On hv01 the K620 sits in group 47 with just its own HDA function — clean. The discovery record therefore carries passthrough_safe: true|false derived from group membership, not merely "a GPU exists".

20.4 Data model

proxmox_cluster(id, name, endpoint, token_ref, state, last_seen)
pve_node(id, cluster_id, name, cores, mem_bytes, online, alloc_cores, alloc_mem)
device(id, pve_node_id, pci_id, class, vendor, device, iommu_group,
       capability, passthrough_safe)          -- capability: gpu|ib|nvme|roce
capability_map(vendor, device, capability, notes)   -- curated, versioned

tenant(id, owner, name, state, lease_expires_at, spec_json)
allocation(tenant_id, cluster_id, bridge, subnet, vlan_tag)
allocated_node(tenant_id, pve_node_id, role, vmid, cores, mem, devices[])

state on tenant is the lifecycle: requested → placed → provisioning → ready → expiring → deleting → deleted, plus failed. Every transition is written before the side effect, so a crashed provisioner is resumable rather than ambiguous.

Request

name: genomics-week12
lease: 14d
head:    { cores: 4, memory_gb: 16, disk_gb: 200 }
compute:
  count: 8
  cores: 8
  memory_gb: 64
  require:
    - capability: gpu
      count: 2
      vendor: nvidia
    - capability: infiniband
      min_speed_gbps: 100
  prefer:
    - capability: nvme
stack:
  os: rocky9
  scheduler: slurm
  identity: freeipa       # or: join-existing

require is a hard filter; prefer only affects scoring. Keeping them separate is what lets the API answer "no cluster has 100 Gb IB" instead of silently returning something slower.

20.5 Placement

Two phases, deliberately in this order:

Filter — eliminate the impossible. Cluster is healthy and quorate; every require is satisfiable on nodes that are online; free cores/memory cover the request with headroom; enough passthrough-safe devices are unallocated; a free subnet exists.

Score — rank what remains:

score =  w1 * capability_fit        // penalise waste: do not burn an IB+GPU
                                    // cluster on a request needing neither
       + w2 * free_capacity_ratio   // prefer emptier clusters
       + w3 * packing_quality       // fewer PVE nodes per tenant = less noise
       - w4 * fragmentation_cost    // avoid leaving unusable remainders

capability_fit is the one that matters and the one most such systems get wrong. Scarce hardware must be reserved for requests that need it; a plain CPU job placed on the only InfiniBand cluster is a scheduling failure even though it "fits".

Selection is committed inside a transaction that increments the allocation counters, so two concurrent requests cannot both win the last GPU.

20.5a Tenants and clusters are not the same thing

A tenant is who owns a cluster. A cluster is one provisioned installation. One tenant may hold several at once, and the difference is not cosmetic: everything that must not collide - the bridge, the subnet, the MAC prefix, the VM names, the public hostname, the Slurm cluster name - is a property of the cluster, not of its owner.

This was conflated in the first implementation. A single tenant_id drove all of them, so a tenant asking for a second cluster while the first was running would have been handed the same bridge, the same subnet and the same MAC prefix - and Warewulf matches nodes by MAC, so the second cluster would have broken the first rather than failing cleanly.

cluster_id   1-254   addressing: bridge, subnet, MACs, names, hostname
tenant_id            attribution: tags, accounting, billing

The bound of 254 is therefore on clusters that exist at the same time, not on tenants. A tenant running three clusters consumes three.

20.6 Tenant isolation

One Linux bridge and one /24 per tenant — the vmbr6 pattern from section 01, multiplied:

Tenant Bridge Subnet Head node
1 vmbrT001 10.64.1.0/24 10.64.1.1
2 vmbrT002 10.64.2.0/24 10.64.2.1
n vmbrT00n 10.64.n.0/24 10.64.n.1

Properties carried over from the working build, and the reasons they matter:

  • No IP on the Proxmox side. Warewulf must own DHCP/TFTP/PXE for its subnet; a second authority on the segment breaks PXE.
  • The head node is the tenant's router, NATing out through the management network — so tenants cannot reach each other even if they guess addresses.
  • PXE broadcasts stay inside the bridge, which is what makes running many Warewulf clusters on one hypervisor safe at all.

IPAM allocates from 10.64.0.0/10 (≈16k /24s), recorded in Postgres. The bridge is created by Terraform and torn down with the tenant; a leaked bridge is a leaked subnet, so deletion is verified rather than assumed.

That day arrived early, and the fabric is now an SDN vnet rather than a bridge stanza written over SSH. The reason was not multi-node: appending to /etc/network/interfaces on a hypervisor running other people's guests is a poor way to work, and an early version of that code wrote a comment line without its #, which would have taken the host's networking down at the next reload. Proxmox owns the lifecycle now, and the module has no provisioner blocks left.

Each cluster gets one proxmox_sdn_vnet named after it, in a zone shared by the site and referenced rather than created — several Terraform states each declaring the same zone would each believe they owned it. The vnet name is the bridge a guest attaches to, so it is c002, not vmbrC002: Proxmox caps vnet names at eight characters.

Permissions. Creating a vnet needs SDN.Allocate on the zone and can be scoped to it. Applying needs it on /sdn and cannot be scoped at all, because Proxmox applies the entire pending SDN configuration at once. Both live in a separate role holding only the three SDN privileges, rather than being added to the provisioner's role, which is bound at /. This is still less than the mechanism it replaced: editing the hypervisor's network configuration needed root SSH.

Several nodes. A simple zone gives every node its own isolated bridge of the same name: correct for one hypervisor, and wrong the moment a cluster spans two, because two nodes would get two unconnected L2 segments and PXE would not cross. When a site uses more than one node of a Proxmox cluster, setup makes the zone vxlan, with every node a peer; each vnet carries the cluster id as its VNI. VXLAN costs 50 bytes a frame, so the bridge is 1450, and the guests' NICs take the bridge's MTU through virtio (mtu = 1). Without that a guest assumes 1500 and full-sized frames are dropped at the bridge without an error.

Heads are not spread. The template lives on one node, and a clone cannot cross nodes without shared storage, so placement puts every head on that node (head_node) and spreads only the diskless compute nodes, over the nodes the site lists (nodes). Both are site settings that setup keeps in step with its answers.

The vxlan zone and the MTU path have been run end to end on one node, where the bridge is still 1450 and an oversized frame would still be dropped: a two-node cluster built, its nodes booted over PXE and ran srun -N 2, both compute NICs came up at 1450, and between them a 1450-byte frame with DF set passed while 1451 did not. Traffic between two Proxmox nodes has not been run: the lab has one.

20.7 Provisioning pipeline

Terraform for infrastructure, Ansible for configuration, with a hard boundary between them.

placement decided, cluster allocated in Postgres
   │
   ├─ tofu apply                one state directory per cluster
   │    • SDN vnet c<NNN>
   │    • head node VM  (cloned from the template; cloud-init vendor data
   │                     carries the cluster's inventory)
   │    • N compute VMs (diskless, boot order=net0, PCI passthrough if requested)
   │
   ├─ the head configures itself on first boot
   │    hpcod-firstboot (role selfconfig) runs site.yml from the copy the
   │    template carries: warewulf, node image, slurm, warewulf_nodes,
   │    apps, ondemand, verify — the install-tagged half already done
   │
   ├─ wait for the head's portal; publish it at c<NNN>.<CLUSTER_DOMAIN>
   │
   └─ state = ready

Terraform (OpenTofu) uses the bpg/proxmox provider, which covers VMs, cloud-init, PCI passthrough (hostpci) and SDN. Each cluster has its own state directory under /var/lib/hpcod/terraform/envs, so a failure touches one cluster. The control plane runs one apply per cluster at a time, which is what stops two writers racing; the state is not in Postgres.

Ansible roles are ports of the scripts in scripts/, which are already proven end to end. Most of the work runs once, when setup builds the head node template; what is specific to one cluster runs on the head itself at first boot. The control plane never runs a playbook against a tenant. Each role keeps the comments explaining why — particularly the twelve traps in section 08, every one of which will otherwise be rediscovered.

Two rules worth stating because they are usually violated:

  • Terraform never configures software; Ansible never creates infrastructure. Mixing them is what makes these pipelines impossible to re-run.
  • Both must be idempotent. Provisioning will be interrupted, and the recovery path must be "run it again", not manual repair.

MAC and VMID allocation

Warewulf matches nodes by MAC, so MACs must be deterministic and unique across clusters. Derive them from the cluster id (§20.5a) and the node index, inside the locally-administered range. VMIDs come from a per-cluster range recorded in Postgres, never from nextid, which races.

20.8 Identity

Two modes, because both are legitimately wanted:

  • Dedicated — a FreeIPA server per tenant, as built in section 04. Full isolation; costs one extra VM and ~8 minutes of provisioning.
  • Shared — tenants enrol against a central IPA with per-tenant groups and HBAC rules restricting which hosts a group may log into.

Shared is cheaper and gives users one identity across clusters; dedicated is the right answer when tenants must not see each other's user lists. The spec field identity: freeipa | join-existing selects it.

Either way the stateless-node enrolment mechanism from section 04.3 applies unchanged: per-node keytabs distributed through Warewulf overlays, because ipa-client-install cannot run on a rootfs rebuilt every boot.

20.9 API surface

POST   /v1/clusters              submit a request        → 202 + tenant id
GET    /v1/clusters/{id}         state, endpoints, progress
DELETE /v1/clusters/{id}         tear down
POST   /v1/clusters/{id}/extend  extend the lease
GET    /v1/capacity              what could be satisfied right now
POST   /v1/dry-run               placement decision without provisioning
GET    /v1/sites                 clusters, capabilities, utilisation

/v1/dry-run earns its place: it lets a user find out that no site has 100 Gb InfiniBand before committing to a fourteen-day lease, and it makes the placement logic testable in isolation.

Implemented so far: /v1/sites, /v1/capacity, /v1/dry-run, and POST/GET /v1/clusters, GET/DELETE /v1/clusters/{id}. The portal's create page takes the same path. A lease is extended from the portal's cluster page; POST /v1/clusters/{id}/extend does not exist.

20.10 Failure modes to design for now

Failure Consequence if ignored Mitigation
Provisioner dies mid-run Orphaned VMs, leaked subnet State written before side effects; resumable; reconciler sweeps orphans
Proxmox unreachable during discovery Stale inventory, bad placement last_seen timestamps; refuse placement on stale data
Two requests, one GPU Double allocation Allocation committed in a transaction
Tenant lease expires while jobs run Angry users Warn at T-24h; drain Slurm; then delete
Bridge not cleaned up Subnet exhaustion Deletion verified against Proxmox, not assumed
PCI ID map out of date New card unusable, or IB wrongly claimed Map is versioned and testable; layer-3 agent confirms
Terraform state corruption Cannot delete a cluster One state directory per cluster, one apply at a time; it is backed up with /var/lib/hpcod

20.11 Suggested build order

Each phase should end in something demonstrable, on the hardware in this lab.

  1. Inventory + discovery against hv01. Read-only, no provisioning. Proves the capability model against real PCI data.
  2. Placement + /v1/dry-run. Pure function over the inventory, heavily unit-tested with synthetic multi-site fixtures. No Proxmox writes yet.
  3. Terraform module producing the bridge, head node and compute VMs for one tenant. Verified by hand-running Ansible afterwards.
  4. Ansible roles ported from scripts/, ending in the same verification A99-verify.sh already performs.
  5. API server + lifecycle, including delete and the reconciler.
  6. Leases, quotas, multi-site, and a second Proxmox to prove placement actually discriminates.

Phases 1–2 carry nearly all the design risk and none of the destructive operations, which is why they come first.

20.12 Open questions

  • Compute node RAM. The Warewulf model unpacks the rootfs into RAM (§01.5, §08.1). Every tenant node therefore needs ~2 GB before user workload. Does the request specify usable memory or total, and who explains the difference?
  • GPU sharing. Passthrough is whole-device. Fractional GPUs need MIG or vGPU licensing — out of scope for a first version, but the data model should not assume one device equals one tenant forever.
  • Image build time. ~12 minutes per node image. Tenants with identical stacks should share a prebuilt image; the cache key is the stack spec.
  • Storage beyond /home. Real HPC wants scratch. Is that per-tenant NFS, or does it come from an existing parallel filesystem the tenant mounts?
  • Billing. Projects and the cluster_usage view now turn allocations into hours held per project. Nothing prices them.