Files
werewulf/hpcod
ilkerandClaude Opus 5.5 25233f5de6
ci / derle (push) Successful in 2m10s
hpcod: Go 1.27.4 and x/crypto v0.59.0, for govulncheck
CI has failed at govulncheck since this morning on every push, the
rewritten tags included: GO-2026-6859, an HTTP/1 server crash in net/http,
is fixed in go1.27.4 and the runner still had 1.27.3. go.mod now names
toolchain go1.27.4, which the go command switches to by itself, and
setup-go checks for the latest patch instead of taking the cached one.

x/crypto goes to v0.59.0 for GO-2026-5932, which the code does not call.
govulncheck now finds nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 18:29:35 +01:00
..

hpcod — HPC-on-Demand control plane

Phase 1 of the architecture: discovery. It reads the backing clouds, translates their inventory into a provider-neutral vocabulary, and reports what could actually be scheduled.

Nothing here writes to a site.

Why the provider abstraction comes first

The portal is meant to sit over more than Proxmox — OpenStack is expected next. So internal/provider defines the interface, internal/model defines the vocabulary everything above it reasons in, and Proxmox is simply the first implementation. Adding a provider means writing one package and registering it; nothing above internal/provider changes.

cmd/hpcod                CLI
internal/model           provider-neutral inventory types
internal/provider        Provider interface, registry, site config
internal/provider/proxmox   first implementation
internal/capability      PCI identity -> scheduling vocabulary
internal/inventory       catalogue loading, concurrent discovery

The thing this code exists to get right

A PCI class code cannot identify InfiniBand.

A ConnectX-4 Lx reports class 0x0200 (Ethernet) because that variant is Ethernet-only. A ConnectX-6 reports 0x0207 in IB mode and 0x0200 in Ethernet mode. So class-based detection produces false negatives and false positives, and a portal that trusts it will eventually hand a tenant a cluster whose fabric is not up.

Detection is therefore layered, and every claim carries how it was made:

Confidence Source Trusted to schedule on
class PCI class code GPU, NVMe — yes. Fabrics — no.
map curated vendor:device table GPU, NVMe — yes. Fabrics — no.
agent live check on the node yes

model.Capability.Trusted enforces it: a fabric capability is never schedulable until an agent has confirmed the link. Unconfirmed devices are still reported, because that difference is what an operator needs to fix.

Presence is not enough either. A GPU is offerable only if its IOMMU group holds nothing else that matters, so Passthrough is derived from group membership — sibling functions of the same card are fine, an unrelated device in the group is not.

Use

cp config/sites.yaml.example config/sites.yaml   # then edit
export HV01_TOKEN='hpcod@pve!discovery=<uuid>'

go run ./cmd/hpcod sites       # configured, reachable?
go run ./cmd/hpcod discover    # full inventory
go run ./cmd/hpcod capacity    # what could be scheduled now
go run ./cmd/hpcod capmap      # the curated table, and why each row exists
go run ./cmd/hpcod discover -json | jq .

Secrets are never written in the catalogue: any setting may be env:NAME and is read from the environment at startup.

The Proxmox token needs read access only — create it against PVEAuditor:

pveum user add hpcod@pve --comment "HPC-on-Demand discovery"
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0

Sample output

=== hv01 (Lab cluster, proxmox)
  node hv01 (online)  28 cores (0 free), 504 GiB RAM (52 GiB free)
       cpu: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
       gpu         0000:01:00.0 SCHEDULABLE via class  GM107GL [Quadro K620]
       roce        0000:07:00.0 unconfirmed via map    MT27710 Family [ConnectX-4 Lx]
       roce        0000:07:00.1 unconfirmed via map    MT27710 Family [ConnectX-4 Lx]

The Mellanox cards are reported as roce, not infiniband, and withheld from the schedulable set — which is the whole point.

Capacity accounting

Free capacity is total minus what running guests have already been allocated, not raw hardware totals. Physical cores are the schedulable total by default (cpu_overcommit: "1.0"): an HPC tenant handed oversubscribed cores simply runs slowly. Raise it per site only where that is acceptable.

Phase 2 — placement

placement.Plan(sites, request, weights) is a pure function: no I/O, no cloud. That is what makes the decision reproducible, unit-testable against synthetic multi-site fixtures, and usable as a dry-run endpoint later.

hpcod plan -f examples/gpu-request.yaml                       # against live sites
hpcod plan -f examples/gpu-request.yaml -inventory snap.json  # against a snapshot
hpcod discover -json > snap.json                              # take a snapshot

Two phases, in this order. Filter eliminates the impossible and records why per site; score ranks the survivors.

score = 3.0 * capability_fit    // penalise consuming scarce hardware the
                                //   request did not ask for
      + 1.0 * free_capacity     // prefer emptier sites
      + 0.8 * packing           // fewer hypervisors per tenant
      - 1.2 * fragmentation     // avoid unusable remainders

capability_fit dominates deliberately. Scarce hardware must be kept for requests that need it: a plain CPU job placed on the only InfiniBand site is a scheduling failure even though it fits.

What the refusals look like

The point of recording reasons is that "no capacity" is useless while this is actionable:

all candidates:
  site-a       feasible, score 4.40
  site-b       rejected
       - gpu: none present (need 4)
  site-c       rejected
       - infiniband: 1 present but unconfirmed - the fabric has not been
         verified by a node agent, so it is not schedulable (need 4, confirmed 0)
  site-d       rejected
       - site is unreachable: dial tcp 10.0.0.40:8006: no route to host

Three different problems, three different fixes: buy hardware, run the agent, fix the network.

Memory is not what the user typed

A compute node unpacks its root filesystem into RAM, so memory_gb in a request is usable memory and the portal adds RootfsReserveGB on top. A tenant that asked for 32 GiB and was allocated exactly 32 GiB would OOM under its own job.

The portal

hpcod serve exposes phases 1 and 2 over HTTP at https://hpc.example.org — sites and their devices, a dry-run form, and the curated device map, plus a JSON API (/v1/sites, /v1/capacity, /v1/dry-run).

It runs on the Caddy host bound to 127.0.0.1:9880, so the reverse proxy is the only thing that can reach it, and it refuses to bind off loopback without a user file. Discovery runs on a timer rather than per request: polling every backing cloud on each page load would hammer their APIs and make the portal as slow as the slowest site, so pages show how old the inventory is instead.

Everything it serves is read-only. It provisions nothing.

To install it against your own Proxmox, copy the binary to a Debian or Ubuntu host and run sudo hpcod setup. See DEPLOY.md.

Phase 3 — Terraform

terraform/modules/tenant builds one tenant: an isolated bridge, a head node with a management and a cluster NIC, and N diskless compute nodes that PXE boot from it. Applied and destroyed cleanly against a real Proxmox.

cd terraform/envs/lab
cp terraform.tfvars.example terraform.tfvars   # then edit
tofu init && tofu plan && tofu apply

MACs are derived, not generated: BC:24:11:E1:<tenant>:<index>. Warewulf matches nodes by MAC, so a MAC that changed on re-apply would orphan the node's registration and it would never boot again.

The E1 prefix is not arbitrary. The hand-built cluster on this hypervisor already uses BC:24:11:E0: with 00 and 0A in the next byte, so a tenant with id 10 would have produced a duplicate MAC on the same L2 segment and broken the running cluster. Tenants get their own prefix so the two estates cannot collide:

hand-built lab hpcod tenants
bridge vmbr6 vmbrT001–vmbrT254
subnet 10.10.10.0/24 10.64.<tenant>.0/24
VMID 300–304 9001+
MAC BC:24:11:E0:* BC:24:11:E1:*

The bridge is not the provider's resource

proxmox_network_linux_bridge writes the stanza and then reads the live interface back before anything has applied it, so creation fails. Proxmox applies pending network changes with ifreload -a, which reloads every interface — on a hypervisor carrying other people's guests that is a bad trade, and on any host with one unrelated broken stanza it returns non-zero and takes the new bridge down with it.

So the bridge is managed with an explicit remote-exec that appends the stanza, runs ifquery -a to prove the file still parses, and brings up only the new interface with ifup. It is idempotent, so an interrupted run recovers by re-running, and the destroy path verifies the interface is really gone — a leaked bridge is a leaked subnet.

Two things this needs from Proxmox

  • The API token's role needs SDN.Use, or attaching a VM to the new bridge fails with a 403 that does not mention SDN anywhere obvious.
  • The cloud image must live in a datastore with the import content type (/var/lib/vz/import/, referenced as local:import/...). An image sitting in the ISO directory is rejected with "unable to parse directory volume name".

Not built yet

Phase 4 onward: Ansible roles ported from scripts/, the API server, leases and the reconciler. See docs/20 §11.