Files
ilkerandClaude Opus 5.5 3297de5d50
ci / derle (push) Successful in 22s
Apache License 2.0
The repository had no license, so it was public but not open source:
nobody could use, change or redistribute it. LICENSE is the text from
apache.org, unmodified. NOTICE says that the IBM Plex fonts embedded in
the print handbook are under the SIL Open Font License, not Apache.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 20:27:25 +01:00

267 lines
10 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# hpcod — HPC-on-Demand control plane
Phase 1 of [the architecture](../docs/20-hpc-on-demand-architecture.md):
**discovery**. It reads the backing clouds, translates their inventory into a
provider-neutral vocabulary, and reports what could actually be scheduled.
Nothing here writes to a site.
## Why the provider abstraction comes first
The portal is meant to sit over more than Proxmox — OpenStack is expected next.
So `internal/provider` defines the interface, `internal/model` defines the
vocabulary everything above it reasons in, and Proxmox is simply the first
implementation. Adding a provider means writing one package and registering it;
nothing above `internal/provider` changes.
```
cmd/hpcod CLI
internal/model provider-neutral inventory types
internal/provider Provider interface, registry, site config
internal/provider/proxmox first implementation
internal/capability PCI identity -> scheduling vocabulary
internal/inventory catalogue loading, concurrent discovery
```
## The thing this code exists to get right
**A PCI class code cannot identify InfiniBand.**
A ConnectX-4 Lx reports class `0x0200` (Ethernet) because that variant is
Ethernet-only. A ConnectX-6 reports `0x0207` in IB mode and `0x0200` in
Ethernet mode. So class-based detection produces false negatives *and* false
positives, and a portal that trusts it will eventually hand a tenant a cluster
whose fabric is not up.
Detection is therefore layered, and every claim carries how it was made:
| Confidence | Source | Trusted to schedule on |
|---|---|---|
| `class` | PCI class code | GPU, NVMe — yes. Fabrics — no. |
| `map` | curated `vendor:device` table | GPU, NVMe — yes. Fabrics — no. |
| `agent` | live check on the node | yes |
`model.Capability.Trusted` enforces it: a fabric capability is never
schedulable until an agent has confirmed the link. Unconfirmed devices are
still reported, because that difference is what an operator needs to fix.
Presence is not enough either. A GPU is offerable only if its IOMMU group
holds nothing else that matters, so `Passthrough` is derived from group
membership — sibling functions of the same card are fine, an unrelated device
in the group is not.
## Use
```bash
cp config/sites.yaml.example config/sites.yaml # then edit
export HV01_TOKEN='hpcod@pve!discovery=<uuid>'
go run ./cmd/hpcod sites # configured, reachable?
go run ./cmd/hpcod discover # full inventory
go run ./cmd/hpcod capacity # what could be scheduled now
go run ./cmd/hpcod capmap # the curated table, and why each row exists
go run ./cmd/hpcod discover -json | jq .
```
Secrets are never written in the catalogue: any setting may be `env:NAME` and
is read from the environment at startup.
The Proxmox token needs read access only — create it against `PVEAuditor`:
```bash
pveum user add hpcod@pve --comment "HPC-on-Demand discovery"
pveum acl modify / --users hpcod@pve --roles PVEAuditor
pveum user token add hpcod@pve discovery --privsep 0
```
## Sample output
```
=== hv01 (Lab cluster, proxmox)
node hv01 (online) 28 cores (0 free), 504 GiB RAM (52 GiB free)
cpu: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
gpu 0000:01:00.0 SCHEDULABLE via class GM107GL [Quadro K620]
roce 0000:07:00.0 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
roce 0000:07:00.1 unconfirmed via map MT27710 Family [ConnectX-4 Lx]
```
The Mellanox cards are reported as `roce`, not `infiniband`, and withheld from
the schedulable set — which is the whole point.
## Capacity accounting
Free capacity is total minus what running guests have already been allocated,
not raw hardware totals. Physical cores are the schedulable total by default
(`cpu_overcommit: "1.0"`): an HPC tenant handed oversubscribed cores simply
runs slowly. Raise it per site only where that is acceptable.
## Phase 2 — placement
`placement.Plan(sites, request, weights)` is a **pure function**: no I/O, no
cloud. That is what makes the decision reproducible, unit-testable against
synthetic multi-site fixtures, and usable as a dry-run endpoint later.
```bash
hpcod plan -f examples/gpu-request.yaml # against live sites
hpcod plan -f examples/gpu-request.yaml -inventory snap.json # against a snapshot
hpcod discover -json > snap.json # take a snapshot
```
Two phases, in this order. **Filter** eliminates the impossible and records why
per site; **score** ranks the survivors.
```
score = 3.0 * capability_fit // penalise consuming scarce hardware the
// request did not ask for
+ 1.0 * free_capacity // prefer emptier sites
+ 0.8 * packing // fewer hypervisors per tenant
- 1.2 * fragmentation // avoid unusable remainders
```
`capability_fit` dominates deliberately. Scarce hardware must be kept for
requests that need it: **a plain CPU job placed on the only InfiniBand site is
a scheduling failure even though it fits.**
### What the refusals look like
The point of recording reasons is that "no capacity" is useless while this is
actionable:
```
all candidates:
site-a feasible, score 4.40
site-b rejected
- gpu: none present (need 4)
site-c rejected
- infiniband: 1 present but unconfirmed - the fabric has not been
verified by a node agent, so it is not schedulable (need 4, confirmed 0)
site-d rejected
- site is unreachable: dial tcp 10.0.0.40:8006: no route to host
```
Three different problems, three different fixes: buy hardware, run the agent,
fix the network.
### Memory is not what the user typed
A compute node unpacks its root filesystem into RAM, so `memory_gb` in a
request is **usable** memory and the portal adds `RootfsReserveGB` on top. A
tenant that asked for 32 GiB and was allocated exactly 32 GiB would OOM under
its own job.
## Installing it
On a Debian 13 or Ubuntu 24.04 host:
```bash
V=v0.2.2 A=linux-amd64 # A=linux-arm64 on an arm64 host
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/hpcod-$V-$A
curl -fLO https://gitea.mynodes.xyz/ilker/werewulf/releases/download/$V/SHA256SUMS
sha256sum -c --ignore-missing SHA256SUMS
install -m 0755 hpcod-$V-$A hpcod
sudo ./hpcod setup
```
That runs the installer, a full-screen TUI for administrators. It installs
the binary as `/usr/local/bin/hpcod`, so afterwards:
```bash
sudo hpcod verify # can this installation build a cluster?
sudo hpcod selftest # build a real cluster, run a job on it, take it down
```
The binary carries the Ansible roles, the Terraform module, the schema and
the application catalogue; setup installs everything else. It installs
against a Proxmox VE you have, or against one it simulates in QEMU on the
same host. See **[DEPLOY.md](DEPLOY.md)**.
## The portal
`hpcod serve` is the portal; setup installs it as the `hpcod-portal`
service. It serves:
- sites and their devices;
- the plan and create form;
- the cluster list, with build logs as they happen;
- projects and usage;
- the curated device map;
- account approval;
- each cluster's own portal at `c<NNN>.<domain>`;
- a JSON API (`/v1/sites`, `/v1/capacity`, `/v1/dry-run`, `/v1/clusters`).
It listens on `127.0.0.1:9880`, and is reached in one of two ways:
- **Behind your reverse proxy.** The proxy is then the only thing that can
reach it, and it refuses to bind off loopback without a user file.
- **On a closed network, serving HTTPS on 443 itself.** Its certificate
comes from the installation's own name-constrained CA. It can also answer
DNS for its names.
Discovery runs on a timer rather than per request. Polling every backing
cloud on each page load would hammer their APIs and make the portal as slow
as the slowest site, so pages show how old the inventory is instead.
## Phase 3 — Terraform
`terraform/modules/tenant` builds one tenant: an isolated bridge, a head node
with a management and a cluster NIC, and N diskless compute nodes that PXE boot
from it. Applied and destroyed cleanly against a real Proxmox.
```bash
cd terraform/envs/lab
cp terraform.tfvars.example terraform.tfvars # then edit
tofu init && tofu plan && tofu apply
```
MACs are **derived, not generated**: `BC:24:11:E1:<tenant>:<index>`. Warewulf
matches nodes by MAC, so a MAC that changed on re-apply would orphan the node's
registration and it would never boot again.
The `E1` prefix is not arbitrary. The hand-built cluster on this hypervisor
already uses `BC:24:11:E0:` with `00` and `0A` in the next byte, so a tenant
with id 10 would have produced a **duplicate MAC on the same L2 segment** and
broken the running cluster. Tenants get their own prefix so the two estates
cannot collide:
| | hand-built lab | hpcod tenants |
|---|---|---|
| bridge | `vmbr6` | `vmbrT001`–`vmbrT254` |
| subnet | `10.10.10.0/24` | `10.64.<tenant>.0/24` |
| VMID | 300–304 | 9001+ |
| MAC | `BC:24:11:E0:*` | `BC:24:11:E1:*` |
### The bridge is not the provider's resource
`proxmox_network_linux_bridge` writes the stanza and then reads the live
interface back before anything has applied it, so creation fails. Proxmox
applies pending network changes with `ifreload -a`, which reloads **every**
interface — on a hypervisor carrying other people's guests that is a bad
trade, and on any host with one unrelated broken stanza it returns non-zero
and takes the new bridge down with it.
So the bridge is managed with an explicit `remote-exec` that appends the
stanza, runs `ifquery -a` to prove the file still parses, and brings up **only
the new interface** with `ifup`. It is idempotent, so an interrupted run
recovers by re-running, and the destroy path verifies the interface is really
gone — a leaked bridge is a leaked subnet.
### Two things this needs from Proxmox
- The API token's role needs **`SDN.Use`**, or attaching a VM to the new
bridge fails with a 403 that does not mention SDN anywhere obvious.
- The cloud image must live in a datastore with the **`import`** content type
(`/var/lib/vz/import/`, referenced as `local:import/...`). An image sitting
in the ISO directory is rejected with *"unable to parse directory volume
name"*.
## Not built yet
Phase 4 onward: Ansible roles ported from `scripts/`, the API server, leases
and the reconciler. See
[docs/20 §11](../docs/20-hpc-on-demand-architecture.md).
## License
[Apache License 2.0](../LICENSE). The fonts embedded in the print handbook
are under the SIL Open Font License; see [NOTICE](../NOTICE).