Files
werewulf/hpcod/internal/placement
ilkerandClaude Opus 5 39700254fa hpcod: stop offering hardware that is already spoken for
Placement knew how to pick the node with the GPU. It did not know which
GPUs were already taken, in two separate ways.

Device.AllocatedTo was read in three places and written in none, so every
discovery reported every card as free. Two clusters provisioned in sequence
would both be assigned the same GPU; the second VM fails to start, hours
after the portal promised the hardware. Ownership now comes from the
hypervisor, which also accounts for guests created outside hpcod. Two
details that are easy to get wrong and are covered by tests: a stopped
guest still owns its card, because the assignment lives in the config and
not in the running process; and ownership is per slot, not per function, so
a guest holding 01:00.0 has the whole card.

Measured against the live site. A stopped VM was given the Quadro K620:
discovery went from alloc='' to alloc='vm 9099 (hpcod-devtest)', the card
dropped out of the schedulable set, and a GPU request went from being
rejected for want of cores to "gpu: none present (need 1)". Destroying the
VM put it back. The ConnectX-4 Lx on the same host is classified roce, not
infiniband, and stays unschedulable for want of agent confirmation - which
is the correct answer for that card.

The second way is the window the hypervisor cannot see. A cluster takes
minutes to build, and until its VMs exist the cards it was promised still
look free. So a reservation is written when the cluster is allocated, in
the same transaction as its address, and the planner reads both. The
partial unique index on live devices enforces one holder per card exactly
as the cluster_id index does for addresses.

Allocation now takes the whole placement rather than a site name, because
storing an address without a record of where the cluster runs creates a
cluster nothing will ever release. A device conflict therefore rolls the
address back too - tested by checking that the next allocation gets the
number the failed one would have consumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 23:06:46 +02:00
..
2026-09-01 12:08:26 +02:00