Files
werewulf/hpcod/db/002-placement.sql
ilkerandClaude Opus 5 39700254fa hpcod: stop offering hardware that is already spoken for
Placement knew how to pick the node with the GPU. It did not know which
GPUs were already taken, in two separate ways.

Device.AllocatedTo was read in three places and written in none, so every
discovery reported every card as free. Two clusters provisioned in sequence
would both be assigned the same GPU; the second VM fails to start, hours
after the portal promised the hardware. Ownership now comes from the
hypervisor, which also accounts for guests created outside hpcod. Two
details that are easy to get wrong and are covered by tests: a stopped
guest still owns its card, because the assignment lives in the config and
not in the running process; and ownership is per slot, not per function, so
a guest holding 01:00.0 has the whole card.

Measured against the live site. A stopped VM was given the Quadro K620:
discovery went from alloc='' to alloc='vm 9099 (hpcod-devtest)', the card
dropped out of the schedulable set, and a GPU request went from being
rejected for want of cores to "gpu: none present (need 1)". Destroying the
VM put it back. The ConnectX-4 Lx on the same host is classified roce, not
infiniband, and stays unschedulable for want of agent confirmation - which
is the correct answer for that card.

The second way is the window the hypervisor cannot see. A cluster takes
minutes to build, and until its VMs exist the cards it was promised still
look free. So a reservation is written when the cluster is allocated, in
the same transaction as its address, and the planner reads both. The
partial unique index on live devices enforces one holder per card exactly
as the cluster_id index does for addresses.

Allocation now takes the whole placement rather than a site name, because
storing an address without a record of where the cluster runs creates a
cluster nothing will ever release. A device conflict therefore rolls the
address back too - tested by checking that the next allocation gets the
number the failed one would have consumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-20 23:06:46 +02:00

65 lines
2.8 KiB
PL/PgSQL

-- Where a cluster was placed, and which hardware it holds.
--
-- Until now placement was advisory: /v1/plan worked out which hypervisor had
-- the GPU and which node had room, printed it, and threw it away. The cluster
-- was then created from a hand-written tfvars that named one hypervisor and
-- no devices at all. So the decision and the act were unconnected, and two
-- requests planned a minute apart would both be told they could have the only
-- GPU in the rack.
--
-- The reservation has to live here rather than only on the hypervisor.
-- Proxmox knows which guest holds a card, but only once that guest exists,
-- and between deciding and creating it there is a window of minutes during
-- which the card still looks free. Both sources are needed and they answer
-- different questions: this table says what we have promised, the hypervisor
-- says what is actually attached.
BEGIN;
CREATE TABLE IF NOT EXISTS cluster_vm (
cluster_serial_id bigint NOT NULL REFERENCES cluster(serial_id) ON DELETE CASCADE,
role text NOT NULL CHECK (role IN ('head', 'compute')),
-- 0 for the head, 1..n for compute. Matches the VM names the provisioner
-- generates, so a row here can be tied back to a running guest.
node_index int NOT NULL,
host_node text NOT NULL,
cores int NOT NULL CHECK (cores > 0),
memory_gb int NOT NULL CHECK (memory_gb > 0),
PRIMARY KEY (cluster_serial_id, role, node_index)
);
COMMENT ON COLUMN cluster_vm.host_node IS
'The hypervisor this VM was placed on. Per VM, not per cluster: a cluster too big for one host is spread, which is the normal case on a real site.';
CREATE TABLE IF NOT EXISTS cluster_device (
id bigserial PRIMARY KEY,
cluster_serial_id bigint NOT NULL REFERENCES cluster(serial_id) ON DELETE CASCADE,
role text NOT NULL,
node_index int NOT NULL,
-- Denormalised from cluster.site on purpose. A hypervisor name is only
-- unique within its site, and the uniqueness index below has to compare
-- whole addresses without joining.
site text NOT NULL,
host_node text NOT NULL,
native_id text NOT NULL,
capability text NOT NULL,
released_at timestamptz,
FOREIGN KEY (cluster_serial_id, role, node_index)
REFERENCES cluster_vm (cluster_serial_id, role, node_index) ON DELETE CASCADE
);
-- The same constraint the cluster_id index enforces, for hardware: one live
-- holder per device. A released cluster gives its cards back.
CREATE UNIQUE INDEX IF NOT EXISTS cluster_device_live_unique
ON cluster_device (site, host_node, native_id)
WHERE released_at IS NULL;
CREATE INDEX IF NOT EXISTS cluster_device_cluster_idx
ON cluster_device (cluster_serial_id);
COMMIT;