The control plane had no state beyond files: sites.yaml, a registry of published portals, and whatever an operator typed into a tfvars. That answers none of the questions it needs to answer - who owns this cluster, when does its lease end, which addresses are in use right now, who held this one before. Two identities, because they have opposite requirements, and this is the part worth getting right before anything depends on it. A cluster id is an address: one byte of a MAC, one octet of a subnet, bounded at 254, and therefore reused. Reuse is safe exactly while no two live clusters share one, which a partial unique index enforces - UNIQUE(cluster_id) WHERE destroyed_at IS NULL. It is in the database rather than the provisioner because check-then-insert lets two simultaneous requests both pass the check, which is the "two requests, one GPU" failure docs/20.10 lists. An accounting name is a record, and must never be reused. Slurm accounting is keyed on ClusterName, so recycling a name merges two unrelated clusters' job histories into one that cannot be separated afterwards. It is generated from a never-reused serial rather than written, so the two identities cannot drift apart: c002-17 is address 2, the seventeenth cluster created. Verified against the real database: a second live allocation of the same id is rejected, destroying one frees it, the two clusters that then shared address 2 got names c002-1 and c002-3, and an id above 254 fails the check constraint. The DSN lives in /etc/hpcod/db.env, not here. Nothing reads this schema yet - the Go store, id allocation at provision time, and lease enforcement are what it exists for, and none of them are written.
79 lines
3.3 KiB
PL/PgSQL
79 lines
3.3 KiB
PL/PgSQL
-- hpcod control plane state.
|
|
--
|
|
-- Everything the control plane must remember across restarts and across the
|
|
-- life of the clusters it creates. Until now this lived in Terraform state
|
|
-- files and in whatever the operator typed into a tfvars, which answers
|
|
-- neither "who owns this cluster" nor "which ids are in use right now".
|
|
--
|
|
-- Two ideas shape it:
|
|
--
|
|
-- A cluster id is an address. It is one byte of a MAC and one octet of a
|
|
-- subnet, so it is bounded at 254 and has to be reused. Reuse is safe only
|
|
-- while no two live clusters share one, which is a constraint the database
|
|
-- can enforce and application code cannot be trusted to.
|
|
--
|
|
-- An accounting name is a record. It must never be reused, because Slurm
|
|
-- accounting is keyed on it and a recycled name silently merges two
|
|
-- different clusters' jobs into one history that cannot be separated
|
|
-- afterwards.
|
|
|
|
BEGIN;
|
|
|
|
CREATE TABLE IF NOT EXISTS tenant (
|
|
id bigserial PRIMARY KEY,
|
|
name text NOT NULL UNIQUE,
|
|
owner text NOT NULL,
|
|
created_at timestamptz NOT NULL DEFAULT now()
|
|
);
|
|
|
|
COMMENT ON TABLE tenant IS
|
|
'Who asks for clusters. A tenant may hold several at once.';
|
|
|
|
CREATE TABLE IF NOT EXISTS cluster (
|
|
-- Never reused. This is what the accounting name is built from, so that
|
|
-- a destroyed cluster and its replacement cannot be confused in slurmdbd.
|
|
serial_id bigserial PRIMARY KEY,
|
|
|
|
-- 1-254, recycled. The addressing identity: bridge, subnet, MAC prefix,
|
|
-- hostnames. Unique only among clusters that still exist - see the index
|
|
-- below, which is what makes recycling safe.
|
|
cluster_id smallint NOT NULL CHECK (cluster_id BETWEEN 1 AND 254),
|
|
|
|
tenant_id bigint NOT NULL REFERENCES tenant(id),
|
|
site text NOT NULL,
|
|
|
|
-- Generated rather than written, so the two identities cannot drift:
|
|
-- c002-17 is cluster_id 2, the seventeenth cluster ever created.
|
|
accounting_name text GENERATED ALWAYS AS
|
|
('c' || lpad(cluster_id::text, 3, '0') || '-' || serial_id) STORED,
|
|
|
|
state text NOT NULL DEFAULT 'requested'
|
|
CHECK (state IN ('requested','placed','provisioning',
|
|
'ready','expiring','deleting','deleted','failed')),
|
|
|
|
lease_expires_at timestamptz,
|
|
created_at timestamptz NOT NULL DEFAULT now(),
|
|
destroyed_at timestamptz,
|
|
|
|
spec jsonb NOT NULL DEFAULT '{}'::jsonb
|
|
);
|
|
|
|
-- The constraint that makes recycling an address safe. Two live clusters
|
|
-- cannot hold the same cluster_id; a destroyed one releases it. Enforced
|
|
-- here rather than by a check-then-insert in the provisioner, because two
|
|
-- requests arriving at once would both pass that check.
|
|
CREATE UNIQUE INDEX IF NOT EXISTS cluster_live_id_unique
|
|
ON cluster (cluster_id)
|
|
WHERE destroyed_at IS NULL;
|
|
|
|
CREATE INDEX IF NOT EXISTS cluster_tenant_idx ON cluster (tenant_id);
|
|
CREATE INDEX IF NOT EXISTS cluster_lease_idx ON cluster (lease_expires_at)
|
|
WHERE destroyed_at IS NULL;
|
|
|
|
COMMENT ON COLUMN cluster.cluster_id IS
|
|
'Addressing identity, 1-254, reused after the cluster is destroyed.';
|
|
COMMENT ON COLUMN cluster.accounting_name IS
|
|
'Slurm ClusterName. Never reused: recycling it would merge two clusters'' job records.';
|
|
|
|
COMMIT;
|