Ilker ManapandClaude Opus 5
5b5e61cb97
CI / test (push) Successful in 1m35s
The first thing HA needs that a shared store does not give. Two control planes against one store both reconcile: both see a model wanting two replicas and none running, and both place. The model ends up with twice what it asked for, each replica holding GPUs the other's inventory believes are free — and nothing crashes to say so. A lease in a table rather than a lock held by a connection. PostgreSQL has advisory locks and SQLite has nothing like them, and this store keeps one set of queries for both; a row with an expiry works everywhere, and it survives a leader being powered off rather than asked to stop, which is the case that matters. Taking and renewing are the same statement, so the two cannot disagree about what a held lease looks like, and it is one statement, so two processes racing cannot both believe they won. The owner id is random rather than the hostname. A hostname would let a restarted process match its own stale lease and believe it had been leading all along — including through the gap that made it restart, during which somebody else may have taken over and started placing. A store that cannot be reached is not evidence that somebody else is leading, so an unreachable store keeps the claim. Two leaders is worse than one, but no leader means nothing is placed and nothing is evicted, and a brief database blip should not stop a cluster healing itself. Unset means leader. Every deployment that existed before this, every single-control-plane install and every test acts unconditionally, with nothing to configure. The same seven assertions run against both backends, which is how the test harness was caught inheriting a lease between subtests — it passed on SQLite, where each subtest gets a fresh file, and failed only on PostgreSQL. What this does not yet do is move agents. A standby holding an agent's stream is one the leader cannot command, so failover is degraded rather than clean until agents know more than one address. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30 lines
1.5 KiB
SQL
30 lines
1.5 KiB
SQL
-- Ported from the SQLite migration of the same number. BIGINT because
|
|
-- PostgreSQL's INTEGER is 32 bits and these are milliseconds since the epoch.
|
|
|
|
-- A lease on being the control plane that places work.
|
|
--
|
|
-- Two control planes against one store both reconcile: both see a model
|
|
-- wanting two replicas and none running, and both place. The result is twice
|
|
-- the replicas, each holding GPUs the other's inventory believes are free —
|
|
-- the same failure a control plane restart used to produce before adoption,
|
|
-- and harder to see because nothing crashed.
|
|
--
|
|
-- A lease rather than a lock held by the connection. PostgreSQL has advisory
|
|
-- locks and SQLite has nothing like them, and this store keeps one set of
|
|
-- queries for both; a row with an expiry works everywhere and survives the
|
|
-- leader being killed rather than asked to stop.
|
|
--
|
|
-- One row, enforced by the primary key: there is one thing to be leader of.
|
|
CREATE TABLE leadership (
|
|
role TEXT PRIMARY KEY,
|
|
-- Who holds it. An opaque id per process, so a process cannot be fooled by
|
|
-- its own stale lease after a restart on the same host.
|
|
owner TEXT NOT NULL,
|
|
-- When the lease lapses. The holder renews well before this; anyone else
|
|
-- may take it once the clock has passed, which is what makes a leader that
|
|
-- was powered off recoverable without anyone being paged.
|
|
expires_at BIGINT NOT NULL,
|
|
-- For an operator asking who has been in charge and since when.
|
|
acquired_at BIGINT NOT NULL
|
|
);
|