Files
llmgw/internal/store/leadership.go
T
Ilker ManapandClaude Opus 5 5b5e61cb97
CI / test (push) Successful in 1m35s
Stop a second control plane from placing work behind the first's back
The first thing HA needs that a shared store does not give. Two control planes
against one store both reconcile: both see a model wanting two replicas and
none running, and both place. The model ends up with twice what it asked for,
each replica holding GPUs the other's inventory believes are free — and nothing
crashes to say so.

A lease in a table rather than a lock held by a connection. PostgreSQL has
advisory locks and SQLite has nothing like them, and this store keeps one set
of queries for both; a row with an expiry works everywhere, and it survives a
leader being powered off rather than asked to stop, which is the case that
matters. Taking and renewing are the same statement, so the two cannot disagree
about what a held lease looks like, and it is one statement, so two processes
racing cannot both believe they won.

The owner id is random rather than the hostname. A hostname would let a
restarted process match its own stale lease and believe it had been leading all
along — including through the gap that made it restart, during which somebody
else may have taken over and started placing.

A store that cannot be reached is not evidence that somebody else is leading,
so an unreachable store keeps the claim. Two leaders is worse than one, but no
leader means nothing is placed and nothing is evicted, and a brief database
blip should not stop a cluster healing itself.

Unset means leader. Every deployment that existed before this, every
single-control-plane install and every test acts unconditionally, with nothing
to configure.

The same seven assertions run against both backends, which is how the test
harness was caught inheriting a lease between subtests — it passed on SQLite,
where each subtest gets a fresh file, and failed only on PostgreSQL.

What this does not yet do is move agents. A standby holding an agent's stream
is one the leader cannot command, so failover is degraded rather than clean
until agents know more than one address.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 18:24:06 +02:00

97 lines
3.4 KiB
Go

package store
import (
"context"
"errors"
"time"
)
// ErrNotLeader means somebody else holds the lease and it has not lapsed.
var ErrNotLeader = errors.New("store: another control plane holds the lease")
// Lease is who is in charge and until when.
type Lease struct {
Owner string
ExpiresAt time.Time
AcquiredAt time.Time
}
// AcquireLeadership takes or renews the lease for a role.
//
// Renewing and taking are the same statement on purpose: a leader that renews
// takes the same path as one starting fresh, so there is no way for the two to
// disagree about what a held lease looks like. It succeeds when the row is
// absent, when this process already owns it, or when whoever owned it let it
// lapse; otherwise it returns ErrNotLeader.
//
// The clock is the caller's rather than the database's, which is a compromise
// worth naming. A database clock would be one clock for everyone; taking it
// means a round trip that says now() inside every comparison, in SQL that is
// spelled differently in each dialect. Two control planes on machines whose
// clocks disagree by more than the lease will fight, so keep them on NTP —
// which any cluster running InfiniBand already has to be.
func (s *SQLite) AcquireLeadership(ctx context.Context, role, owner string, ttl time.Duration) (Lease, error) {
now := time.Now()
expires := now.Add(ttl)
// One statement, so two processes racing cannot both believe they won:
// the WHERE decides, and the loser changes no rows.
res, err := s.db.ExecContext(ctx,
`INSERT INTO leadership (role, owner, expires_at, acquired_at) VALUES (?, ?, ?, ?)
ON CONFLICT (role) DO UPDATE SET
owner = excluded.owner,
expires_at = excluded.expires_at,
acquired_at = CASE WHEN leadership.owner = excluded.owner
THEN leadership.acquired_at
ELSE excluded.acquired_at END
WHERE leadership.owner = excluded.owner OR leadership.expires_at < ?`,
role, owner, expires.UnixMilli(), now.UnixMilli(), now.UnixMilli())
if err != nil {
return Lease{}, err
}
n, err := res.RowsAffected()
if err != nil {
return Lease{}, err
}
if n == 0 {
// Lost, and the caller is told who to, because "not leader" without a
// name is the start of a long afternoon. The lease is returned beside
// the error rather than instead of it.
held, lerr := s.Leadership(ctx, role)
if lerr != nil {
return Lease{}, lerr
}
return held, ErrNotLeader
}
return s.Leadership(ctx, role)
}
// Leadership reports who holds a role's lease.
func (s *SQLite) Leadership(ctx context.Context, role string) (Lease, error) {
var owner string
var expires, acquired int64
err := s.db.QueryRowContext(ctx,
`SELECT owner, expires_at, acquired_at FROM leadership WHERE role = ?`, role).
Scan(&owner, &expires, &acquired)
if err != nil {
return Lease{}, err
}
return Lease{
Owner: owner,
ExpiresAt: fromMillis(expires),
AcquiredAt: fromMillis(acquired),
}, nil
}
// ReleaseLeadership gives the lease up, so a standby takes over in seconds
// rather than waiting out the clock.
//
// Only the holder can release it: a process that has already lost the lease
// must not be able to take it from whoever has it now, which is exactly what
// an unconditional delete during a slow shutdown would do.
func (s *SQLite) ReleaseLeadership(ctx context.Context, role, owner string) error {
_, err := s.db.ExecContext(ctx,
`DELETE FROM leadership WHERE role = ? AND owner = ?`, role, owner)
return err
}