Files
llmgw/internal/store/copy.go
T
Ilker ManapandClaude Opus 5 5b5e61cb97
CI / test (push) Successful in 1m35s
Stop a second control plane from placing work behind the first's back
The first thing HA needs that a shared store does not give. Two control planes
against one store both reconcile: both see a model wanting two replicas and
none running, and both place. The model ends up with twice what it asked for,
each replica holding GPUs the other's inventory believes are free — and nothing
crashes to say so.

A lease in a table rather than a lock held by a connection. PostgreSQL has
advisory locks and SQLite has nothing like them, and this store keeps one set
of queries for both; a row with an expiry works everywhere, and it survives a
leader being powered off rather than asked to stop, which is the case that
matters. Taking and renewing are the same statement, so the two cannot disagree
about what a held lease looks like, and it is one statement, so two processes
racing cannot both believe they won.

The owner id is random rather than the hostname. A hostname would let a
restarted process match its own stale lease and believe it had been leading all
along — including through the gap that made it restart, during which somebody
else may have taken over and started placing.

A store that cannot be reached is not evidence that somebody else is leading,
so an unreachable store keeps the claim. Two leaders is worse than one, but no
leader means nothing is placed and nothing is evicted, and a brief database
blip should not stop a cluster healing itself.

Unset means leader. Every deployment that existed before this, every
single-control-plane install and every test acts unconditionally, with nothing
to configure.

The same seven assertions run against both backends, which is how the test
harness was caught inheriting a lease between subtests — it passed on SQLite,
where each subtest gets a fresh file, and failed only on PostgreSQL.

What this does not yet do is move agents. A standby holding an agent's stream
is one the leader cannot command, so failover is degraded rather than clean
until agents know more than one address.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-25 18:24:06 +02:00

220 lines
7.2 KiB
Go

package store
import (
"context"
"fmt"
"strings"
)
// copyOrder is every table, parents before children.
//
// Written out rather than discovered, because the order is a fact about the
// foreign keys and getting it wrong produces a constraint violation halfway
// through — with some tables copied and some not, which is the worst place to
// stop. A table added without being added here is copied by nothing, and the
// destination comes up quietly missing it; the test below is what catches
// that.
var copyOrder = []string{
"organizations",
"teams",
"projects",
// Before api_keys, which references it. The first version of this list
// followed the order the tables are created across the migrations, and
// users is created in a later one than api_keys even though api_keys
// points at it — so the copy failed on a foreign key three tables in.
"users",
"api_keys",
"key_claims",
"sessions",
"limits",
"usage",
"models",
"threads",
"thread_messages",
"memory_notes",
}
// notCopied are tables deliberately left behind.
//
// Leadership is a claim about a process that is running right now. Copying it
// would hand the new store a leader that does not exist, and the standby
// waiting on that lease would wait out its full expiry before noticing — at
// the exact moment somebody is watching a migration and wondering why nothing
// is being placed.
var notCopied = map[string]string{
"leadership": "a lease on a running process, meaningless in a new store",
}
// CopyResult is how many rows each table received.
type CopyResult struct {
Table string
Rows int64
}
// CopyTo copies every row into another store.
//
// For moving from SQLite to PostgreSQL, which otherwise means reissuing every
// API key: the keys are stored hashed, so there is no way to recreate them and
// no way for anyone to keep using the one they have.
//
// The destination must be empty. Merging two stores would mean deciding what
// happens when both have a project with the same id and different contents,
// and every answer to that is a guess about which one someone meant.
//
// Not a transaction across the whole copy. It is one transaction per table,
// because a single transaction holding every row of a large usage table is how
// a migration runs out of memory on the machine least able to spare it. A
// failure therefore leaves a partial copy, which is why the destination has to
// have been empty: starting again is dropping and recreating it.
func (s *SQLite) CopyTo(ctx context.Context, dst *SQLite) ([]CopyResult, error) {
empty, where, err := dst.isEmpty(ctx)
if err != nil {
return nil, err
}
if !empty {
return nil, fmt.Errorf("the destination already has rows in %s; copy into an empty store", where)
}
var out []CopyResult
for _, table := range copyOrder {
n, err := s.copyTable(ctx, dst, table)
if err != nil {
return out, fmt.Errorf("copying %s: %w", table, err)
}
out = append(out, CopyResult{Table: table, Rows: n})
}
// Part of the copy, not a step someone has to remember: a destination
// whose sequences still start at 1 is not a destination anyone can use.
if err := dst.resetIdentities(ctx); err != nil {
return out, fmt.Errorf("pointing the identity columns past the copied rows: %w", err)
}
return out, nil
}
// isEmpty reports whether any table has rows, and which one first does.
func (s *SQLite) isEmpty(ctx context.Context) (bool, string, error) {
for _, table := range copyOrder {
var n int64
//nolint:gosec // table comes from copyOrder, not from input
if err := s.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM `+table).Scan(&n); err != nil {
return false, "", err
}
if n > 0 {
return false, table, nil
}
}
return true, "", nil
}
// copyTable moves one table, column names taken from the source.
//
// Reading the columns rather than listing them keeps this working when a
// migration adds one: a hand-written column list would copy the old shape and
// silently drop the new field, and nothing would notice until someone looked
// for a setting that was there before the move.
func (s *SQLite) copyTable(ctx context.Context, dst *SQLite, table string) (int64, error) {
//nolint:gosec // table comes from copyOrder, not from input
rows, err := s.db.QueryContext(ctx, `SELECT * FROM `+table)
if err != nil {
return 0, err
}
defer rows.Close()
cols, err := rows.Columns()
if err != nil {
return 0, err
}
q, tx, err := dst.begin(ctx)
if err != nil {
return 0, err
}
defer func() { _ = tx.Rollback() }()
insert := `INSERT INTO ` + table + ` (` + strings.Join(quoteAll(cols), ", ") +
`) VALUES (` + strings.Repeat("?, ", len(cols)-1) + `?)`
var n int64
for rows.Next() {
vals := make([]any, len(cols))
ptrs := make([]any, len(cols))
for i := range vals {
ptrs[i] = &vals[i]
}
if err := rows.Scan(ptrs...); err != nil {
return n, err
}
for i, v := range vals {
vals[i] = keepEmptyBytes(v)
}
if _, err := q.ExecContext(ctx, insert, vals...); err != nil {
return n, err
}
n++
}
if err := rows.Err(); err != nil {
return n, err
}
return n, tx.Commit()
}
// keepEmptyBytes stops an empty byte column becoming NULL on the way across.
//
// An empty BLOB scans as a typed nil slice — []byte(nil) inside a non-nil any
// — while a real NULL scans as an untyped nil. The two are distinguishable
// here and not in pgx, which encodes any nil slice as NULL, so the difference
// has to be restored before the insert.
//
// It is not a detail. key_claims.sealed is emptied the moment a claim is used,
// so every claimed row holds an empty blob in a NOT NULL column, and the copy
// failed on the first one — after the gateway had been stopped and four tables
// had already been written.
func keepEmptyBytes(v any) any {
if b, ok := v.([]byte); ok && b == nil {
return []byte{}
}
return v
}
// quoteAll quotes column names, because `key` and `text` are reserved words in
// one dialect or the other and both are column names here.
func quoteAll(cols []string) []string {
out := make([]string, len(cols))
for i, c := range cols {
out[i] = `"` + c + `"`
}
return out
}
// resetIdentities points PostgreSQL's identity columns past the copied rows.
//
// Without it the first insert after a migration collides with a row that was
// copied in: the sequence still starts at 1 while the table already holds a
// thousand notes, and the failure arrives later, to somebody else, looking
// like a bug in the feature rather than in the move.
func (s *SQLite) resetIdentities(ctx context.Context) error {
if s.dialect != PostgresDialect {
return nil
}
// Only the tables with a generated id. SQLite's rowid needs nothing.
for _, t := range []string{"memory_notes", "usage"} {
var has bool
if err := s.db.QueryRowContext(ctx,
`SELECT EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_name = ? AND column_name = 'id'
AND is_identity = 'YES')`, t).Scan(&has); err != nil {
return err
}
if !has {
continue
}
//nolint:gosec // t is a literal from the list above
if _, err := s.db.ExecContext(ctx,
`SELECT setval(pg_get_serial_sequence('`+t+`', 'id'),
GREATEST((SELECT COALESCE(MAX(id), 0) FROM `+t+`), 1))`); err != nil {
return err
}
}
return nil
}