diff --git a/BACKLOG.md b/BACKLOG.md index 24270927..191caae3 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -829,25 +829,41 @@ ACL yet); a personal key or session names a project or team and gets that chain only, never the union of its person's memberships; PostgreSQL store only, off on SQLite with a sentence saying why. -### F1. Store: collections, grants, chunks (M) -- `pgvector` ≥ 0.8 installed on every Patroni member by `server init` - (`internal/cli/serverinit_postgres.go`), `CREATE EXTENSION` in the - migration; the SQLite migration has no tables, and the store reports the - feature unavailable. -- Tables: `kb_collections` (id, org_id, owner `team:…`/`project:…`/`user:…`, - name, embed_model, dims, created_at), `kb_grants` (scope, scope_id, - collection, perm read|write — like `guard_attachments`), `kb_documents` - (name, digest, bytes, state, chunk count), `kb_chunks` (org_id, - collection, doc_id, ord, body, tsv, embedding) `PARTITION BY LIST - (org_id)`, a partition per organization, HNSW and GIN indexes per - partition. -- Row-level security on `kb_chunks` against `current_setting('llmgw.org')` - and `llmgw.collections`, retrieval through its own role. -- Rules, refused before anything is written: a grant never crosses - organizations; owner and grants resolve to existing scopes; deleting a - project/team/org and `users forget` delete their collections (`forget.go`). -- Done: store tests on PostgreSQL, including an RLS test that a query - without the settings returns no rows. +### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`) +Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only): +- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants` + (read|write, any scope, never across organizations; a key needs a + project; a person needs a membership in the organization), + `kb_documents` (digest: the same again is no change, a new one replaces + in one transaction). +- `kb_chunks` is created when the store opens, not in a migration (a + migration needing an extension would stop the control plane; one that + skipped it would never run again): partitioned by organization (hashed + partition name), `tsv` generated (`simple`), GIN on the parent, one + partial HNSW index per collection on its organization's partition + (`embedding::vector(dims)`, `WHERE collection = ''`). EXPLAIN on + 20k rows: partition pruned, that index used, policy applied. +- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections` + (set with set_config per transaction), one role — the table owner — not + a separate one. Chunks are deleted by the foreign-key cascade from their + document, which RLS does not block. +- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain; + no organization (personal key without a project) reads nothing. +- `SearchChunks`: per collection nearest-neighbour (literal id and cast so + the partial index is used) plus a full-text arm with any word + (`a | b`, lowercased by the database, not Go), fused by reciprocal rank. +- Deleting a project/team/org that owns collections is refused like one + with members (changed from "deleted with them": every other delete in the + tree refuses); DeleteUser/ForgetUser delete the person's collections; + deleting an organization drops its partition; grants to a deleted scope go. +- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables + it; pgha installs it on every member at setup and enables it on the + leader on every run. Neither stops on failure. +Left: a cluster set up before (the PoC) gets no package from a rerun of +pgha on a running cluster — install by hand or add an upgrade step; +`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only, +and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a +pgvector database as a non-superuser (a superuser bypasses RLS). ### F2. Ingest (M) - `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add diff --git a/CHANGELOG.md b/CHANGELOG.md index 0ea1a5ff..b5657c56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,23 @@ is only released once its **exit criterion** is met, not when its code lands. Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. +## Unreleased + +- **Knowledge bases, the store's part** (BACKLOG F1, branch `rag`): + collections of documents owned by an organization, team, project or + person, grants that add up down the tenancy tree and never cross an + organization, chunks in PostgreSQL with pgvector — partitioned by + organization, an HNSW index per collection, row-level security on the + table as a second line behind every query's own filter — and a search + that fuses nearest-neighbour and full-text ranks. Off on SQLite, and on + PostgreSQL without pgvector, saying why. `server init` and the + high-availability database setup install pgvector and enable it; a + cluster set up before needs `postgresql--pgvector` on every + database machine and `CREATE EXTENSION vector` in llmgw's database. A + project, team or organization that owns collections is not deleted; a + person's collections go with their account. Nothing reads or writes them + yet from a command or a request. + ## v0.13.0 — 2026-10-08 The whole system stopped, started and restarted from one command, a diff --git a/internal/cli/serverinit_postgres.go b/internal/cli/serverinit_postgres.go index 56c7d1ab..41417191 100644 --- a/internal/cli/serverinit_postgres.go +++ b/internal/cli/serverinit_postgres.go @@ -370,6 +370,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) { in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)") } + in.setupPgvector(ctx, discovery.DiscoverHost()) + // Proof the address works, before anything is moved into it. address := "@" + in.dsnPath() s, err := store.Open(ctx, address) @@ -482,8 +484,17 @@ func lastLines(b []byte, n int) string { // psql runs one statement as the postgres superuser, over the local socket. func (in *serverInit) psql(ctx context.Context, query string) (string, error) { - cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--", - in.pg.tool("psql"), "-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query) + return in.psqlIn(ctx, "", query) +} + +// psqlIn runs a statement as postgres in one database, or in postgres's +// default one when db is "". +func (in *serverInit) psqlIn(ctx context.Context, db, query string) (string, error) { + args := []string{"-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query} + if db != "" { + args = append(args, "-d", db) + } + cmd := exec.CommandContext(ctx, "runuser", append([]string{"-u", "postgres", "--", in.pg.tool("psql")}, args...)...) // postgres cannot enter root's home, and psql says so on every call. cmd.Dir = "/" out, err := cmd.CombinedOutput() @@ -538,3 +549,50 @@ func (in *serverInit) stopService(ctx context.Context) (bool, error) { in.done("llmgw-server stopped while its store moves") return true, nil } + +// pgvectorPackages are the package names of pgvector for a PostgreSQL major +// version: Debian's and Ubuntu's (and PGDG's apt repository) name it after the +// server, PGDG's rpm repository after the version alone. +func pgvectorPackages(major int) map[string][]string { + return map[string][]string{ + "deb": {fmt.Sprintf("postgresql-%d-pgvector", major)}, + "rpm": {fmt.Sprintf("pgvector_%d", major)}, + } +} + +// setupPgvector installs pgvector and enables it in llmgw's database, for +// knowledge bases. Not a reason to stop: without it everything else works, +// and the store says why knowledge bases are off when somebody asks for one. +func (in *serverInit) setupPgvector(ctx context.Context, h discovery.Host) { + warn := func(format string, a ...any) { + fmt.Fprintf(in.out, " !! pgvector: "+format+"; knowledge bases stay off\n", a...) + } + if on, err := in.psqlIn(ctx, pgDatabase, "SELECT 1 FROM pg_extension WHERE extname = 'vector'"); err == nil && on == "1" { + in.kept("pgvector in database %s", pgDatabase) + return + } + have, err := in.psqlBool(ctx, "SELECT 1 FROM pg_available_extensions WHERE name = 'vector'") + if err != nil { + warn("%v", err) + return + } + if !have { + num, err := in.psql(ctx, "SHOW server_version_num") + n, convErr := strconv.Atoi(num) + if err != nil || convErr != nil || h.Family() == "" { + warn("cannot tell which package to install (version %q, %s)", num, h.OSID) + return + } + pkgs := pgvectorPackages(n / 10000) + pi := &provision.Installer{Host: h, Log: slog.Default()} + if err := pi.InstallPackages(ctx, pkgs); err != nil { + warn("installing %s: %v", strings.Join(pkgs[h.Family()], " "), err) + return + } + } + if _, err := in.psqlIn(ctx, pgDatabase, "CREATE EXTENSION IF NOT EXISTS vector"); err != nil { + warn("%v", err) + return + } + in.done("pgvector enabled in database %s", pgDatabase) +} diff --git a/internal/pgha/pgha_test.go b/internal/pgha/pgha_test.go index de281922..dc7eb792 100644 --- a/internal/pgha/pgha_test.go +++ b/internal/pgha/pgha_test.go @@ -159,6 +159,11 @@ func TestARunningClusterIsLeftAsItIs(t *testing.T) { if !a.ranAny("CREATE ROLE") { t.Error("the role was not made sure of on the leader") } + // Enabled where the package is there, which on a running cluster is + // all a rerun does about it: it installs nothing. + if !a.ranAny("CREATE EXTENSION IF NOT EXISTS vector") { + t.Error("pgvector was not enabled in llmgw's database on the leader") + } } func TestTwoDatabaseMachinesWithoutAThirdVoteAreRefused(t *testing.T) { diff --git a/internal/pgha/setup.go b/internal/pgha/setup.go index 255dd6ff..05ad9e5b 100644 --- a/internal/pgha/setup.go +++ b/internal/pgha/setup.go @@ -502,6 +502,14 @@ if [ -n "$need" ]; then sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y $need >/dev/null fi v=$(ls /usr/lib/postgresql | sort -n | tail -1) +# pgvector, for knowledge bases: on every member, since any of them may lead. +# Not a reason to stop; the store says knowledge bases are off without it. +if [ ! -f /usr/share/postgresql/$v/extension/vector.control ]; then + sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y postgresql-$v-pgvector >/dev/null 2>&1 || + { sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null 2>&1 && + sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y postgresql-$v-pgvector >/dev/null 2>&1; } || + echo "no-pgvector" +fi if [ -d /var/lib/postgresql/$v/main ]; then if sudo systemctl is-active --quiet postgresql@$v-main; then n=$(sudo -u postgres psql -Atc "select count(*) from pg_database where datname not in ('postgres','template0','template1')") @@ -553,13 +561,21 @@ sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL' SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec ALTER ROLE llmgw PASSWORD :'pw'; SELECT format('CREATE DATABASE %I OWNER %I', 'llmgw', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = 'llmgw') \gexec -SQL`, nil) +SQL +sudo -u postgres psql -q -d llmgw -v ON_ERROR_STOP=1 <<'SQL' || echo "pgvector-failed" +SELECT 'CREATE EXTENSION IF NOT EXISTS vector' WHERE EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'vector') \gexec +SQL +sudo -u postgres psql -d llmgw -Atc "select 'vector=' || count(*) from pg_extension where extname = 'vector'"`, nil) if err != nil { return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out))) } // Said as a change only when something was made: a rerun that finds // both is not a reason to restart the gateway. s.emit("database", !strings.Contains(string(out), "had=2"), "role and database %s on %s", Database, leader.Name) + if !strings.Contains(string(out), "vector=1") { + s.emit("pgvector", false, "pgvector is not enabled in %s, so knowledge bases stay off "+ + "(install postgresql--pgvector on every member and run setup again)", Database) + } return nil } diff --git a/internal/store/hierarchy.go b/internal/store/hierarchy.go index cc9d128c..5728ab28 100644 --- a/internal/store/hierarchy.go +++ b/internal/store/hierarchy.go @@ -214,6 +214,12 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check busy = append(busy, plural(n, ch.noun)) } } + // Not an emptyCheck: the table is there only on PostgreSQL. + if n, err := s.kbOwnedBy(ctx, c, scope, id); err != nil { + return err + } else if n > 0 { + busy = append(busy, plural(n, "collection")) + } if len(busy) > 0 { return fmt.Errorf("%w: it still has %s", ErrNotEmpty, strings.Join(busy, " and ")) } @@ -222,6 +228,9 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check string(scope), string(MemberScope(scope)), id); err != nil { return err } + if err := s.kbForgetScope(ctx, c, scope, id); err != nil { + return err + } res, err := c.ExecContext(ctx, del, id) if err != nil { return err @@ -229,7 +238,7 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check if n, _ := res.RowsAffected(); n == 0 { return ErrNotFound } - return nil + return s.kbDropPartition(ctx, c, scope, id) }) } @@ -248,6 +257,10 @@ func (s *DB) DeleteUser(ctx context.Context, userID string, at time.Time) error millis(at), userID); err != nil { return err } + // A person's own collections go with the account. + if err := s.kbForgetScope(ctx, c, ScopeUser, userID); err != nil { + return err + } res, err := c.ExecContext(ctx, `DELETE FROM users WHERE id = ?`, userID) if err != nil { return err diff --git a/internal/store/knowledge.go b/internal/store/knowledge.go new file mode 100644 index 00000000..33545705 --- /dev/null +++ b/internal/store/knowledge.go @@ -0,0 +1,833 @@ +package store + +import ( + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "errors" + "fmt" + "regexp" + "sort" + "strconv" + "strings" + "time" + "unicode" +) + +// Knowledge bases (BACKLOG F): collections of documents, split into chunks +// and embedded, that a request is answered from. +// +// Which tenant may read which chunk is decided twice. The queries here filter +// by organization and by the collections the caller may read, and the table +// enforces the same with row-level security against two settings each +// transaction makes — so a query that forgot its filter returns nothing +// rather than another tenant's passages. The role llmgw connects as owns the +// table, so the policy is FORCEd; a superuser would bypass it, which is one +// more reason the store never connects as one. + +// ErrKnowledgeOff is a store that cannot hold knowledge bases. +var ErrKnowledgeOff = errors.New("knowledge bases are off") + +// MaxEmbeddingDims is the most dimensions pgvector can index (HNSW on vector). +const MaxEmbeddingDims = 2000 + +// KBPerm is what a grant allows. +type KBPerm string + +// The permissions. Write includes read. +const ( + KBRead KBPerm = "read" + KBWrite KBPerm = "write" +) + +// KBCollection is a set of documents with one owner and one embedding model. +type KBCollection struct { + ID string + OrgID string + OwnerScope Scope + OwnerID string + Name string + EmbedModel string + Dims int + CreatedAt time.Time + CreatedBy string +} + +// KBGrant lets a scope, and everything below it, read or write a collection. +type KBGrant struct { + Scope Scope + ScopeID string + Collection string + Perm KBPerm + AddedAt time.Time + AddedBy string +} + +// KBDocument is a document as added to a collection. +type KBDocument struct { + ID string + Collection string + Name string + Digest string + Bytes int64 + Chunks int + AddedAt time.Time + AddedBy string +} + +// KBChunk is a piece of a document and its embedding. +type KBChunk struct { + Body string + Embedding []float32 +} + +// KBHit is a chunk a search returned, best first. +type KBHit struct { + Collection string + DocID string + Document string + Ord int + Body string + // Score is the fused rank (reciprocal rank fusion over the vector and + // text searches): only the order means anything. + Score float64 +} + +// KnowledgeAvailable says whether this store can hold knowledge bases, and +// why not when it cannot. +func (s *DB) KnowledgeAvailable() error { + if s.dialect != PostgresDialect { + return fmt.Errorf("%w: they need the PostgreSQL store (with pgvector), and this one is SQLite", ErrKnowledgeOff) + } + return s.kbErr +} + +// ensureKnowledge creates kb_chunks when the vector extension allows it. +// +// At open rather than in a migration: a migration that needs an extension the +// database servers do not have would stop the control plane from starting, +// and one that skipped it would be recorded as applied and never run again +// once the extension arrived. +func (s *DB) ensureKnowledge(ctx context.Context) error { + var have bool + if err := s.db.QueryRowContext(ctx, + `SELECT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'vector')`).Scan(&have); err != nil { + return fmt.Errorf("%w: reading the database's extensions: %v", ErrKnowledgeOff, err) + } + if !have { + var available bool + if err := s.db.QueryRowContext(ctx, + `SELECT EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'vector')`).Scan(&available); err != nil { + return fmt.Errorf("%w: reading the database's extensions: %v", ErrKnowledgeOff, err) + } + if !available { + return fmt.Errorf("%w: pgvector is not installed on the database servers "+ + "(package postgresql--pgvector on every one; llmgw's database setup installs it)", ErrKnowledgeOff) + } + // Works only where the extension is trusted or llmgw's role may + // create it; elsewhere a superuser has to, once. + if _, err := s.db.ExecContext(ctx, `CREATE EXTENSION IF NOT EXISTS vector`); err != nil { + return fmt.Errorf("%w: pgvector is installed but not enabled in llmgw's database; "+ + "as postgres, run CREATE EXTENSION vector in it (%v)", ErrKnowledgeOff, err) + } + } + var exists bool + if err := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); err != nil { + return fmt.Errorf("%w: %v", ErrKnowledgeOff, err) + } + if exists { + return nil + } + err := s.inTx(ctx, func(c conn) error { + for _, q := range []string{ + // Partitioned by organization: each one's indexes stay its own + // size, and an organization's chunks go with one DROP. + `CREATE TABLE kb_chunks ( + org_id TEXT NOT NULL, + collection TEXT NOT NULL, + doc_id TEXT NOT NULL REFERENCES kb_documents(id) ON DELETE CASCADE, + ord INTEGER NOT NULL, + body TEXT NOT NULL, + tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', body)) STORED, + -- No dimension here: collections differ. Each collection's + -- index casts to its own (kbVectorIndex). + embedding vector NOT NULL, + PRIMARY KEY (org_id, doc_id, ord) + ) PARTITION BY LIST (org_id)`, + `CREATE INDEX kb_chunks_tsv ON kb_chunks USING gin (tsv)`, + `ALTER TABLE kb_chunks ENABLE ROW LEVEL SECURITY`, + `ALTER TABLE kb_chunks FORCE ROW LEVEL SECURITY`, + `CREATE POLICY kb_scope ON kb_chunks + USING (org_id = current_setting('llmgw.kb_org', true) + AND collection = ANY (string_to_array(current_setting('llmgw.kb_collections', true), ','))) + WITH CHECK (org_id = current_setting('llmgw.kb_org', true) + AND collection = ANY (string_to_array(current_setting('llmgw.kb_collections', true), ',')))`, + } { + if _, err := c.ExecContext(ctx, q); err != nil { + return err + } + } + return nil + }) + if err != nil { + // Another control plane opening at the same moment made it first. + if e2 := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); e2 == nil && exists { + return nil + } + return fmt.Errorf("%w: creating kb_chunks: %v", ErrKnowledgeOff, err) + } + return nil +} + +// kbIDPattern is what a collection id looks like (NewID("kb_")); ids are put +// into DDL and partial-index predicates, so nothing else is let through. +var kbIDPattern = regexp.MustCompile(`^kb_[a-z2-7]{16}$`) + +// kbPartition is the chunk partition of an organization. Hashed, because old +// organization ids were slugs of names and could hold anything. +func kbPartition(orgID string) string { + h := sha256.Sum256([]byte(orgID)) + return "kb_chunks_" + hex.EncodeToString(h[:12]) +} + +// kbVectorIndex is the name of a collection's vector index. +func kbVectorIndex(collection string) string { + return "kb_vec_" + strings.TrimPrefix(collection, "kb_") +} + +func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" } + +// kbScope sets, for this transaction, which rows of kb_chunks it may touch. +func kbScope(ctx context.Context, c conn, orgID string, collections []string) error { + _, err := c.ExecContext(ctx, `SELECT set_config('llmgw.kb_org', ?, true), set_config('llmgw.kb_collections', ?, true)`, + orgID, strings.Join(collections, ",")) + return err +} + +// orgOfScope is the organization a scope sits in. A person has none of their +// own, so a person is checked against org instead: they must be a member of +// it, or of something in it. +func (s *DB) orgOfScope(ctx context.Context, c conn, scope Scope, id, org string) (string, error) { + var q string + switch scope { + case ScopeOrg: + q = `SELECT id FROM organizations WHERE id = ?` + case ScopeTeam: + q = `SELECT org_id FROM teams WHERE id = ?` + case ScopeProject: + q = `SELECT t.org_id FROM projects p JOIN teams t ON t.id = p.team_id WHERE p.id = ?` + case ScopeKey: + q = `SELECT t.org_id FROM api_keys k JOIN projects p ON p.id = k.project_id + JOIN teams t ON t.id = p.team_id WHERE k.id = ?` + case ScopeUser: + var n int + err := c.QueryRowContext(ctx, `SELECT COUNT(*) FROM memberships m WHERE m.user_id = ? AND ( + (m.scope_type = 'org' AND m.scope_id = ?) + OR (m.scope_type = 'team' AND m.scope_id IN (SELECT id FROM teams WHERE org_id = ?)) + OR (m.scope_type = 'project' AND m.scope_id IN ( + SELECT p.id FROM projects p JOIN teams t ON t.id = p.team_id WHERE t.org_id = ?)))`, + id, org, org, org).Scan(&n) + if err != nil { + return "", err + } + if n == 0 { + return "", fmt.Errorf("%s is not a member of anything in that organization", id) + } + return org, nil + default: + return "", fmt.Errorf("a collection cannot be owned by or granted to a %s", scope) + } + var got string + err := c.QueryRowContext(ctx, q, id).Scan(&got) + if errors.Is(err, sql.ErrNoRows) { + if scope == ScopeKey { + return "", fmt.Errorf("no key %s in a project (a personal key reads no collection)", id) + } + return "", fmt.Errorf("no %s %s: %w", scope, id, ErrNotFound) + } + return got, err +} + +// CreateCollection makes an empty collection, its organization's chunk +// partition when this is the first, and its vector index. c.OrgID is needed +// only when a person owns it; otherwise it comes from the owner. +func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection, error) { + if err := s.KnowledgeAvailable(); err != nil { + return KBCollection{}, err + } + c.Name = strings.TrimSpace(c.Name) + switch { + case c.Name == "": + return KBCollection{}, errors.New("a collection needs a name") + case c.EmbedModel == "": + return KBCollection{}, errors.New("a collection needs the embedding model its documents are embedded with") + case c.Dims < 1 || c.Dims > MaxEmbeddingDims: + return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, MaxEmbeddingDims) + case c.OwnerScope == ScopeUser && c.OrgID == "": + return KBCollection{}, errors.New("a person's collection names the organization it belongs to") + case c.OwnerScope == ScopeKey: + return KBCollection{}, errors.New("a key cannot own a collection; its project or its person can") + } + c.ID = NewID("kb_") + if c.CreatedAt.IsZero() { + c.CreatedAt = time.Now() + } + err := s.inTx(ctx, func(tx conn) error { + org, err := s.orgOfScope(ctx, tx, c.OwnerScope, c.OwnerID, c.OrgID) + if err != nil { + return err + } + if c.OrgID != "" && c.OrgID != org { + return fmt.Errorf("the owner is in organization %s, not %s", org, c.OrgID) + } + c.OrgID = org + var n int + if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM kb_collections WHERE org_id = ? AND name = ?`, + c.OrgID, c.Name).Scan(&n); err != nil { + return err + } + if n > 0 { + return fmt.Errorf("%w: collection %q", ErrDuplicateName, c.Name) + } + if _, err := tx.ExecContext(ctx, `INSERT INTO kb_collections + (id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`, + c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims, + millis(c.CreatedAt), c.CreatedBy); err != nil { + return err + } + part := kbPartition(c.OrgID) + if _, err := tx.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS `+part+ + ` PARTITION OF kb_chunks FOR VALUES IN (`+sqlLiteral(c.OrgID)+`)`); err != nil { + return err + } + // A partial index per collection, on its organization's partition + // only: the dimension is the collection's, and a search, which + // names one collection at a time, uses exactly this one. + _, err = tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+ + ` USING hnsw ((embedding::vector(`+strconv.Itoa(c.Dims)+`)) vector_cosine_ops)`+ + ` WHERE collection = `+sqlLiteral(c.ID)) + return err + }) + if err != nil { + return KBCollection{}, err + } + return c, nil +} + +const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by` + +func scanCollection(sc scanner) (KBCollection, error) { + var c KBCollection + var scope string + var at int64 + err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &at, &c.CreatedBy) + c.OwnerScope, c.CreatedAt = Scope(scope), fromMillis(at) + return c, err +} + +// Collections lists every collection, by organization and name. +func (s *DB) Collections(ctx context.Context) ([]KBCollection, error) { + if err := s.KnowledgeAvailable(); err != nil { + return nil, err + } + return queryAll(ctx, s.db, `SELECT `+kbCollectionCols+` FROM kb_collections ORDER BY org_id, name`, nil, + func(r *sql.Rows) (KBCollection, error) { return scanCollection(r) }) +} + +// Collection reads one collection. +func (s *DB) Collection(ctx context.Context, id string) (KBCollection, error) { + if err := s.KnowledgeAvailable(); err != nil { + return KBCollection{}, err + } + c, err := scanCollection(s.db.QueryRowContext(ctx, `SELECT `+kbCollectionCols+` FROM kb_collections WHERE id = ?`, id)) + if errors.Is(err, sql.ErrNoRows) { + return KBCollection{}, ErrNotFound + } + return c, err +} + +// DeleteCollection removes a collection, its documents, chunks, grants and +// index. +func (s *DB) DeleteCollection(ctx context.Context, id string) error { + if err := s.KnowledgeAvailable(); err != nil { + return err + } + if !kbIDPattern.MatchString(id) { + return ErrNotFound + } + return s.inTx(ctx, func(c conn) error { + // Chunks go with their documents (a foreign-key cascade, which row + // security does not stand in the way of), documents and grants with + // the collection. + res, err := c.ExecContext(ctx, `DELETE FROM kb_collections WHERE id = ?`, id) + if err != nil { + return err + } + if n, _ := res.RowsAffected(); n == 0 { + return ErrNotFound + } + _, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id)) + return err + }) +} + +// Grant lets a scope read or write a collection; granting again replaces the +// permission. The scope must be in the collection's organization. +func (s *DB) Grant(ctx context.Context, g KBGrant) error { + if err := s.KnowledgeAvailable(); err != nil { + return err + } + if g.Perm != KBRead && g.Perm != KBWrite { + return fmt.Errorf("permission %q: read or write", g.Perm) + } + if g.AddedAt.IsZero() { + g.AddedAt = time.Now() + } + return s.inTx(ctx, func(c conn) error { + var org string + err := c.QueryRowContext(ctx, `SELECT org_id FROM kb_collections WHERE id = ?`, g.Collection).Scan(&org) + if errors.Is(err, sql.ErrNoRows) { + return fmt.Errorf("collection %s: %w", g.Collection, ErrNotFound) + } + if err != nil { + return err + } + got, err := s.orgOfScope(ctx, c, g.Scope, g.ScopeID, org) + if err != nil { + return err + } + if got != org { + return fmt.Errorf("%s %s is in another organization; a collection is never granted outside its own", g.Scope, g.ScopeID) + } + _, err = c.ExecContext(ctx, `INSERT INTO kb_grants (scope, scope_id, collection, perm, added_at, added_by) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT (scope, scope_id, collection) DO UPDATE SET perm = excluded.perm, + added_at = excluded.added_at, added_by = excluded.added_by`, + string(g.Scope), g.ScopeID, g.Collection, string(g.Perm), millis(g.AddedAt), g.AddedBy) + return err + }) +} + +// Revoke removes a grant, reporting whether it was there. +func (s *DB) Revoke(ctx context.Context, scope Scope, scopeID, collection string) (bool, error) { + if err := s.KnowledgeAvailable(); err != nil { + return false, err + } + res, err := s.db.ExecContext(ctx, `DELETE FROM kb_grants WHERE scope = ? AND scope_id = ? AND collection = ?`, + string(scope), scopeID, collection) + if err != nil { + return false, err + } + n, _ := res.RowsAffected() + return n > 0, nil +} + +// Grants lists a collection's grants, or every grant when collection is "". +func (s *DB) Grants(ctx context.Context, collection string) ([]KBGrant, error) { + if err := s.KnowledgeAvailable(); err != nil { + return nil, err + } + q, args := `SELECT scope, scope_id, collection, perm, added_at, added_by FROM kb_grants`, []any(nil) + if collection != "" { + q, args = q+` WHERE collection = ?`, []any{collection} + } + return queryAll(ctx, s.db, q+` ORDER BY collection, scope, scope_id`, args, func(r *sql.Rows) (KBGrant, error) { + var g KBGrant + var scope, perm string + var at int64 + err := r.Scan(&scope, &g.ScopeID, &g.Collection, &perm, &at, &g.AddedBy) + g.Scope, g.Perm, g.AddedAt = Scope(scope), KBPerm(perm), fromMillis(at) + return g, err + }) +} + +// chainScopes is a caller's place in the tree, widest first. +func chainScopes(e Effective) [][2]string { + var out [][2]string + add := func(s Scope, id string) { + if id != "" { + out = append(out, [2]string{string(s), id}) + } + } + add(ScopeOrg, e.OrgID) + add(ScopeTeam, e.TeamID) + add(ScopeProject, e.ProjectID) + add(ScopeUser, e.UserID) + // The chat page's caller is a session, not a key row; its id matches no + // key grant, which is as it should be. + add(ScopeKey, e.KeyID) + return out +} + +// CollectionsFor lists the collections a caller may read, or with write +// those it may change. A caller outside any organization — a personal key +// that named no project — may read none: joining a second team must not +// widen what every key of that person reads. +func (s *DB) CollectionsFor(ctx context.Context, e Effective, write bool) ([]KBCollection, error) { + if err := s.KnowledgeAvailable(); err != nil { + return nil, err + } + if e.OrgID == "" { + return nil, nil + } + chain := chainScopes(e) + var match []string + args := []any{e.OrgID} + for _, sc := range chain { + match = append(match, `(c.owner_scope = ? AND c.owner_id = ?)`) + args = append(args, sc[0], sc[1]) + } + var grant []string + for _, sc := range chain { + grant = append(grant, `(g.scope = ? AND g.scope_id = ?)`) + args = append(args, sc[0], sc[1]) + } + perm := "" + if write { + perm = ` AND g.perm = 'write'` + } + q := `SELECT ` + prefixCols("c.", kbCollectionCols) + ` FROM kb_collections c WHERE c.org_id = ? AND (` + + strings.Join(match, " OR ") + ` OR EXISTS (SELECT 1 FROM kb_grants g WHERE g.collection = c.id` + perm + + ` AND (` + strings.Join(grant, " OR ") + `))) ORDER BY c.name` + return queryAll(ctx, s.db, q, args, func(r *sql.Rows) (KBCollection, error) { return scanCollection(r) }) +} + +func prefixCols(p, cols string) string { + parts := strings.Split(cols, ", ") + for i := range parts { + parts[i] = p + parts[i] + } + return strings.Join(parts, ", ") +} + +// PutDocument adds a document to a collection, or replaces the one of the +// same name, in one transaction: it is searchable whole or not at all. The +// same digest again changes nothing, and reports so. +func (s *DB) PutDocument(ctx context.Context, c KBCollection, d KBDocument, chunks []KBChunk) (KBDocument, bool, error) { + if err := s.KnowledgeAvailable(); err != nil { + return KBDocument{}, false, err + } + if !kbIDPattern.MatchString(c.ID) { + return KBDocument{}, false, fmt.Errorf("collection %s: %w", c.ID, ErrNotFound) + } + for i, ch := range chunks { + if len(ch.Embedding) != c.Dims { + return KBDocument{}, false, fmt.Errorf("chunk %d has %d dimensions; collection %s takes %d", + i, len(ch.Embedding), c.Name, c.Dims) + } + } + d.Collection, d.Chunks = c.ID, len(chunks) + if d.AddedAt.IsZero() { + d.AddedAt = time.Now() + } + changed := true + err := s.inTx(ctx, func(tx conn) error { + var oldID, oldDigest string + err := tx.QueryRowContext(ctx, `SELECT id, digest FROM kb_documents WHERE collection = ? AND name = ?`, + c.ID, d.Name).Scan(&oldID, &oldDigest) + switch { + case err == nil && oldDigest == d.Digest: + changed = false + return nil + case err == nil: + if _, err := tx.ExecContext(ctx, `DELETE FROM kb_documents WHERE id = ?`, oldID); err != nil { + return err + } + case !errors.Is(err, sql.ErrNoRows): + return err + } + d.ID = NewID("doc_") + if _, err := tx.ExecContext(ctx, `INSERT INTO kb_documents + (id, collection, name, digest, bytes, chunks, added_at, added_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, + d.ID, c.ID, d.Name, d.Digest, d.Bytes, d.Chunks, millis(d.AddedAt), d.AddedBy); err != nil { + return err + } + if err := kbScope(ctx, tx, c.OrgID, []string{c.ID}); err != nil { + return err + } + const batch = 100 + for start := 0; start < len(chunks); start += batch { + end := min(start+batch, len(chunks)) + var vals []string + var args []any + for i := start; i < end; i++ { + vals = append(vals, `(?, ?, ?, ?, ?, ?::vector)`) + args = append(args, c.OrgID, c.ID, d.ID, i, chunks[i].Body, vectorText(chunks[i].Embedding)) + } + if _, err := tx.ExecContext(ctx, `INSERT INTO kb_chunks (org_id, collection, doc_id, ord, body, embedding) VALUES `+ + strings.Join(vals, ", "), args...); err != nil { + return err + } + } + return nil + }) + if err != nil { + return KBDocument{}, false, err + } + return d, changed, nil +} + +// vectorText is pgvector's text form of an embedding. +func vectorText(v []float32) string { + var b strings.Builder + b.Grow(len(v) * 10) + b.WriteByte('[') + for i, f := range v { + if i > 0 { + b.WriteByte(',') + } + b.WriteString(strconv.FormatFloat(float64(f), 'g', -1, 32)) + } + b.WriteByte(']') + return b.String() +} + +// Documents lists a collection's documents, by name. +func (s *DB) Documents(ctx context.Context, collection string) ([]KBDocument, error) { + if err := s.KnowledgeAvailable(); err != nil { + return nil, err + } + return queryAll(ctx, s.db, `SELECT id, collection, name, digest, bytes, chunks, added_at, added_by + FROM kb_documents WHERE collection = ? ORDER BY name`, []any{collection}, + func(r *sql.Rows) (KBDocument, error) { + var d KBDocument + var at int64 + err := r.Scan(&d.ID, &d.Collection, &d.Name, &d.Digest, &d.Bytes, &d.Chunks, &at, &d.AddedBy) + d.AddedAt = fromMillis(at) + return d, err + }) +} + +// DeleteDocument removes a document and its chunks, reporting whether it was +// there. +func (s *DB) DeleteDocument(ctx context.Context, collection, name string) (bool, error) { + if err := s.KnowledgeAvailable(); err != nil { + return false, err + } + res, err := s.db.ExecContext(ctx, `DELETE FROM kb_documents WHERE collection = ? AND name = ?`, collection, name) + if err != nil { + return false, err + } + n, _ := res.RowsAffected() + return n > 0, nil +} + +// KBSearch is one search: the caller's organization, the collections it may +// read and asked for, the query embedded by each collection's model, and the +// query's text for the full-text arm. +type KBSearch struct { + OrgID string + Collections []KBCollection + // Embeddings is the query embedded by each model, keyed by model name. + Embeddings map[string][]float32 + Text string + K int +} + +// rrfK is reciprocal rank fusion's constant: 60, as in the paper and nearly +// everywhere it is used. +const rrfK = 60 + +// SearchChunks finds the best chunks for a query across collections: a +// nearest-neighbour search in each collection and a full-text search across +// them, fused by rank. Every collection must be in OrgID; the caller decides +// which ones the requester may read, and the table refuses rows outside them +// regardless. +func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) { + if err := s.KnowledgeAvailable(); err != nil { + return nil, err + } + if q.K <= 0 || len(q.Collections) == 0 || q.OrgID == "" { + return nil, nil + } + ids := make([]string, 0, len(q.Collections)) + quoted := make([]string, 0, len(q.Collections)) + for _, c := range q.Collections { + if c.OrgID != q.OrgID || !kbIDPattern.MatchString(c.ID) { + return nil, fmt.Errorf("collection %s is not in organization %s", c.ID, q.OrgID) + } + ids = append(ids, c.ID) + quoted = append(quoted, sqlLiteral(c.ID)) + } + type key struct { + doc string + ord int + } + hits := map[key]*KBHit{} + add := func(h KBHit, rank int) { + k := key{h.DocID, h.Ord} + if have, ok := hits[k]; ok { + have.Score += 1.0 / float64(rrfK+rank) + return + } + h.Score = 1.0 / float64(rrfK+rank) + hits[k] = &h + } + scan := func(rows *sql.Rows) ([]KBHit, error) { + defer rows.Close() + var out []KBHit + for rows.Next() { + var h KBHit + if err := rows.Scan(&h.Collection, &h.DocID, &h.Document, &h.Ord, &h.Body); err != nil { + return nil, err + } + out = append(out, h) + } + return out, rows.Err() + } + err := s.inTx(ctx, func(tx conn) error { + if err := kbScope(ctx, tx, q.OrgID, ids); err != nil { + return err + } + for _, c := range q.Collections { + v, ok := q.Embeddings[c.EmbedModel] + if !ok { + continue + } + if len(v) != c.Dims { + return fmt.Errorf("the query has %d dimensions; collection %s takes %d", len(v), c.Name, c.Dims) + } + // The literal collection and cast are what let the planner use + // that collection's partial index. + cast := `::vector(` + strconv.Itoa(c.Dims) + `)` + rows, err := tx.QueryContext(ctx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body + FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id + WHERE k.org_id = ? AND k.collection = `+sqlLiteral(c.ID)+` + ORDER BY k.embedding`+cast+` <=> ?`+cast+` LIMIT ?`, + q.OrgID, vectorText(v), q.K) + if err != nil { + return err + } + got, err := scan(rows) + if err != nil { + return err + } + for i, h := range got { + add(h, i+1) + } + } + if tsq := orQuery(q.Text); tsq != "" { + rows, err := tx.QueryContext(ctx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body + FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id, to_tsquery('simple', ?) q + WHERE k.org_id = ? AND k.collection IN (`+strings.Join(quoted, ", ")+`) AND k.tsv @@ q + ORDER BY ts_rank(k.tsv, q) DESC LIMIT ?`, + tsq, q.OrgID, q.K) + if err != nil { + return err + } + got, err := scan(rows) + if err != nil { + return err + } + for i, h := range got { + add(h, i+1) + } + } + return nil + }) + if err != nil { + return nil, err + } + out := make([]KBHit, 0, len(hits)) + for _, h := range hits { + out = append(out, *h) + } + sort.Slice(out, func(i, j int) bool { + if out[i].Score != out[j].Score { + return out[i].Score > out[j].Score + } + if out[i].DocID != out[j].DocID { + return out[i].DocID < out[j].DocID + } + return out[i].Ord < out[j].Ord + }) + if len(out) > q.K { + out = out[:q.K] + } + return out, nil +} + +// orQuery turns a question into a tsquery matching any of its words. All of +// them (plainto_tsquery) would match almost nothing a person types as a +// question; the rank then prefers chunks with more of them. +func orQuery(text string) string { + seen := map[string]bool{} + var words []string + // Not lowercased here: to_tsquery does that with the database's own + // rules, the same ones to_tsvector used on the chunks — Go's lower case + // of a Turkish İ is not the database's. + for _, w := range strings.FieldsFunc(text, func(r rune) bool { + return !unicode.IsLetter(r) && !unicode.IsDigit(r) && !unicode.IsMark(r) + }) { + if len([]rune(w)) < 2 || seen[w] { + continue + } + seen[w] = true + words = append(words, w) + if len(words) == 32 { + break + } + } + return strings.Join(words, " | ") +} + +// kbOwnedBy counts the collections a scope owns, for refusing to delete it +// from under them. Zero on a store without knowledge bases. +func (s *DB) kbOwnedBy(ctx context.Context, c conn, scope Scope, id string) (int, error) { + if s.dialect != PostgresDialect { + return 0, nil + } + var n int + q := `SELECT COUNT(*) FROM kb_collections WHERE owner_scope = ? AND owner_id = ?` + args := []any{string(scope), id} + if scope == ScopeOrg { + q, args = `SELECT COUNT(*) FROM kb_collections WHERE org_id = ?`, []any{id} + } + err := c.QueryRowContext(ctx, q, args...).Scan(&n) + return n, err +} + +// kbForgetScope removes what a deleted scope had: its grants, and for a +// person the collections they owned. +func (s *DB) kbForgetScope(ctx context.Context, c conn, scope Scope, id string) error { + if s.dialect != PostgresDialect { + return nil + } + if _, err := c.ExecContext(ctx, `DELETE FROM kb_grants WHERE scope = ? AND scope_id = ?`, string(scope), id); err != nil { + return err + } + if scope != ScopeUser { + return nil + } + owned, err := queryAll(ctx, c, `SELECT id FROM kb_collections WHERE owner_scope = 'user' AND owner_id = ?`, + []any{id}, func(r *sql.Rows) (string, error) { + var v string + return v, r.Scan(&v) + }) + if err != nil { + return err + } + for _, cid := range owned { + if _, err := c.ExecContext(ctx, `DELETE FROM kb_collections WHERE id = ?`, cid); err != nil { + return err + } + if kbIDPattern.MatchString(cid) { + if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(cid)); err != nil { + return err + } + } + } + return nil +} + +// kbDropPartition drops a deleted organization's chunk partition, empty by +// then: an organization with collections is not deleted. +func (s *DB) kbDropPartition(ctx context.Context, c conn, scope Scope, id string) error { + if s.dialect != PostgresDialect || scope != ScopeOrg || s.kbErr != nil { + return nil + } + _, err := c.ExecContext(ctx, `DROP TABLE IF EXISTS `+kbPartition(id)) + return err +} diff --git a/internal/store/knowledge_test.go b/internal/store/knowledge_test.go new file mode 100644 index 00000000..4c67da93 --- /dev/null +++ b/internal/store/knowledge_test.go @@ -0,0 +1,362 @@ +package store + +import ( + "context" + "errors" + "math" + "strings" + "testing" + "time" +) + +func TestKnowledgeIsOffOnSQLite(t *testing.T) { + s := openDB(t) + tree(t, s) + err := s.KnowledgeAvailable() + if !errors.Is(err, ErrKnowledgeOff) || !strings.Contains(err.Error(), "SQLite") { + t.Fatalf("SQLite store says %v", err) + } + if _, err := s.CreateCollection(context.Background(), KBCollection{ + OwnerScope: ScopeProject, OwnerID: "proj1", Name: "x", EmbedModel: "m", Dims: 3, + }); !errors.Is(err, ErrKnowledgeOff) { + t.Errorf("creating a collection on SQLite gave %v", err) + } + // Deleting from the tree still works without the tables. + if err := s.DeleteProject(context.Background(), "proj1"); err != nil { + t.Errorf("deleting a project on SQLite: %v", err) + } +} + +// kbTwoOrgs is org1 (team1/proj1, team2/proj2) and org2 (team3/proj3). +func kbTwoOrgs(t *testing.T, s *DB) { + t.Helper() + ctx := context.Background() + tree(t, s) + for _, err := range []error{ + s.PutTeam(ctx, Team{ID: "team2", OrgID: "org1", Name: "Finance"}), + s.PutProject(ctx, Project{ID: "proj2", TeamID: "team2", Name: "Payroll"}), + s.PutOrg(ctx, Org{ID: "org2", Name: "Other"}), + s.PutTeam(ctx, Team{ID: "team3", OrgID: "org2", Name: "Research"}), + s.PutProject(ctx, Project{ID: "proj3", TeamID: "team3", Name: "Chat"}), + } { + if err != nil { + t.Fatal(err) + } + } +} + +// unit is a 3-dimensional unit vector pointing mostly along axis i. +func unit(i int) []float32 { + v := []float32{0.1, 0.1, 0.1} + v[i] = 1 + n := float32(math.Sqrt(float64(v[0]*v[0] + v[1]*v[1] + v[2]*v[2]))) + for j := range v { + v[j] /= n + } + return v +} + +func mustCollection(t *testing.T, s *DB, scope Scope, owner, org, name string) KBCollection { + t.Helper() + c, err := s.CreateCollection(context.Background(), KBCollection{ + OwnerScope: scope, OwnerID: owner, OrgID: org, Name: name, EmbedModel: "emb", Dims: 3, + }) + if err != nil { + t.Fatalf("creating %s: %v", name, err) + } + return c +} + +func mustDoc(t *testing.T, s *DB, c KBCollection, name string, bodies ...string) { + t.Helper() + var chunks []KBChunk + for i, b := range bodies { + chunks = append(chunks, KBChunk{Body: b, Embedding: unit(i % 3)}) + } + if _, _, err := s.PutDocument(context.Background(), c, KBDocument{Name: name, Digest: name + "-1", Bytes: 10}, chunks); err != nil { + t.Fatalf("adding %s: %v", name, err) + } +} + +func openKB(t *testing.T) *DB { + t.Helper() + s := openPG(t) + if err := s.KnowledgeAvailable(); err != nil { + t.Skipf("the test database has no pgvector: %v", err) + } + return s +} + +func names(cs []KBCollection) string { + var out []string + for _, c := range cs { + out = append(out, c.Name) + } + return strings.Join(out, ",") +} + +func TestPostgresCollectionsReachOnlyTheirChain(t *testing.T) { + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + mustCollection(t, s, ScopeOrg, "org1", "", "handbook") + mustCollection(t, s, ScopeTeam, "team1", "", "research") + pay := mustCollection(t, s, ScopeProject, "proj2", "", "payroll") + mustCollection(t, s, ScopeProject, "proj3", "", "other-org") + + read := func(e Effective) string { + t.Helper() + got, err := s.CollectionsFor(ctx, e, false) + if err != nil { + t.Fatal(err) + } + return names(got) + } + inProj1 := Effective{KeyID: "k1", ProjectID: "proj1", TeamID: "team1", OrgID: "org1"} + if got := read(inProj1); got != "handbook,research" { + t.Errorf("proj1 reads %q", got) + } + if got := read(Effective{KeyID: "k3", ProjectID: "proj3", TeamID: "team3", OrgID: "org2"}); got != "other-org" { + t.Errorf("org2 reads %q", got) + } + // A personal key with no project reads nothing, whatever its person + // belongs to. + if got := read(Effective{KeyID: "kp", UserID: "u1"}); got != "" { + t.Errorf("a caller outside any organization reads %q", got) + } + + // A grant to a key adds that collection for that key alone. + _, k := mustKey(t, s, "k1", "proj1") + if err := s.Grant(ctx, KBGrant{Scope: ScopeKey, ScopeID: k.ID, Collection: pay.ID, Perm: KBRead}); err != nil { + t.Fatal(err) + } + if got := read(inProj1); got != "handbook,payroll,research" { + t.Errorf("proj1 with a key grant reads %q", got) + } + if got := read(Effective{KeyID: "k-other", ProjectID: "proj1", TeamID: "team1", OrgID: "org1"}); got != "handbook,research" { + t.Errorf("another key of proj1 reads %q", got) + } + // Read is not write. + w, err := s.CollectionsFor(ctx, inProj1, true) + if err != nil { + t.Fatal(err) + } + if strings.Contains(names(w), "payroll") { + t.Errorf("a read grant gave write: %s", names(w)) + } +} + +func TestPostgresGrantsNeverCrossOrganizations(t *testing.T) { + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + c := mustCollection(t, s, ScopeProject, "proj1", "", "docs") + for _, g := range []KBGrant{ + {Scope: ScopeOrg, ScopeID: "org2"}, + {Scope: ScopeTeam, ScopeID: "team3"}, + {Scope: ScopeProject, ScopeID: "proj3"}, + } { + g.Collection, g.Perm = c.ID, KBRead + if err := s.Grant(ctx, g); err == nil || !strings.Contains(err.Error(), "another organization") { + t.Errorf("granting to %s %s gave %v", g.Scope, g.ScopeID, err) + } + } + _, k3 := mustKey(t, s, "k3", "proj3") + if err := s.Grant(ctx, KBGrant{Scope: ScopeKey, ScopeID: k3.ID, Collection: c.ID, Perm: KBRead}); err == nil { + t.Error("a key of another organization was granted a collection") + } + // A person who belongs to nothing in the organization cannot own or be + // granted one there. + if err := s.CreateUser(ctx, User{ID: "u1", Email: "u1@example.com", Name: "u", PasswordHash: "x", + State: UserActive, CreatedAt: time.Now()}); err != nil { + t.Fatal(err) + } + if err := s.Grant(ctx, KBGrant{Scope: ScopeUser, ScopeID: "u1", Collection: c.ID, Perm: KBRead}); err == nil { + t.Error("a person with no membership in the organization was granted a collection") + } + if err := s.AddMembership(ctx, "u1", ScopeProject, "proj2", "member", "t", time.Now()); err != nil { + t.Fatal(err) + } + if err := s.Grant(ctx, KBGrant{Scope: ScopeUser, ScopeID: "u1", Collection: c.ID, Perm: KBRead}); err != nil { + t.Errorf("a member of a project in the organization: %v", err) + } +} + +func TestPostgresSearchReturnsOnlyAllowedChunks(t *testing.T) { + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + mine := mustCollection(t, s, ScopeProject, "proj1", "", "mine") + secret := mustCollection(t, s, ScopeProject, "proj2", "", "secret") + foreign := mustCollection(t, s, ScopeProject, "proj3", "", "foreign") + mustDoc(t, s, mine, "a.md", "vacation policy is twenty days", "expenses go through finance") + mustDoc(t, s, secret, "salaries.md", "vacation salaries are confidential") + mustDoc(t, s, foreign, "theirs.md", "vacation at the other company") + + hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{mine}, + Embeddings: map[string][]float32{"emb": unit(0)}, Text: "vacation policy", K: 5}) + if err != nil { + t.Fatal(err) + } + if len(hits) == 0 || hits[0].Body != "vacation policy is twenty days" || hits[0].Document != "a.md" { + t.Fatalf("best hit %+v", hits) + } + for _, h := range hits { + if h.Collection != mine.ID { + t.Errorf("a search of %s returned a chunk of %s: %q", mine.Name, h.Collection, h.Body) + } + } + + // A collection of another organization is refused, not searched. + if _, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{mine, foreign}, + Embeddings: map[string][]float32{"emb": unit(0)}, Text: "vacation", K: 5}); err == nil { + t.Error("a search across organizations was run") + } +} + +func TestPostgresRowSecurityHoldsWithoutTheFilter(t *testing.T) { + // The second line: a query that forgets its WHERE gets nothing outside + // the scope its transaction set, and nothing at all without one. + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + mine := mustCollection(t, s, ScopeProject, "proj1", "", "mine") + other := mustCollection(t, s, ScopeProject, "proj2", "", "other") + mustDoc(t, s, mine, "a.md", "one", "two") + mustDoc(t, s, other, "b.md", "three") + + count := func(org string, colls ...string) int { + t.Helper() + var n int + err := s.inTx(ctx, func(c conn) error { + if org != "" { + if err := kbScope(ctx, c, org, colls); err != nil { + return err + } + } + return c.QueryRowContext(ctx, `SELECT COUNT(*) FROM kb_chunks`).Scan(&n) + }) + if err != nil { + t.Fatal(err) + } + return n + } + if n := count(""); n != 0 { + t.Errorf("with no scope set, %d chunks are visible", n) + } + if n := count("org1", mine.ID); n != 2 { + t.Errorf("scoped to one collection, %d chunks are visible", n) + } + if n := count("org2", mine.ID); n != 0 { + t.Errorf("a collection named under the wrong organization shows %d chunks", n) + } + // Nor can a write put a chunk outside its scope. + err := s.inTx(ctx, func(c conn) error { + if err := kbScope(ctx, c, "org1", []string{mine.ID}); err != nil { + return err + } + docs, _ := s.Documents(ctx, other.ID) + _, err := c.ExecContext(ctx, `INSERT INTO kb_chunks (org_id, collection, doc_id, ord, body, embedding) + VALUES ('org1', ?, ?, 9, 'x', '[1,0,0]')`, other.ID, docs[0].ID) + return err + }) + if err == nil || !strings.Contains(err.Error(), "row-level security") { + t.Errorf("writing into another collection gave %v", err) + } +} + +func TestPostgresDocumentsReplaceWholeAndSkipTheSame(t *testing.T) { + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + c := mustCollection(t, s, ScopeProject, "proj1", "", "docs") + put := func(digest string, bodies ...string) bool { + t.Helper() + var chunks []KBChunk + for _, b := range bodies { + chunks = append(chunks, KBChunk{Body: b, Embedding: unit(0)}) + } + _, changed, err := s.PutDocument(ctx, c, KBDocument{Name: "a.md", Digest: digest}, chunks) + if err != nil { + t.Fatal(err) + } + return changed + } + if !put("d1", "old one", "old two") { + t.Error("a new document was not a change") + } + if put("d1", "ignored") { + t.Error("the same digest again was a change") + } + if !put("d2", "new") { + t.Error("a new digest was not a change") + } + hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{c}, + Embeddings: map[string][]float32{"emb": unit(0)}, K: 10}) + if err != nil { + t.Fatal(err) + } + if len(hits) != 1 || hits[0].Body != "new" { + t.Errorf("after replacing, the collection holds %+v", hits) + } + if _, _, err := s.PutDocument(ctx, c, KBDocument{Name: "b.md", Digest: "x"}, + []KBChunk{{Body: "x", Embedding: []float32{1, 0}}}); err == nil { + t.Error("a chunk of the wrong dimension was accepted") + } +} + +func TestPostgresScopesWithCollectionsAreNotDeleted(t *testing.T) { + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + c := mustCollection(t, s, ScopeProject, "proj2", "", "docs") + mustDoc(t, s, c, "a.md", "x") + if err := s.DeleteProject(ctx, "proj2"); !errors.Is(err, ErrNotEmpty) || !strings.Contains(err.Error(), "collection") { + t.Errorf("deleting a project with a collection gave %v", err) + } + if err := s.DeleteCollection(ctx, c.ID); err != nil { + t.Fatal(err) + } + if err := s.DeleteProject(ctx, "proj2"); err != nil { + t.Errorf("deleting the project once its collection is gone: %v", err) + } + var n int + if err := s.raw.QueryRowContext(ctx, `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`, + kbVectorIndex(c.ID)).Scan(&n); err != nil || n != 0 { + t.Errorf("the collection's index is left behind (%d, %v)", n, err) + } +} + +func TestPostgresAPersonsCollectionsGoWithTheAccount(t *testing.T) { + ctx := context.Background() + s := openKB(t) + kbTwoOrgs(t, s) + if err := s.CreateUser(ctx, User{ID: "u1", Email: "u1@example.com", Name: "u", PasswordHash: "x", + State: UserActive, CreatedAt: time.Now()}); err != nil { + t.Fatal(err) + } + if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeUser, OwnerID: "u1", OrgID: "org1", + Name: "notes", EmbedModel: "emb", Dims: 3}); err == nil { + t.Error("a person outside the organization made a collection in it") + } + if err := s.AddMembership(ctx, "u1", ScopeTeam, "team1", "member", "t", time.Now()); err != nil { + t.Fatal(err) + } + c := mustCollection(t, s, ScopeUser, "u1", "org1", "notes") + mustDoc(t, s, c, "n.md", "x") + if err := s.DeleteUser(ctx, "u1", time.Now()); err != nil { + t.Fatal(err) + } + if _, err := s.Collection(ctx, c.ID); !errors.Is(err, ErrNotFound) { + t.Errorf("the person's collection outlived the account: %v", err) + } +} + +func TestOrQueryMatchesAnyWord(t *testing.T) { + if got := orQuery("Yıllık izin kaç gün? İzin & 'x' -- a"); got != "Yıllık | izin | kaç | gün | İzin" { + t.Errorf("orQuery gave %q", got) + } + if got := orQuery("?!"); got != "" { + t.Errorf("no words gave %q", got) + } +} diff --git a/internal/store/migrations/postgres/0035_knowledge.sql b/internal/store/migrations/postgres/0035_knowledge.sql new file mode 100644 index 00000000..ff329e1f --- /dev/null +++ b/internal/store/migrations/postgres/0035_knowledge.sql @@ -0,0 +1,55 @@ +-- Knowledge bases: collections of a company's documents that requests are +-- answered from (BACKLOG F). PostgreSQL only: there is no SQLite file of this +-- number, and on SQLite the feature is off. +-- +-- The chunks themselves are not here. They need the pgvector extension, which +-- only a superuser can create and which a cluster may not have installed; a +-- migration that failed for want of it would stop the control plane from +-- starting at all. The store creates kb_chunks when it opens, if it can +-- (knowledge.go), and says why not when it cannot. + +-- A collection has one owner in the tenancy tree and lives inside one +-- organization, which no grant can cross. +CREATE TABLE kb_collections ( + id TEXT PRIMARY KEY, + org_id TEXT NOT NULL REFERENCES organizations(id), + owner_scope TEXT NOT NULL, + owner_id TEXT NOT NULL, + name TEXT NOT NULL, + -- Vectors from two models are not comparable, so a collection is embedded + -- by one model and its dimension is fixed with it. + embed_model TEXT NOT NULL, + dims INTEGER NOT NULL, + created_at BIGINT NOT NULL, + created_by TEXT NOT NULL DEFAULT '', + UNIQUE (org_id, name) +); +CREATE INDEX kb_collections_by_owner ON kb_collections(owner_scope, owner_id); + +-- Who else may read or write a collection. Like guard_attachments, keyed by +-- scope with everything below the scope included; unlike guard policies, +-- grants add up rather than narrow. +CREATE TABLE kb_grants ( + scope TEXT NOT NULL, + scope_id TEXT NOT NULL, + collection TEXT NOT NULL REFERENCES kb_collections(id) ON DELETE CASCADE, + perm TEXT NOT NULL, + added_at BIGINT NOT NULL, + added_by TEXT NOT NULL DEFAULT '', + PRIMARY KEY (scope, scope_id, collection) +); +CREATE INDEX kb_grants_by_collection ON kb_grants(collection); + +-- A document as added, by name within its collection. The digest is what +-- decides whether adding it again changes anything. +CREATE TABLE kb_documents ( + id TEXT PRIMARY KEY, + collection TEXT NOT NULL REFERENCES kb_collections(id) ON DELETE CASCADE, + name TEXT NOT NULL, + digest TEXT NOT NULL, + bytes BIGINT NOT NULL, + chunks INTEGER NOT NULL, + added_at BIGINT NOT NULL, + added_by TEXT NOT NULL DEFAULT '', + UNIQUE (collection, name) +); diff --git a/internal/store/postgres.go b/internal/store/postgres.go index 7c1ef324..a17779d6 100644 --- a/internal/store/postgres.go +++ b/internal/store/postgres.go @@ -55,9 +55,13 @@ func OpenPostgres(ctx context.Context, dsn string) (*DB, error) { _ = db.Close() return nil, err } - return &DB{ + s := &DB{ raw: db, dialect: PostgresDialect, db: bound{c: PostgresDialect, to: db}, - }, nil + } + // Not fatal: a store without pgvector serves everything but knowledge + // bases, and says why when somebody asks for one. + s.kbErr = s.ensureKnowledge(ctx) + return s, nil } diff --git a/internal/store/sqlite.go b/internal/store/sqlite.go index f4bd8b29..74a6005b 100644 --- a/internal/store/sqlite.go +++ b/internal/store/sqlite.go @@ -48,6 +48,10 @@ type DB struct { // personal is what a person's own use is held to when nobody wrote a // limit for them; see SetPersonalDefaults. personal *Limits + + // kbErr is why this store cannot hold knowledge bases, nil when it can; + // set once at open (knowledge.go). + kbErr error } // OpenSQLite opens (creating if needed) a database and brings it up to date.