Knowledge bases in the store: collections, grants and chunks kept apart per tenant

Collections owned by an organization, team, project or person; grants that
add up down the tree and never cross an organization; chunks in PostgreSQL
with pgvector, partitioned by organization, an HNSW index per collection, and
row-level security as a second line behind each query's filter. Search fuses
nearest-neighbour and full-text ranks. Off on SQLite and without pgvector,
saying why. server init and the HA database setup install and enable it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 12:47:57 +02:00
1 parent 0a7fb821d8
commit d35630defe
11 files changed
+1408 -25

No files matched your search

+35 -19
View File
@@ -829,25 +829,41 @@ ACL yet); a personal key or session names a project or team and gets that
chain only, never the union of its person's memberships; PostgreSQL store
only, off on SQLite with a sentence saying why.
### F1. Store: collections, grants, chunks (M)
- `pgvector` ≥ 0.8 installed on every Patroni member by `server init`
(`internal/cli/serverinit_postgres.go`), `CREATE EXTENSION` in the
migration; the SQLite migration has no tables, and the store reports the
feature unavailable.
- Tables: `kb_collections` (id, org_id, owner `team:…`/`project:…`/`user:…`,
name, embed_model, dims, created_at), `kb_grants` (scope, scope_id,
collection, perm read|write — like `guard_attachments`), `kb_documents`
(name, digest, bytes, state, chunk count), `kb_chunks` (org_id,
collection, doc_id, ord, body, tsv, embedding) `PARTITION BY LIST
(org_id)`, a partition per organization, HNSW and GIN indexes per
partition.
- Row-level security on `kb_chunks` against `current_setting('llmgw.org')`
and `llmgw.collections`, retrieval through its own role.
- Rules, refused before anything is written: a grant never crosses
organizations; owner and grants resolve to existing scopes; deleting a
project/team/org and `users forget` delete their collections (`forget.go`).
- Done: store tests on PostgreSQL, including an RLS test that a query
without the settings returns no rows.
### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
(read|write, any scope, never across organizations; a key needs a
project; a person needs a membership in the organization),
`kb_documents` (digest: the same again is no change, a new one replaces
in one transaction).
- `kb_chunks` is created when the store opens, not in a migration (a
migration needing an extension would stop the control plane; one that
skipped it would never run again): partitioned by organization (hashed
partition name), `tsv` generated (`simple`), GIN on the parent, one
partial HNSW index per collection on its organization's partition
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
20k rows: partition pruned, that index used, policy applied.
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
(set with set_config per transaction), one role — the table owner — not
a separate one. Chunks are deleted by the foreign-key cascade from their
document, which RLS does not block.
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
no organization (personal key without a project) reads nothing.
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
the partial index is used) plus a full-text arm with any word
(`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
- Deleting a project/team/org that owns collections is refused like one
with members (changed from "deleted with them": every other delete in the
tree refuses); DeleteUser/ForgetUser delete the person's collections;
deleting an organization drops its partition; grants to a deleted scope go.
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
it; pgha installs it on every member at setup and enables it on the
leader on every run. Neither stops on failure.
Left: a cluster set up before (the PoC) gets no package from a rerun of
pgha on a running cluster — install by hand or add an upgrade step;
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).
### F2. Ingest (M)
- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add
+17
View File
@@ -23,6 +23,23 @@ is only released once its **exit criterion** is met, not when its code lands.
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
- **Knowledge bases, the store's part** (BACKLOG F1, branch `rag`):
collections of documents owned by an organization, team, project or
person, grants that add up down the tenancy tree and never cross an
organization, chunks in PostgreSQL with pgvector — partitioned by
organization, an HNSW index per collection, row-level security on the
table as a second line behind every query's own filter — and a search
that fuses nearest-neighbour and full-text ranks. Off on SQLite, and on
PostgreSQL without pgvector, saying why. `server init` and the
high-availability database setup install pgvector and enable it; a
cluster set up before needs `postgresql-<version>-pgvector` on every
database machine and `CREATE EXTENSION vector` in llmgw's database. A
project, team or organization that owns collections is not deleted; a
person's collections go with their account. Nothing reads or writes them
yet from a command or a request.
## v0.13.0 — 2026-10-08
The whole system stopped, started and restarted from one command, a
+60 -2
View File
@@ -370,6 +370,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
}
in.setupPgvector(ctx, discovery.DiscoverHost())
// Proof the address works, before anything is moved into it.
address := "@" + in.dsnPath()
s, err := store.Open(ctx, address)
@@ -482,8 +484,17 @@ func lastLines(b []byte, n int) string {
// psql runs one statement as the postgres superuser, over the local socket.
func (in *serverInit) psql(ctx context.Context, query string) (string, error) {
cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--",
in.pg.tool("psql"), "-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query)
return in.psqlIn(ctx, "", query)
}
// psqlIn runs a statement as postgres in one database, or in postgres's
// default one when db is "".
func (in *serverInit) psqlIn(ctx context.Context, db, query string) (string, error) {
args := []string{"-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query}
if db != "" {
args = append(args, "-d", db)
}
cmd := exec.CommandContext(ctx, "runuser", append([]string{"-u", "postgres", "--", in.pg.tool("psql")}, args...)...)
// postgres cannot enter root's home, and psql says so on every call.
cmd.Dir = "/"
out, err := cmd.CombinedOutput()
@@ -538,3 +549,50 @@ func (in *serverInit) stopService(ctx context.Context) (bool, error) {
in.done("llmgw-server stopped while its store moves")
return true, nil
}
// pgvectorPackages are the package names of pgvector for a PostgreSQL major
// version: Debian's and Ubuntu's (and PGDG's apt repository) name it after the
// server, PGDG's rpm repository after the version alone.
func pgvectorPackages(major int) map[string][]string {
return map[string][]string{
"deb": {fmt.Sprintf("postgresql-%d-pgvector", major)},
"rpm": {fmt.Sprintf("pgvector_%d", major)},
}
}
// setupPgvector installs pgvector and enables it in llmgw's database, for
// knowledge bases. Not a reason to stop: without it everything else works,
// and the store says why knowledge bases are off when somebody asks for one.
func (in *serverInit) setupPgvector(ctx context.Context, h discovery.Host) {
warn := func(format string, a ...any) {
fmt.Fprintf(in.out, " !! pgvector: "+format+"; knowledge bases stay off\n", a...)
}
if on, err := in.psqlIn(ctx, pgDatabase, "SELECT 1 FROM pg_extension WHERE extname = 'vector'"); err == nil && on == "1" {
in.kept("pgvector in database %s", pgDatabase)
return
}
have, err := in.psqlBool(ctx, "SELECT 1 FROM pg_available_extensions WHERE name = 'vector'")
if err != nil {
warn("%v", err)
return
}
if !have {
num, err := in.psql(ctx, "SHOW server_version_num")
n, convErr := strconv.Atoi(num)
if err != nil || convErr != nil || h.Family() == "" {
warn("cannot tell which package to install (version %q, %s)", num, h.OSID)
return
}
pkgs := pgvectorPackages(n / 10000)
pi := &provision.Installer{Host: h, Log: slog.Default()}
if err := pi.InstallPackages(ctx, pkgs); err != nil {
warn("installing %s: %v", strings.Join(pkgs[h.Family()], " "), err)
return
}
}
if _, err := in.psqlIn(ctx, pgDatabase, "CREATE EXTENSION IF NOT EXISTS vector"); err != nil {
warn("%v", err)
return
}
in.done("pgvector enabled in database %s", pgDatabase)
}
+5
View File
@@ -159,6 +159,11 @@ func TestARunningClusterIsLeftAsItIs(t *testing.T) {
if !a.ranAny("CREATE ROLE") {
t.Error("the role was not made sure of on the leader")
}
// Enabled where the package is there, which on a running cluster is
// all a rerun does about it: it installs nothing.
if !a.ranAny("CREATE EXTENSION IF NOT EXISTS vector") {
t.Error("pgvector was not enabled in llmgw's database on the leader")
}
}
func TestTwoDatabaseMachinesWithoutAThirdVoteAreRefused(t *testing.T) {
+17 -1
View File
@@ -502,6 +502,14 @@ if [ -n "$need" ]; then
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y $need >/dev/null
fi
v=$(ls /usr/lib/postgresql | sort -n | tail -1)
# pgvector, for knowledge bases: on every member, since any of them may lead.
# Not a reason to stop; the store says knowledge bases are off without it.
if [ ! -f /usr/share/postgresql/$v/extension/vector.control ]; then
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y postgresql-$v-pgvector >/dev/null 2>&1 ||
{ sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null 2>&1 &&
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y postgresql-$v-pgvector >/dev/null 2>&1; } ||
echo "no-pgvector"
fi
if [ -d /var/lib/postgresql/$v/main ]; then
if sudo systemctl is-active --quiet postgresql@$v-main; then
n=$(sudo -u postgres psql -Atc "select count(*) from pg_database where datname not in ('postgres','template0','template1')")
@@ -553,13 +561,21 @@ sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL'
SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec
ALTER ROLE llmgw PASSWORD :'pw';
SELECT format('CREATE DATABASE %I OWNER %I', 'llmgw', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = 'llmgw') \gexec
SQL`, nil)
SQL
sudo -u postgres psql -q -d llmgw -v ON_ERROR_STOP=1 <<'SQL' || echo "pgvector-failed"
SELECT 'CREATE EXTENSION IF NOT EXISTS vector' WHERE EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'vector') \gexec
SQL
sudo -u postgres psql -d llmgw -Atc "select 'vector=' || count(*) from pg_extension where extname = 'vector'"`, nil)
if err != nil {
return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out)))
}
// Said as a change only when something was made: a rerun that finds
// both is not a reason to restart the gateway.
s.emit("database", !strings.Contains(string(out), "had=2"), "role and database %s on %s", Database, leader.Name)
if !strings.Contains(string(out), "vector=1") {
s.emit("pgvector", false, "pgvector is not enabled in %s, so knowledge bases stay off "+
"(install postgresql-<version>-pgvector on every member and run setup again)", Database)
}
return nil
}
+14 -1
View File
@@ -214,6 +214,12 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check
busy = append(busy, plural(n, ch.noun))
}
}
// Not an emptyCheck: the table is there only on PostgreSQL.
if n, err := s.kbOwnedBy(ctx, c, scope, id); err != nil {
return err
} else if n > 0 {
busy = append(busy, plural(n, "collection"))
}
if len(busy) > 0 {
return fmt.Errorf("%w: it still has %s", ErrNotEmpty, strings.Join(busy, " and "))
}
@@ -222,6 +228,9 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check
string(scope), string(MemberScope(scope)), id); err != nil {
return err
}
if err := s.kbForgetScope(ctx, c, scope, id); err != nil {
return err
}
res, err := c.ExecContext(ctx, del, id)
if err != nil {
return err
@@ -229,7 +238,7 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check
if n, _ := res.RowsAffected(); n == 0 {
return ErrNotFound
}
return nil
return s.kbDropPartition(ctx, c, scope, id)
})
}
@@ -248,6 +257,10 @@ func (s *DB) DeleteUser(ctx context.Context, userID string, at time.Time) error
millis(at), userID); err != nil {
return err
}
// A person's own collections go with the account.
if err := s.kbForgetScope(ctx, c, ScopeUser, userID); err != nil {
return err
}
res, err := c.ExecContext(ctx, `DELETE FROM users WHERE id = ?`, userID)
if err != nil {
return err
+833
View File
@@ -0,0 +1,833 @@
package store
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/hex"
"errors"
"fmt"
"regexp"
"sort"
"strconv"
"strings"
"time"
"unicode"
)
// Knowledge bases (BACKLOG F): collections of documents, split into chunks
// and embedded, that a request is answered from.
//
// Which tenant may read which chunk is decided twice. The queries here filter
// by organization and by the collections the caller may read, and the table
// enforces the same with row-level security against two settings each
// transaction makes — so a query that forgot its filter returns nothing
// rather than another tenant's passages. The role llmgw connects as owns the
// table, so the policy is FORCEd; a superuser would bypass it, which is one
// more reason the store never connects as one.
// ErrKnowledgeOff is a store that cannot hold knowledge bases.
var ErrKnowledgeOff = errors.New("knowledge bases are off")
// MaxEmbeddingDims is the most dimensions pgvector can index (HNSW on vector).
const MaxEmbeddingDims = 2000
// KBPerm is what a grant allows.
type KBPerm string
// The permissions. Write includes read.
const (
KBRead KBPerm = "read"
KBWrite KBPerm = "write"
)
// KBCollection is a set of documents with one owner and one embedding model.
type KBCollection struct {
ID string
OrgID string
OwnerScope Scope
OwnerID string
Name string
EmbedModel string
Dims int
CreatedAt time.Time
CreatedBy string
}
// KBGrant lets a scope, and everything below it, read or write a collection.
type KBGrant struct {
Scope Scope
ScopeID string
Collection string
Perm KBPerm
AddedAt time.Time
AddedBy string
}
// KBDocument is a document as added to a collection.
type KBDocument struct {
ID string
Collection string
Name string
Digest string
Bytes int64
Chunks int
AddedAt time.Time
AddedBy string
}
// KBChunk is a piece of a document and its embedding.
type KBChunk struct {
Body string
Embedding []float32
}
// KBHit is a chunk a search returned, best first.
type KBHit struct {
Collection string
DocID string
Document string
Ord int
Body string
// Score is the fused rank (reciprocal rank fusion over the vector and
// text searches): only the order means anything.
Score float64
}
// KnowledgeAvailable says whether this store can hold knowledge bases, and
// why not when it cannot.
func (s *DB) KnowledgeAvailable() error {
if s.dialect != PostgresDialect {
return fmt.Errorf("%w: they need the PostgreSQL store (with pgvector), and this one is SQLite", ErrKnowledgeOff)
}
return s.kbErr
}
// ensureKnowledge creates kb_chunks when the vector extension allows it.
//
// At open rather than in a migration: a migration that needs an extension the
// database servers do not have would stop the control plane from starting,
// and one that skipped it would be recorded as applied and never run again
// once the extension arrived.
func (s *DB) ensureKnowledge(ctx context.Context) error {
var have bool
if err := s.db.QueryRowContext(ctx,
`SELECT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'vector')`).Scan(&have); err != nil {
return fmt.Errorf("%w: reading the database's extensions: %v", ErrKnowledgeOff, err)
}
if !have {
var available bool
if err := s.db.QueryRowContext(ctx,
`SELECT EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'vector')`).Scan(&available); err != nil {
return fmt.Errorf("%w: reading the database's extensions: %v", ErrKnowledgeOff, err)
}
if !available {
return fmt.Errorf("%w: pgvector is not installed on the database servers "+
"(package postgresql-<version>-pgvector on every one; llmgw's database setup installs it)", ErrKnowledgeOff)
}
// Works only where the extension is trusted or llmgw's role may
// create it; elsewhere a superuser has to, once.
if _, err := s.db.ExecContext(ctx, `CREATE EXTENSION IF NOT EXISTS vector`); err != nil {
return fmt.Errorf("%w: pgvector is installed but not enabled in llmgw's database; "+
"as postgres, run CREATE EXTENSION vector in it (%v)", ErrKnowledgeOff, err)
}
}
var exists bool
if err := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); err != nil {
return fmt.Errorf("%w: %v", ErrKnowledgeOff, err)
}
if exists {
return nil
}
err := s.inTx(ctx, func(c conn) error {
for _, q := range []string{
// Partitioned by organization: each one's indexes stay its own
// size, and an organization's chunks go with one DROP.
`CREATE TABLE kb_chunks (
org_id TEXT NOT NULL,
collection TEXT NOT NULL,
doc_id TEXT NOT NULL REFERENCES kb_documents(id) ON DELETE CASCADE,
ord INTEGER NOT NULL,
body TEXT NOT NULL,
tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', body)) STORED,
-- No dimension here: collections differ. Each collection's
-- index casts to its own (kbVectorIndex).
embedding vector NOT NULL,
PRIMARY KEY (org_id, doc_id, ord)
) PARTITION BY LIST (org_id)`,
`CREATE INDEX kb_chunks_tsv ON kb_chunks USING gin (tsv)`,
`ALTER TABLE kb_chunks ENABLE ROW LEVEL SECURITY`,
`ALTER TABLE kb_chunks FORCE ROW LEVEL SECURITY`,
`CREATE POLICY kb_scope ON kb_chunks
USING (org_id = current_setting('llmgw.kb_org', true)
AND collection = ANY (string_to_array(current_setting('llmgw.kb_collections', true), ',')))
WITH CHECK (org_id = current_setting('llmgw.kb_org', true)
AND collection = ANY (string_to_array(current_setting('llmgw.kb_collections', true), ',')))`,
} {
if _, err := c.ExecContext(ctx, q); err != nil {
return err
}
}
return nil
})
if err != nil {
// Another control plane opening at the same moment made it first.
if e2 := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); e2 == nil && exists {
return nil
}
return fmt.Errorf("%w: creating kb_chunks: %v", ErrKnowledgeOff, err)
}
return nil
}
// kbIDPattern is what a collection id looks like (NewID("kb_")); ids are put
// into DDL and partial-index predicates, so nothing else is let through.
var kbIDPattern = regexp.MustCompile(`^kb_[a-z2-7]{16}$`)
// kbPartition is the chunk partition of an organization. Hashed, because old
// organization ids were slugs of names and could hold anything.
func kbPartition(orgID string) string {
h := sha256.Sum256([]byte(orgID))
return "kb_chunks_" + hex.EncodeToString(h[:12])
}
// kbVectorIndex is the name of a collection's vector index.
func kbVectorIndex(collection string) string {
return "kb_vec_" + strings.TrimPrefix(collection, "kb_")
}
func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" }
// kbScope sets, for this transaction, which rows of kb_chunks it may touch.
func kbScope(ctx context.Context, c conn, orgID string, collections []string) error {
_, err := c.ExecContext(ctx, `SELECT set_config('llmgw.kb_org', ?, true), set_config('llmgw.kb_collections', ?, true)`,
orgID, strings.Join(collections, ","))
return err
}
// orgOfScope is the organization a scope sits in. A person has none of their
// own, so a person is checked against org instead: they must be a member of
// it, or of something in it.
func (s *DB) orgOfScope(ctx context.Context, c conn, scope Scope, id, org string) (string, error) {
var q string
switch scope {
case ScopeOrg:
q = `SELECT id FROM organizations WHERE id = ?`
case ScopeTeam:
q = `SELECT org_id FROM teams WHERE id = ?`
case ScopeProject:
q = `SELECT t.org_id FROM projects p JOIN teams t ON t.id = p.team_id WHERE p.id = ?`
case ScopeKey:
q = `SELECT t.org_id FROM api_keys k JOIN projects p ON p.id = k.project_id
JOIN teams t ON t.id = p.team_id WHERE k.id = ?`
case ScopeUser:
var n int
err := c.QueryRowContext(ctx, `SELECT COUNT(*) FROM memberships m WHERE m.user_id = ? AND (
(m.scope_type = 'org' AND m.scope_id = ?)
OR (m.scope_type = 'team' AND m.scope_id IN (SELECT id FROM teams WHERE org_id = ?))
OR (m.scope_type = 'project' AND m.scope_id IN (
SELECT p.id FROM projects p JOIN teams t ON t.id = p.team_id WHERE t.org_id = ?)))`,
id, org, org, org).Scan(&n)
if err != nil {
return "", err
}
if n == 0 {
return "", fmt.Errorf("%s is not a member of anything in that organization", id)
}
return org, nil
default:
return "", fmt.Errorf("a collection cannot be owned by or granted to a %s", scope)
}
var got string
err := c.QueryRowContext(ctx, q, id).Scan(&got)
if errors.Is(err, sql.ErrNoRows) {
if scope == ScopeKey {
return "", fmt.Errorf("no key %s in a project (a personal key reads no collection)", id)
}
return "", fmt.Errorf("no %s %s: %w", scope, id, ErrNotFound)
}
return got, err
}
// CreateCollection makes an empty collection, its organization's chunk
// partition when this is the first, and its vector index. c.OrgID is needed
// only when a person owns it; otherwise it comes from the owner.
func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection, error) {
if err := s.KnowledgeAvailable(); err != nil {
return KBCollection{}, err
}
c.Name = strings.TrimSpace(c.Name)
switch {
case c.Name == "":
return KBCollection{}, errors.New("a collection needs a name")
case c.EmbedModel == "":
return KBCollection{}, errors.New("a collection needs the embedding model its documents are embedded with")
case c.Dims < 1 || c.Dims > MaxEmbeddingDims:
return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, MaxEmbeddingDims)
case c.OwnerScope == ScopeUser && c.OrgID == "":
return KBCollection{}, errors.New("a person's collection names the organization it belongs to")
case c.OwnerScope == ScopeKey:
return KBCollection{}, errors.New("a key cannot own a collection; its project or its person can")
}
c.ID = NewID("kb_")
if c.CreatedAt.IsZero() {
c.CreatedAt = time.Now()
}
err := s.inTx(ctx, func(tx conn) error {
org, err := s.orgOfScope(ctx, tx, c.OwnerScope, c.OwnerID, c.OrgID)
if err != nil {
return err
}
if c.OrgID != "" && c.OrgID != org {
return fmt.Errorf("the owner is in organization %s, not %s", org, c.OrgID)
}
c.OrgID = org
var n int
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM kb_collections WHERE org_id = ? AND name = ?`,
c.OrgID, c.Name).Scan(&n); err != nil {
return err
}
if n > 0 {
return fmt.Errorf("%w: collection %q", ErrDuplicateName, c.Name)
}
if _, err := tx.ExecContext(ctx, `INSERT INTO kb_collections
(id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims,
millis(c.CreatedAt), c.CreatedBy); err != nil {
return err
}
part := kbPartition(c.OrgID)
if _, err := tx.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS `+part+
` PARTITION OF kb_chunks FOR VALUES IN (`+sqlLiteral(c.OrgID)+`)`); err != nil {
return err
}
// A partial index per collection, on its organization's partition
// only: the dimension is the collection's, and a search, which
// names one collection at a time, uses exactly this one.
_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
` USING hnsw ((embedding::vector(`+strconv.Itoa(c.Dims)+`)) vector_cosine_ops)`+
` WHERE collection = `+sqlLiteral(c.ID))
return err
})
if err != nil {
return KBCollection{}, err
}
return c, nil
}
const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by`
func scanCollection(sc scanner) (KBCollection, error) {
var c KBCollection
var scope string
var at int64
err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &at, &c.CreatedBy)
c.OwnerScope, c.CreatedAt = Scope(scope), fromMillis(at)
return c, err
}
// Collections lists every collection, by organization and name.
func (s *DB) Collections(ctx context.Context) ([]KBCollection, error) {
if err := s.KnowledgeAvailable(); err != nil {
return nil, err
}
return queryAll(ctx, s.db, `SELECT `+kbCollectionCols+` FROM kb_collections ORDER BY org_id, name`, nil,
func(r *sql.Rows) (KBCollection, error) { return scanCollection(r) })
}
// Collection reads one collection.
func (s *DB) Collection(ctx context.Context, id string) (KBCollection, error) {
if err := s.KnowledgeAvailable(); err != nil {
return KBCollection{}, err
}
c, err := scanCollection(s.db.QueryRowContext(ctx, `SELECT `+kbCollectionCols+` FROM kb_collections WHERE id = ?`, id))
if errors.Is(err, sql.ErrNoRows) {
return KBCollection{}, ErrNotFound
}
return c, err
}
// DeleteCollection removes a collection, its documents, chunks, grants and
// index.
func (s *DB) DeleteCollection(ctx context.Context, id string) error {
if err := s.KnowledgeAvailable(); err != nil {
return err
}
if !kbIDPattern.MatchString(id) {
return ErrNotFound
}
return s.inTx(ctx, func(c conn) error {
// Chunks go with their documents (a foreign-key cascade, which row
// security does not stand in the way of), documents and grants with
// the collection.
res, err := c.ExecContext(ctx, `DELETE FROM kb_collections WHERE id = ?`, id)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrNotFound
}
_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id))
return err
})
}
// Grant lets a scope read or write a collection; granting again replaces the
// permission. The scope must be in the collection's organization.
func (s *DB) Grant(ctx context.Context, g KBGrant) error {
if err := s.KnowledgeAvailable(); err != nil {
return err
}
if g.Perm != KBRead && g.Perm != KBWrite {
return fmt.Errorf("permission %q: read or write", g.Perm)
}
if g.AddedAt.IsZero() {
g.AddedAt = time.Now()
}
return s.inTx(ctx, func(c conn) error {
var org string
err := c.QueryRowContext(ctx, `SELECT org_id FROM kb_collections WHERE id = ?`, g.Collection).Scan(&org)
if errors.Is(err, sql.ErrNoRows) {
return fmt.Errorf("collection %s: %w", g.Collection, ErrNotFound)
}
if err != nil {
return err
}
got, err := s.orgOfScope(ctx, c, g.Scope, g.ScopeID, org)
if err != nil {
return err
}
if got != org {
return fmt.Errorf("%s %s is in another organization; a collection is never granted outside its own", g.Scope, g.ScopeID)
}
_, err = c.ExecContext(ctx, `INSERT INTO kb_grants (scope, scope_id, collection, perm, added_at, added_by)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (scope, scope_id, collection) DO UPDATE SET perm = excluded.perm,
added_at = excluded.added_at, added_by = excluded.added_by`,
string(g.Scope), g.ScopeID, g.Collection, string(g.Perm), millis(g.AddedAt), g.AddedBy)
return err
})
}
// Revoke removes a grant, reporting whether it was there.
func (s *DB) Revoke(ctx context.Context, scope Scope, scopeID, collection string) (bool, error) {
if err := s.KnowledgeAvailable(); err != nil {
return false, err
}
res, err := s.db.ExecContext(ctx, `DELETE FROM kb_grants WHERE scope = ? AND scope_id = ? AND collection = ?`,
string(scope), scopeID, collection)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// Grants lists a collection's grants, or every grant when collection is "".
func (s *DB) Grants(ctx context.Context, collection string) ([]KBGrant, error) {
if err := s.KnowledgeAvailable(); err != nil {
return nil, err
}
q, args := `SELECT scope, scope_id, collection, perm, added_at, added_by FROM kb_grants`, []any(nil)
if collection != "" {
q, args = q+` WHERE collection = ?`, []any{collection}
}
return queryAll(ctx, s.db, q+` ORDER BY collection, scope, scope_id`, args, func(r *sql.Rows) (KBGrant, error) {
var g KBGrant
var scope, perm string
var at int64
err := r.Scan(&scope, &g.ScopeID, &g.Collection, &perm, &at, &g.AddedBy)
g.Scope, g.Perm, g.AddedAt = Scope(scope), KBPerm(perm), fromMillis(at)
return g, err
})
}
// chainScopes is a caller's place in the tree, widest first.
func chainScopes(e Effective) [][2]string {
var out [][2]string
add := func(s Scope, id string) {
if id != "" {
out = append(out, [2]string{string(s), id})
}
}
add(ScopeOrg, e.OrgID)
add(ScopeTeam, e.TeamID)
add(ScopeProject, e.ProjectID)
add(ScopeUser, e.UserID)
// The chat page's caller is a session, not a key row; its id matches no
// key grant, which is as it should be.
add(ScopeKey, e.KeyID)
return out
}
// CollectionsFor lists the collections a caller may read, or with write
// those it may change. A caller outside any organization — a personal key
// that named no project — may read none: joining a second team must not
// widen what every key of that person reads.
func (s *DB) CollectionsFor(ctx context.Context, e Effective, write bool) ([]KBCollection, error) {
if err := s.KnowledgeAvailable(); err != nil {
return nil, err
}
if e.OrgID == "" {
return nil, nil
}
chain := chainScopes(e)
var match []string
args := []any{e.OrgID}
for _, sc := range chain {
match = append(match, `(c.owner_scope = ? AND c.owner_id = ?)`)
args = append(args, sc[0], sc[1])
}
var grant []string
for _, sc := range chain {
grant = append(grant, `(g.scope = ? AND g.scope_id = ?)`)
args = append(args, sc[0], sc[1])
}
perm := ""
if write {
perm = ` AND g.perm = 'write'`
}
q := `SELECT ` + prefixCols("c.", kbCollectionCols) + ` FROM kb_collections c WHERE c.org_id = ? AND (` +
strings.Join(match, " OR ") + ` OR EXISTS (SELECT 1 FROM kb_grants g WHERE g.collection = c.id` + perm +
` AND (` + strings.Join(grant, " OR ") + `))) ORDER BY c.name`
return queryAll(ctx, s.db, q, args, func(r *sql.Rows) (KBCollection, error) { return scanCollection(r) })
}
func prefixCols(p, cols string) string {
parts := strings.Split(cols, ", ")
for i := range parts {
parts[i] = p + parts[i]
}
return strings.Join(parts, ", ")
}
// PutDocument adds a document to a collection, or replaces the one of the
// same name, in one transaction: it is searchable whole or not at all. The
// same digest again changes nothing, and reports so.
func (s *DB) PutDocument(ctx context.Context, c KBCollection, d KBDocument, chunks []KBChunk) (KBDocument, bool, error) {
if err := s.KnowledgeAvailable(); err != nil {
return KBDocument{}, false, err
}
if !kbIDPattern.MatchString(c.ID) {
return KBDocument{}, false, fmt.Errorf("collection %s: %w", c.ID, ErrNotFound)
}
for i, ch := range chunks {
if len(ch.Embedding) != c.Dims {
return KBDocument{}, false, fmt.Errorf("chunk %d has %d dimensions; collection %s takes %d",
i, len(ch.Embedding), c.Name, c.Dims)
}
}
d.Collection, d.Chunks = c.ID, len(chunks)
if d.AddedAt.IsZero() {
d.AddedAt = time.Now()
}
changed := true
err := s.inTx(ctx, func(tx conn) error {
var oldID, oldDigest string
err := tx.QueryRowContext(ctx, `SELECT id, digest FROM kb_documents WHERE collection = ? AND name = ?`,
c.ID, d.Name).Scan(&oldID, &oldDigest)
switch {
case err == nil && oldDigest == d.Digest:
changed = false
return nil
case err == nil:
if _, err := tx.ExecContext(ctx, `DELETE FROM kb_documents WHERE id = ?`, oldID); err != nil {
return err
}
case !errors.Is(err, sql.ErrNoRows):
return err
}
d.ID = NewID("doc_")
if _, err := tx.ExecContext(ctx, `INSERT INTO kb_documents
(id, collection, name, digest, bytes, chunks, added_at, added_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
d.ID, c.ID, d.Name, d.Digest, d.Bytes, d.Chunks, millis(d.AddedAt), d.AddedBy); err != nil {
return err
}
if err := kbScope(ctx, tx, c.OrgID, []string{c.ID}); err != nil {
return err
}
const batch = 100
for start := 0; start < len(chunks); start += batch {
end := min(start+batch, len(chunks))
var vals []string
var args []any
for i := start; i < end; i++ {
vals = append(vals, `(?, ?, ?, ?, ?, ?::vector)`)
args = append(args, c.OrgID, c.ID, d.ID, i, chunks[i].Body, vectorText(chunks[i].Embedding))
}
if _, err := tx.ExecContext(ctx, `INSERT INTO kb_chunks (org_id, collection, doc_id, ord, body, embedding) VALUES `+
strings.Join(vals, ", "), args...); err != nil {
return err
}
}
return nil
})
if err != nil {
return KBDocument{}, false, err
}
return d, changed, nil
}
// vectorText is pgvector's text form of an embedding.
func vectorText(v []float32) string {
var b strings.Builder
b.Grow(len(v) * 10)
b.WriteByte('[')
for i, f := range v {
if i > 0 {
b.WriteByte(',')
}
b.WriteString(strconv.FormatFloat(float64(f), 'g', -1, 32))
}
b.WriteByte(']')
return b.String()
}
// Documents lists a collection's documents, by name.
func (s *DB) Documents(ctx context.Context, collection string) ([]KBDocument, error) {
if err := s.KnowledgeAvailable(); err != nil {
return nil, err
}
return queryAll(ctx, s.db, `SELECT id, collection, name, digest, bytes, chunks, added_at, added_by
FROM kb_documents WHERE collection = ? ORDER BY name`, []any{collection},
func(r *sql.Rows) (KBDocument, error) {
var d KBDocument
var at int64
err := r.Scan(&d.ID, &d.Collection, &d.Name, &d.Digest, &d.Bytes, &d.Chunks, &at, &d.AddedBy)
d.AddedAt = fromMillis(at)
return d, err
})
}
// DeleteDocument removes a document and its chunks, reporting whether it was
// there.
func (s *DB) DeleteDocument(ctx context.Context, collection, name string) (bool, error) {
if err := s.KnowledgeAvailable(); err != nil {
return false, err
}
res, err := s.db.ExecContext(ctx, `DELETE FROM kb_documents WHERE collection = ? AND name = ?`, collection, name)
if err != nil {
return false, err
}
n, _ := res.RowsAffected()
return n > 0, nil
}
// KBSearch is one search: the caller's organization, the collections it may
// read and asked for, the query embedded by each collection's model, and the
// query's text for the full-text arm.
type KBSearch struct {
OrgID string
Collections []KBCollection
// Embeddings is the query embedded by each model, keyed by model name.
Embeddings map[string][]float32
Text string
K int
}
// rrfK is reciprocal rank fusion's constant: 60, as in the paper and nearly
// everywhere it is used.
const rrfK = 60
// SearchChunks finds the best chunks for a query across collections: a
// nearest-neighbour search in each collection and a full-text search across
// them, fused by rank. Every collection must be in OrgID; the caller decides
// which ones the requester may read, and the table refuses rows outside them
// regardless.
func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
if err := s.KnowledgeAvailable(); err != nil {
return nil, err
}
if q.K <= 0 || len(q.Collections) == 0 || q.OrgID == "" {
return nil, nil
}
ids := make([]string, 0, len(q.Collections))
quoted := make([]string, 0, len(q.Collections))
for _, c := range q.Collections {
if c.OrgID != q.OrgID || !kbIDPattern.MatchString(c.ID) {
return nil, fmt.Errorf("collection %s is not in organization %s", c.ID, q.OrgID)
}
ids = append(ids, c.ID)
quoted = append(quoted, sqlLiteral(c.ID))
}
type key struct {
doc string
ord int
}
hits := map[key]*KBHit{}
add := func(h KBHit, rank int) {
k := key{h.DocID, h.Ord}
if have, ok := hits[k]; ok {
have.Score += 1.0 / float64(rrfK+rank)
return
}
h.Score = 1.0 / float64(rrfK+rank)
hits[k] = &h
}
scan := func(rows *sql.Rows) ([]KBHit, error) {
defer rows.Close()
var out []KBHit
for rows.Next() {
var h KBHit
if err := rows.Scan(&h.Collection, &h.DocID, &h.Document, &h.Ord, &h.Body); err != nil {
return nil, err
}
out = append(out, h)
}
return out, rows.Err()
}
err := s.inTx(ctx, func(tx conn) error {
if err := kbScope(ctx, tx, q.OrgID, ids); err != nil {
return err
}
for _, c := range q.Collections {
v, ok := q.Embeddings[c.EmbedModel]
if !ok {
continue
}
if len(v) != c.Dims {
return fmt.Errorf("the query has %d dimensions; collection %s takes %d", len(v), c.Name, c.Dims)
}
// The literal collection and cast are what let the planner use
// that collection's partial index.
cast := `::vector(` + strconv.Itoa(c.Dims) + `)`
rows, err := tx.QueryContext(ctx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id
WHERE k.org_id = ? AND k.collection = `+sqlLiteral(c.ID)+`
ORDER BY k.embedding`+cast+` <=> ?`+cast+` LIMIT ?`,
q.OrgID, vectorText(v), q.K)
if err != nil {
return err
}
got, err := scan(rows)
if err != nil {
return err
}
for i, h := range got {
add(h, i+1)
}
}
if tsq := orQuery(q.Text); tsq != "" {
rows, err := tx.QueryContext(ctx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id, to_tsquery('simple', ?) q
WHERE k.org_id = ? AND k.collection IN (`+strings.Join(quoted, ", ")+`) AND k.tsv @@ q
ORDER BY ts_rank(k.tsv, q) DESC LIMIT ?`,
tsq, q.OrgID, q.K)
if err != nil {
return err
}
got, err := scan(rows)
if err != nil {
return err
}
for i, h := range got {
add(h, i+1)
}
}
return nil
})
if err != nil {
return nil, err
}
out := make([]KBHit, 0, len(hits))
for _, h := range hits {
out = append(out, *h)
}
sort.Slice(out, func(i, j int) bool {
if out[i].Score != out[j].Score {
return out[i].Score > out[j].Score
}
if out[i].DocID != out[j].DocID {
return out[i].DocID < out[j].DocID
}
return out[i].Ord < out[j].Ord
})
if len(out) > q.K {
out = out[:q.K]
}
return out, nil
}
// orQuery turns a question into a tsquery matching any of its words. All of
// them (plainto_tsquery) would match almost nothing a person types as a
// question; the rank then prefers chunks with more of them.
func orQuery(text string) string {
seen := map[string]bool{}
var words []string
// Not lowercased here: to_tsquery does that with the database's own
// rules, the same ones to_tsvector used on the chunks — Go's lower case
// of a Turkish İ is not the database's.
for _, w := range strings.FieldsFunc(text, func(r rune) bool {
return !unicode.IsLetter(r) && !unicode.IsDigit(r) && !unicode.IsMark(r)
}) {
if len([]rune(w)) < 2 || seen[w] {
continue
}
seen[w] = true
words = append(words, w)
if len(words) == 32 {
break
}
}
return strings.Join(words, " | ")
}
// kbOwnedBy counts the collections a scope owns, for refusing to delete it
// from under them. Zero on a store without knowledge bases.
func (s *DB) kbOwnedBy(ctx context.Context, c conn, scope Scope, id string) (int, error) {
if s.dialect != PostgresDialect {
return 0, nil
}
var n int
q := `SELECT COUNT(*) FROM kb_collections WHERE owner_scope = ? AND owner_id = ?`
args := []any{string(scope), id}
if scope == ScopeOrg {
q, args = `SELECT COUNT(*) FROM kb_collections WHERE org_id = ?`, []any{id}
}
err := c.QueryRowContext(ctx, q, args...).Scan(&n)
return n, err
}
// kbForgetScope removes what a deleted scope had: its grants, and for a
// person the collections they owned.
func (s *DB) kbForgetScope(ctx context.Context, c conn, scope Scope, id string) error {
if s.dialect != PostgresDialect {
return nil
}
if _, err := c.ExecContext(ctx, `DELETE FROM kb_grants WHERE scope = ? AND scope_id = ?`, string(scope), id); err != nil {
return err
}
if scope != ScopeUser {
return nil
}
owned, err := queryAll(ctx, c, `SELECT id FROM kb_collections WHERE owner_scope = 'user' AND owner_id = ?`,
[]any{id}, func(r *sql.Rows) (string, error) {
var v string
return v, r.Scan(&v)
})
if err != nil {
return err
}
for _, cid := range owned {
if _, err := c.ExecContext(ctx, `DELETE FROM kb_collections WHERE id = ?`, cid); err != nil {
return err
}
if kbIDPattern.MatchString(cid) {
if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(cid)); err != nil {
return err
}
}
}
return nil
}
// kbDropPartition drops a deleted organization's chunk partition, empty by
// then: an organization with collections is not deleted.
func (s *DB) kbDropPartition(ctx context.Context, c conn, scope Scope, id string) error {
if s.dialect != PostgresDialect || scope != ScopeOrg || s.kbErr != nil {
return nil
}
_, err := c.ExecContext(ctx, `DROP TABLE IF EXISTS `+kbPartition(id))
return err
}
+362
View File
@@ -0,0 +1,362 @@
package store
import (
"context"
"errors"
"math"
"strings"
"testing"
"time"
)
func TestKnowledgeIsOffOnSQLite(t *testing.T) {
s := openDB(t)
tree(t, s)
err := s.KnowledgeAvailable()
if !errors.Is(err, ErrKnowledgeOff) || !strings.Contains(err.Error(), "SQLite") {
t.Fatalf("SQLite store says %v", err)
}
if _, err := s.CreateCollection(context.Background(), KBCollection{
OwnerScope: ScopeProject, OwnerID: "proj1", Name: "x", EmbedModel: "m", Dims: 3,
}); !errors.Is(err, ErrKnowledgeOff) {
t.Errorf("creating a collection on SQLite gave %v", err)
}
// Deleting from the tree still works without the tables.
if err := s.DeleteProject(context.Background(), "proj1"); err != nil {
t.Errorf("deleting a project on SQLite: %v", err)
}
}
// kbTwoOrgs is org1 (team1/proj1, team2/proj2) and org2 (team3/proj3).
func kbTwoOrgs(t *testing.T, s *DB) {
t.Helper()
ctx := context.Background()
tree(t, s)
for _, err := range []error{
s.PutTeam(ctx, Team{ID: "team2", OrgID: "org1", Name: "Finance"}),
s.PutProject(ctx, Project{ID: "proj2", TeamID: "team2", Name: "Payroll"}),
s.PutOrg(ctx, Org{ID: "org2", Name: "Other"}),
s.PutTeam(ctx, Team{ID: "team3", OrgID: "org2", Name: "Research"}),
s.PutProject(ctx, Project{ID: "proj3", TeamID: "team3", Name: "Chat"}),
} {
if err != nil {
t.Fatal(err)
}
}
}
// unit is a 3-dimensional unit vector pointing mostly along axis i.
func unit(i int) []float32 {
v := []float32{0.1, 0.1, 0.1}
v[i] = 1
n := float32(math.Sqrt(float64(v[0]*v[0] + v[1]*v[1] + v[2]*v[2])))
for j := range v {
v[j] /= n
}
return v
}
func mustCollection(t *testing.T, s *DB, scope Scope, owner, org, name string) KBCollection {
t.Helper()
c, err := s.CreateCollection(context.Background(), KBCollection{
OwnerScope: scope, OwnerID: owner, OrgID: org, Name: name, EmbedModel: "emb", Dims: 3,
})
if err != nil {
t.Fatalf("creating %s: %v", name, err)
}
return c
}
func mustDoc(t *testing.T, s *DB, c KBCollection, name string, bodies ...string) {
t.Helper()
var chunks []KBChunk
for i, b := range bodies {
chunks = append(chunks, KBChunk{Body: b, Embedding: unit(i % 3)})
}
if _, _, err := s.PutDocument(context.Background(), c, KBDocument{Name: name, Digest: name + "-1", Bytes: 10}, chunks); err != nil {
t.Fatalf("adding %s: %v", name, err)
}
}
func openKB(t *testing.T) *DB {
t.Helper()
s := openPG(t)
if err := s.KnowledgeAvailable(); err != nil {
t.Skipf("the test database has no pgvector: %v", err)
}
return s
}
func names(cs []KBCollection) string {
var out []string
for _, c := range cs {
out = append(out, c.Name)
}
return strings.Join(out, ",")
}
func TestPostgresCollectionsReachOnlyTheirChain(t *testing.T) {
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
mustCollection(t, s, ScopeOrg, "org1", "", "handbook")
mustCollection(t, s, ScopeTeam, "team1", "", "research")
pay := mustCollection(t, s, ScopeProject, "proj2", "", "payroll")
mustCollection(t, s, ScopeProject, "proj3", "", "other-org")
read := func(e Effective) string {
t.Helper()
got, err := s.CollectionsFor(ctx, e, false)
if err != nil {
t.Fatal(err)
}
return names(got)
}
inProj1 := Effective{KeyID: "k1", ProjectID: "proj1", TeamID: "team1", OrgID: "org1"}
if got := read(inProj1); got != "handbook,research" {
t.Errorf("proj1 reads %q", got)
}
if got := read(Effective{KeyID: "k3", ProjectID: "proj3", TeamID: "team3", OrgID: "org2"}); got != "other-org" {
t.Errorf("org2 reads %q", got)
}
// A personal key with no project reads nothing, whatever its person
// belongs to.
if got := read(Effective{KeyID: "kp", UserID: "u1"}); got != "" {
t.Errorf("a caller outside any organization reads %q", got)
}
// A grant to a key adds that collection for that key alone.
_, k := mustKey(t, s, "k1", "proj1")
if err := s.Grant(ctx, KBGrant{Scope: ScopeKey, ScopeID: k.ID, Collection: pay.ID, Perm: KBRead}); err != nil {
t.Fatal(err)
}
if got := read(inProj1); got != "handbook,payroll,research" {
t.Errorf("proj1 with a key grant reads %q", got)
}
if got := read(Effective{KeyID: "k-other", ProjectID: "proj1", TeamID: "team1", OrgID: "org1"}); got != "handbook,research" {
t.Errorf("another key of proj1 reads %q", got)
}
// Read is not write.
w, err := s.CollectionsFor(ctx, inProj1, true)
if err != nil {
t.Fatal(err)
}
if strings.Contains(names(w), "payroll") {
t.Errorf("a read grant gave write: %s", names(w))
}
}
func TestPostgresGrantsNeverCrossOrganizations(t *testing.T) {
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
for _, g := range []KBGrant{
{Scope: ScopeOrg, ScopeID: "org2"},
{Scope: ScopeTeam, ScopeID: "team3"},
{Scope: ScopeProject, ScopeID: "proj3"},
} {
g.Collection, g.Perm = c.ID, KBRead
if err := s.Grant(ctx, g); err == nil || !strings.Contains(err.Error(), "another organization") {
t.Errorf("granting to %s %s gave %v", g.Scope, g.ScopeID, err)
}
}
_, k3 := mustKey(t, s, "k3", "proj3")
if err := s.Grant(ctx, KBGrant{Scope: ScopeKey, ScopeID: k3.ID, Collection: c.ID, Perm: KBRead}); err == nil {
t.Error("a key of another organization was granted a collection")
}
// A person who belongs to nothing in the organization cannot own or be
// granted one there.
if err := s.CreateUser(ctx, User{ID: "u1", Email: "u1@example.com", Name: "u", PasswordHash: "x",
State: UserActive, CreatedAt: time.Now()}); err != nil {
t.Fatal(err)
}
if err := s.Grant(ctx, KBGrant{Scope: ScopeUser, ScopeID: "u1", Collection: c.ID, Perm: KBRead}); err == nil {
t.Error("a person with no membership in the organization was granted a collection")
}
if err := s.AddMembership(ctx, "u1", ScopeProject, "proj2", "member", "t", time.Now()); err != nil {
t.Fatal(err)
}
if err := s.Grant(ctx, KBGrant{Scope: ScopeUser, ScopeID: "u1", Collection: c.ID, Perm: KBRead}); err != nil {
t.Errorf("a member of a project in the organization: %v", err)
}
}
func TestPostgresSearchReturnsOnlyAllowedChunks(t *testing.T) {
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
mine := mustCollection(t, s, ScopeProject, "proj1", "", "mine")
secret := mustCollection(t, s, ScopeProject, "proj2", "", "secret")
foreign := mustCollection(t, s, ScopeProject, "proj3", "", "foreign")
mustDoc(t, s, mine, "a.md", "vacation policy is twenty days", "expenses go through finance")
mustDoc(t, s, secret, "salaries.md", "vacation salaries are confidential")
mustDoc(t, s, foreign, "theirs.md", "vacation at the other company")
hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{mine},
Embeddings: map[string][]float32{"emb": unit(0)}, Text: "vacation policy", K: 5})
if err != nil {
t.Fatal(err)
}
if len(hits) == 0 || hits[0].Body != "vacation policy is twenty days" || hits[0].Document != "a.md" {
t.Fatalf("best hit %+v", hits)
}
for _, h := range hits {
if h.Collection != mine.ID {
t.Errorf("a search of %s returned a chunk of %s: %q", mine.Name, h.Collection, h.Body)
}
}
// A collection of another organization is refused, not searched.
if _, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{mine, foreign},
Embeddings: map[string][]float32{"emb": unit(0)}, Text: "vacation", K: 5}); err == nil {
t.Error("a search across organizations was run")
}
}
func TestPostgresRowSecurityHoldsWithoutTheFilter(t *testing.T) {
// The second line: a query that forgets its WHERE gets nothing outside
// the scope its transaction set, and nothing at all without one.
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
mine := mustCollection(t, s, ScopeProject, "proj1", "", "mine")
other := mustCollection(t, s, ScopeProject, "proj2", "", "other")
mustDoc(t, s, mine, "a.md", "one", "two")
mustDoc(t, s, other, "b.md", "three")
count := func(org string, colls ...string) int {
t.Helper()
var n int
err := s.inTx(ctx, func(c conn) error {
if org != "" {
if err := kbScope(ctx, c, org, colls); err != nil {
return err
}
}
return c.QueryRowContext(ctx, `SELECT COUNT(*) FROM kb_chunks`).Scan(&n)
})
if err != nil {
t.Fatal(err)
}
return n
}
if n := count(""); n != 0 {
t.Errorf("with no scope set, %d chunks are visible", n)
}
if n := count("org1", mine.ID); n != 2 {
t.Errorf("scoped to one collection, %d chunks are visible", n)
}
if n := count("org2", mine.ID); n != 0 {
t.Errorf("a collection named under the wrong organization shows %d chunks", n)
}
// Nor can a write put a chunk outside its scope.
err := s.inTx(ctx, func(c conn) error {
if err := kbScope(ctx, c, "org1", []string{mine.ID}); err != nil {
return err
}
docs, _ := s.Documents(ctx, other.ID)
_, err := c.ExecContext(ctx, `INSERT INTO kb_chunks (org_id, collection, doc_id, ord, body, embedding)
VALUES ('org1', ?, ?, 9, 'x', '[1,0,0]')`, other.ID, docs[0].ID)
return err
})
if err == nil || !strings.Contains(err.Error(), "row-level security") {
t.Errorf("writing into another collection gave %v", err)
}
}
func TestPostgresDocumentsReplaceWholeAndSkipTheSame(t *testing.T) {
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
put := func(digest string, bodies ...string) bool {
t.Helper()
var chunks []KBChunk
for _, b := range bodies {
chunks = append(chunks, KBChunk{Body: b, Embedding: unit(0)})
}
_, changed, err := s.PutDocument(ctx, c, KBDocument{Name: "a.md", Digest: digest}, chunks)
if err != nil {
t.Fatal(err)
}
return changed
}
if !put("d1", "old one", "old two") {
t.Error("a new document was not a change")
}
if put("d1", "ignored") {
t.Error("the same digest again was a change")
}
if !put("d2", "new") {
t.Error("a new digest was not a change")
}
hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{c},
Embeddings: map[string][]float32{"emb": unit(0)}, K: 10})
if err != nil {
t.Fatal(err)
}
if len(hits) != 1 || hits[0].Body != "new" {
t.Errorf("after replacing, the collection holds %+v", hits)
}
if _, _, err := s.PutDocument(ctx, c, KBDocument{Name: "b.md", Digest: "x"},
[]KBChunk{{Body: "x", Embedding: []float32{1, 0}}}); err == nil {
t.Error("a chunk of the wrong dimension was accepted")
}
}
func TestPostgresScopesWithCollectionsAreNotDeleted(t *testing.T) {
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
c := mustCollection(t, s, ScopeProject, "proj2", "", "docs")
mustDoc(t, s, c, "a.md", "x")
if err := s.DeleteProject(ctx, "proj2"); !errors.Is(err, ErrNotEmpty) || !strings.Contains(err.Error(), "collection") {
t.Errorf("deleting a project with a collection gave %v", err)
}
if err := s.DeleteCollection(ctx, c.ID); err != nil {
t.Fatal(err)
}
if err := s.DeleteProject(ctx, "proj2"); err != nil {
t.Errorf("deleting the project once its collection is gone: %v", err)
}
var n int
if err := s.raw.QueryRowContext(ctx, `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
kbVectorIndex(c.ID)).Scan(&n); err != nil || n != 0 {
t.Errorf("the collection's index is left behind (%d, %v)", n, err)
}
}
func TestPostgresAPersonsCollectionsGoWithTheAccount(t *testing.T) {
ctx := context.Background()
s := openKB(t)
kbTwoOrgs(t, s)
if err := s.CreateUser(ctx, User{ID: "u1", Email: "u1@example.com", Name: "u", PasswordHash: "x",
State: UserActive, CreatedAt: time.Now()}); err != nil {
t.Fatal(err)
}
if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeUser, OwnerID: "u1", OrgID: "org1",
Name: "notes", EmbedModel: "emb", Dims: 3}); err == nil {
t.Error("a person outside the organization made a collection in it")
}
if err := s.AddMembership(ctx, "u1", ScopeTeam, "team1", "member", "t", time.Now()); err != nil {
t.Fatal(err)
}
c := mustCollection(t, s, ScopeUser, "u1", "org1", "notes")
mustDoc(t, s, c, "n.md", "x")
if err := s.DeleteUser(ctx, "u1", time.Now()); err != nil {
t.Fatal(err)
}
if _, err := s.Collection(ctx, c.ID); !errors.Is(err, ErrNotFound) {
t.Errorf("the person's collection outlived the account: %v", err)
}
}
func TestOrQueryMatchesAnyWord(t *testing.T) {
if got := orQuery("Yıllık izin kaç gün? İzin & 'x' -- a"); got != "Yıllık | izin | kaç | gün | İzin" {
t.Errorf("orQuery gave %q", got)
}
if got := orQuery("?!"); got != "" {
t.Errorf("no words gave %q", got)
}
}
@@ -0,0 +1,55 @@
-- Knowledge bases: collections of a company's documents that requests are
-- answered from (BACKLOG F). PostgreSQL only: there is no SQLite file of this
-- number, and on SQLite the feature is off.
--
-- The chunks themselves are not here. They need the pgvector extension, which
-- only a superuser can create and which a cluster may not have installed; a
-- migration that failed for want of it would stop the control plane from
-- starting at all. The store creates kb_chunks when it opens, if it can
-- (knowledge.go), and says why not when it cannot.
-- A collection has one owner in the tenancy tree and lives inside one
-- organization, which no grant can cross.
CREATE TABLE kb_collections (
id TEXT PRIMARY KEY,
org_id TEXT NOT NULL REFERENCES organizations(id),
owner_scope TEXT NOT NULL,
owner_id TEXT NOT NULL,
name TEXT NOT NULL,
-- Vectors from two models are not comparable, so a collection is embedded
-- by one model and its dimension is fixed with it.
embed_model TEXT NOT NULL,
dims INTEGER NOT NULL,
created_at BIGINT NOT NULL,
created_by TEXT NOT NULL DEFAULT '',
UNIQUE (org_id, name)
);
CREATE INDEX kb_collections_by_owner ON kb_collections(owner_scope, owner_id);
-- Who else may read or write a collection. Like guard_attachments, keyed by
-- scope with everything below the scope included; unlike guard policies,
-- grants add up rather than narrow.
CREATE TABLE kb_grants (
scope TEXT NOT NULL,
scope_id TEXT NOT NULL,
collection TEXT NOT NULL REFERENCES kb_collections(id) ON DELETE CASCADE,
perm TEXT NOT NULL,
added_at BIGINT NOT NULL,
added_by TEXT NOT NULL DEFAULT '',
PRIMARY KEY (scope, scope_id, collection)
);
CREATE INDEX kb_grants_by_collection ON kb_grants(collection);
-- A document as added, by name within its collection. The digest is what
-- decides whether adding it again changes anything.
CREATE TABLE kb_documents (
id TEXT PRIMARY KEY,
collection TEXT NOT NULL REFERENCES kb_collections(id) ON DELETE CASCADE,
name TEXT NOT NULL,
digest TEXT NOT NULL,
bytes BIGINT NOT NULL,
chunks INTEGER NOT NULL,
added_at BIGINT NOT NULL,
added_by TEXT NOT NULL DEFAULT '',
UNIQUE (collection, name)
);
+6 -2
View File
@@ -55,9 +55,13 @@ func OpenPostgres(ctx context.Context, dsn string) (*DB, error) {
_ = db.Close()
return nil, err
}
return &DB{
s := &DB{
raw: db,
dialect: PostgresDialect,
db: bound{c: PostgresDialect, to: db},
}, nil
}
// Not fatal: a store without pgvector serves everything but knowledge
// bases, and says why when somebody asks for one.
s.kbErr = s.ensureKnowledge(ctx)
return s, nil
}
+4
View File
@@ -48,6 +48,10 @@ type DB struct {
// personal is what a person's own use is held to when nobody wrote a
// limit for them; see SetPersonalDefaults.
personal *Limits
// kbErr is why this store cannot hold knowledge bases, nil when it can;
// set once at open (knowledge.go).
kbErr error
}
// OpenSQLite opens (creating if needed) a database and brings it up to date.