Knowledge bases in the store: collections, grants and chunks kept apart per tenant
Collections owned by an organization, team, project or person; grants that add up down the tree and never cross an organization; chunks in PostgreSQL with pgvector, partitioned by organization, an HNSW index per collection, and row-level security as a second line behind each query's filter. Search fuses nearest-neighbour and full-text ranks. Off on SQLite and without pgvector, saying why. server init and the HA database setup install and enable it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
0a7fb821d8
commit
d35630defe
11 files changed
+1408
-25
No files matched your search
+35
-19
@@ -829,25 +829,41 @@ ACL yet); a personal key or session names a project or team and gets that
|
||||
chain only, never the union of its person's memberships; PostgreSQL store
|
||||
only, off on SQLite with a sentence saying why.
|
||||
|
||||
### F1. Store: collections, grants, chunks (M)
|
||||
- `pgvector` ≥ 0.8 installed on every Patroni member by `server init`
|
||||
(`internal/cli/serverinit_postgres.go`), `CREATE EXTENSION` in the
|
||||
migration; the SQLite migration has no tables, and the store reports the
|
||||
feature unavailable.
|
||||
- Tables: `kb_collections` (id, org_id, owner `team:…`/`project:…`/`user:…`,
|
||||
name, embed_model, dims, created_at), `kb_grants` (scope, scope_id,
|
||||
collection, perm read|write — like `guard_attachments`), `kb_documents`
|
||||
(name, digest, bytes, state, chunk count), `kb_chunks` (org_id,
|
||||
collection, doc_id, ord, body, tsv, embedding) `PARTITION BY LIST
|
||||
(org_id)`, a partition per organization, HNSW and GIN indexes per
|
||||
partition.
|
||||
- Row-level security on `kb_chunks` against `current_setting('llmgw.org')`
|
||||
and `llmgw.collections`, retrieval through its own role.
|
||||
- Rules, refused before anything is written: a grant never crosses
|
||||
organizations; owner and grants resolve to existing scopes; deleting a
|
||||
project/team/org and `users forget` delete their collections (`forget.go`).
|
||||
- Done: store tests on PostgreSQL, including an RLS test that a query
|
||||
without the settings returns no rows.
|
||||
### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
|
||||
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
|
||||
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
|
||||
(read|write, any scope, never across organizations; a key needs a
|
||||
project; a person needs a membership in the organization),
|
||||
`kb_documents` (digest: the same again is no change, a new one replaces
|
||||
in one transaction).
|
||||
- `kb_chunks` is created when the store opens, not in a migration (a
|
||||
migration needing an extension would stop the control plane; one that
|
||||
skipped it would never run again): partitioned by organization (hashed
|
||||
partition name), `tsv` generated (`simple`), GIN on the parent, one
|
||||
partial HNSW index per collection on its organization's partition
|
||||
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
|
||||
20k rows: partition pruned, that index used, policy applied.
|
||||
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
|
||||
(set with set_config per transaction), one role — the table owner — not
|
||||
a separate one. Chunks are deleted by the foreign-key cascade from their
|
||||
document, which RLS does not block.
|
||||
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
|
||||
no organization (personal key without a project) reads nothing.
|
||||
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
|
||||
the partial index is used) plus a full-text arm with any word
|
||||
(`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
|
||||
- Deleting a project/team/org that owns collections is refused like one
|
||||
with members (changed from "deleted with them": every other delete in the
|
||||
tree refuses); DeleteUser/ForgetUser delete the person's collections;
|
||||
deleting an organization drops its partition; grants to a deleted scope go.
|
||||
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
|
||||
it; pgha installs it on every member at setup and enables it on the
|
||||
leader on every run. Neither stops on failure.
|
||||
Left: a cluster set up before (the PoC) gets no package from a rerun of
|
||||
pgha on a running cluster — install by hand or add an upgrade step;
|
||||
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
|
||||
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
|
||||
pgvector database as a non-superuser (a superuser bypasses RLS).
|
||||
|
||||
### F2. Ingest (M)
|
||||
- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add
|
||||
|
||||
@@ -23,6 +23,23 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
- **Knowledge bases, the store's part** (BACKLOG F1, branch `rag`):
|
||||
collections of documents owned by an organization, team, project or
|
||||
person, grants that add up down the tenancy tree and never cross an
|
||||
organization, chunks in PostgreSQL with pgvector — partitioned by
|
||||
organization, an HNSW index per collection, row-level security on the
|
||||
table as a second line behind every query's own filter — and a search
|
||||
that fuses nearest-neighbour and full-text ranks. Off on SQLite, and on
|
||||
PostgreSQL without pgvector, saying why. `server init` and the
|
||||
high-availability database setup install pgvector and enable it; a
|
||||
cluster set up before needs `postgresql-<version>-pgvector` on every
|
||||
database machine and `CREATE EXTENSION vector` in llmgw's database. A
|
||||
project, team or organization that owns collections is not deleted; a
|
||||
person's collections go with their account. Nothing reads or writes them
|
||||
yet from a command or a request.
|
||||
|
||||
## v0.13.0 — 2026-10-08
|
||||
|
||||
The whole system stopped, started and restarted from one command, a
|
||||
|
||||
@@ -370,6 +370,8 @@ func (in *serverInit) setupPostgres(ctx context.Context) (string, error) {
|
||||
in.done("PostgreSQL sized for a machine it shares (shared_buffers 96MB, 40 connections)")
|
||||
}
|
||||
|
||||
in.setupPgvector(ctx, discovery.DiscoverHost())
|
||||
|
||||
// Proof the address works, before anything is moved into it.
|
||||
address := "@" + in.dsnPath()
|
||||
s, err := store.Open(ctx, address)
|
||||
@@ -482,8 +484,17 @@ func lastLines(b []byte, n int) string {
|
||||
|
||||
// psql runs one statement as the postgres superuser, over the local socket.
|
||||
func (in *serverInit) psql(ctx context.Context, query string) (string, error) {
|
||||
cmd := exec.CommandContext(ctx, "runuser", "-u", "postgres", "--",
|
||||
in.pg.tool("psql"), "-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query)
|
||||
return in.psqlIn(ctx, "", query)
|
||||
}
|
||||
|
||||
// psqlIn runs a statement as postgres in one database, or in postgres's
|
||||
// default one when db is "".
|
||||
func (in *serverInit) psqlIn(ctx context.Context, db, query string) (string, error) {
|
||||
args := []string{"-v", "ON_ERROR_STOP=1", "-qAtX", "-c", query}
|
||||
if db != "" {
|
||||
args = append(args, "-d", db)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, "runuser", append([]string{"-u", "postgres", "--", in.pg.tool("psql")}, args...)...)
|
||||
// postgres cannot enter root's home, and psql says so on every call.
|
||||
cmd.Dir = "/"
|
||||
out, err := cmd.CombinedOutput()
|
||||
@@ -538,3 +549,50 @@ func (in *serverInit) stopService(ctx context.Context) (bool, error) {
|
||||
in.done("llmgw-server stopped while its store moves")
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// pgvectorPackages are the package names of pgvector for a PostgreSQL major
|
||||
// version: Debian's and Ubuntu's (and PGDG's apt repository) name it after the
|
||||
// server, PGDG's rpm repository after the version alone.
|
||||
func pgvectorPackages(major int) map[string][]string {
|
||||
return map[string][]string{
|
||||
"deb": {fmt.Sprintf("postgresql-%d-pgvector", major)},
|
||||
"rpm": {fmt.Sprintf("pgvector_%d", major)},
|
||||
}
|
||||
}
|
||||
|
||||
// setupPgvector installs pgvector and enables it in llmgw's database, for
|
||||
// knowledge bases. Not a reason to stop: without it everything else works,
|
||||
// and the store says why knowledge bases are off when somebody asks for one.
|
||||
func (in *serverInit) setupPgvector(ctx context.Context, h discovery.Host) {
|
||||
warn := func(format string, a ...any) {
|
||||
fmt.Fprintf(in.out, " !! pgvector: "+format+"; knowledge bases stay off\n", a...)
|
||||
}
|
||||
if on, err := in.psqlIn(ctx, pgDatabase, "SELECT 1 FROM pg_extension WHERE extname = 'vector'"); err == nil && on == "1" {
|
||||
in.kept("pgvector in database %s", pgDatabase)
|
||||
return
|
||||
}
|
||||
have, err := in.psqlBool(ctx, "SELECT 1 FROM pg_available_extensions WHERE name = 'vector'")
|
||||
if err != nil {
|
||||
warn("%v", err)
|
||||
return
|
||||
}
|
||||
if !have {
|
||||
num, err := in.psql(ctx, "SHOW server_version_num")
|
||||
n, convErr := strconv.Atoi(num)
|
||||
if err != nil || convErr != nil || h.Family() == "" {
|
||||
warn("cannot tell which package to install (version %q, %s)", num, h.OSID)
|
||||
return
|
||||
}
|
||||
pkgs := pgvectorPackages(n / 10000)
|
||||
pi := &provision.Installer{Host: h, Log: slog.Default()}
|
||||
if err := pi.InstallPackages(ctx, pkgs); err != nil {
|
||||
warn("installing %s: %v", strings.Join(pkgs[h.Family()], " "), err)
|
||||
return
|
||||
}
|
||||
}
|
||||
if _, err := in.psqlIn(ctx, pgDatabase, "CREATE EXTENSION IF NOT EXISTS vector"); err != nil {
|
||||
warn("%v", err)
|
||||
return
|
||||
}
|
||||
in.done("pgvector enabled in database %s", pgDatabase)
|
||||
}
|
||||
@@ -159,6 +159,11 @@ func TestARunningClusterIsLeftAsItIs(t *testing.T) {
|
||||
if !a.ranAny("CREATE ROLE") {
|
||||
t.Error("the role was not made sure of on the leader")
|
||||
}
|
||||
// Enabled where the package is there, which on a running cluster is
|
||||
// all a rerun does about it: it installs nothing.
|
||||
if !a.ranAny("CREATE EXTENSION IF NOT EXISTS vector") {
|
||||
t.Error("pgvector was not enabled in llmgw's database on the leader")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTwoDatabaseMachinesWithoutAThirdVoteAreRefused(t *testing.T) {
|
||||
|
||||
+17
-1
@@ -502,6 +502,14 @@ if [ -n "$need" ]; then
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y $need >/dev/null
|
||||
fi
|
||||
v=$(ls /usr/lib/postgresql | sort -n | tail -1)
|
||||
# pgvector, for knowledge bases: on every member, since any of them may lead.
|
||||
# Not a reason to stop; the store says knowledge bases are off without it.
|
||||
if [ ! -f /usr/share/postgresql/$v/extension/vector.control ]; then
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y postgresql-$v-pgvector >/dev/null 2>&1 ||
|
||||
{ sudo DEBIAN_FRONTEND=noninteractive apt-get -qq update >/dev/null 2>&1 &&
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get -qq install -y postgresql-$v-pgvector >/dev/null 2>&1; } ||
|
||||
echo "no-pgvector"
|
||||
fi
|
||||
if [ -d /var/lib/postgresql/$v/main ]; then
|
||||
if sudo systemctl is-active --quiet postgresql@$v-main; then
|
||||
n=$(sudo -u postgres psql -Atc "select count(*) from pg_database where datname not in ('postgres','template0','template1')")
|
||||
@@ -553,13 +561,21 @@ sudo -u postgres psql -q -v ON_ERROR_STOP=1 -v pw="$pw" <<'SQL'
|
||||
SELECT format('CREATE ROLE %I LOGIN', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'llmgw') \gexec
|
||||
ALTER ROLE llmgw PASSWORD :'pw';
|
||||
SELECT format('CREATE DATABASE %I OWNER %I', 'llmgw', 'llmgw') WHERE NOT EXISTS (SELECT 1 FROM pg_database WHERE datname = 'llmgw') \gexec
|
||||
SQL`, nil)
|
||||
SQL
|
||||
sudo -u postgres psql -q -d llmgw -v ON_ERROR_STOP=1 <<'SQL' || echo "pgvector-failed"
|
||||
SELECT 'CREATE EXTENSION IF NOT EXISTS vector' WHERE EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'vector') \gexec
|
||||
SQL
|
||||
sudo -u postgres psql -d llmgw -Atc "select 'vector=' || count(*) from pg_extension where extname = 'vector'"`, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("on %s: %v: %s", leader.Name, err, strings.TrimSpace(string(out)))
|
||||
}
|
||||
// Said as a change only when something was made: a rerun that finds
|
||||
// both is not a reason to restart the gateway.
|
||||
s.emit("database", !strings.Contains(string(out), "had=2"), "role and database %s on %s", Database, leader.Name)
|
||||
if !strings.Contains(string(out), "vector=1") {
|
||||
s.emit("pgvector", false, "pgvector is not enabled in %s, so knowledge bases stay off "+
|
||||
"(install postgresql-<version>-pgvector on every member and run setup again)", Database)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
@@ -214,6 +214,12 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check
|
||||
busy = append(busy, plural(n, ch.noun))
|
||||
}
|
||||
}
|
||||
// Not an emptyCheck: the table is there only on PostgreSQL.
|
||||
if n, err := s.kbOwnedBy(ctx, c, scope, id); err != nil {
|
||||
return err
|
||||
} else if n > 0 {
|
||||
busy = append(busy, plural(n, "collection"))
|
||||
}
|
||||
if len(busy) > 0 {
|
||||
return fmt.Errorf("%w: it still has %s", ErrNotEmpty, strings.Join(busy, " and "))
|
||||
}
|
||||
@@ -222,6 +228,9 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check
|
||||
string(scope), string(MemberScope(scope)), id); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.kbForgetScope(ctx, c, scope, id); err != nil {
|
||||
return err
|
||||
}
|
||||
res, err := c.ExecContext(ctx, del, id)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -229,7 +238,7 @@ func (s *DB) deleteEmpty(ctx context.Context, scope Scope, id, del string, check
|
||||
if n, _ := res.RowsAffected(); n == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
return s.kbDropPartition(ctx, c, scope, id)
|
||||
})
|
||||
}
|
||||
|
||||
@@ -248,6 +257,10 @@ func (s *DB) DeleteUser(ctx context.Context, userID string, at time.Time) error
|
||||
millis(at), userID); err != nil {
|
||||
return err
|
||||
}
|
||||
// A person's own collections go with the account.
|
||||
if err := s.kbForgetScope(ctx, c, ScopeUser, userID); err != nil {
|
||||
return err
|
||||
}
|
||||
res, err := c.ExecContext(ctx, `DELETE FROM users WHERE id = ?`, userID)
|
||||
if err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,833 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// Knowledge bases (BACKLOG F): collections of documents, split into chunks
|
||||
// and embedded, that a request is answered from.
|
||||
//
|
||||
// Which tenant may read which chunk is decided twice. The queries here filter
|
||||
// by organization and by the collections the caller may read, and the table
|
||||
// enforces the same with row-level security against two settings each
|
||||
// transaction makes — so a query that forgot its filter returns nothing
|
||||
// rather than another tenant's passages. The role llmgw connects as owns the
|
||||
// table, so the policy is FORCEd; a superuser would bypass it, which is one
|
||||
// more reason the store never connects as one.
|
||||
|
||||
// ErrKnowledgeOff is a store that cannot hold knowledge bases.
|
||||
var ErrKnowledgeOff = errors.New("knowledge bases are off")
|
||||
|
||||
// MaxEmbeddingDims is the most dimensions pgvector can index (HNSW on vector).
|
||||
const MaxEmbeddingDims = 2000
|
||||
|
||||
// KBPerm is what a grant allows.
|
||||
type KBPerm string
|
||||
|
||||
// The permissions. Write includes read.
|
||||
const (
|
||||
KBRead KBPerm = "read"
|
||||
KBWrite KBPerm = "write"
|
||||
)
|
||||
|
||||
// KBCollection is a set of documents with one owner and one embedding model.
|
||||
type KBCollection struct {
|
||||
ID string
|
||||
OrgID string
|
||||
OwnerScope Scope
|
||||
OwnerID string
|
||||
Name string
|
||||
EmbedModel string
|
||||
Dims int
|
||||
CreatedAt time.Time
|
||||
CreatedBy string
|
||||
}
|
||||
|
||||
// KBGrant lets a scope, and everything below it, read or write a collection.
|
||||
type KBGrant struct {
|
||||
Scope Scope
|
||||
ScopeID string
|
||||
Collection string
|
||||
Perm KBPerm
|
||||
AddedAt time.Time
|
||||
AddedBy string
|
||||
}
|
||||
|
||||
// KBDocument is a document as added to a collection.
|
||||
type KBDocument struct {
|
||||
ID string
|
||||
Collection string
|
||||
Name string
|
||||
Digest string
|
||||
Bytes int64
|
||||
Chunks int
|
||||
AddedAt time.Time
|
||||
AddedBy string
|
||||
}
|
||||
|
||||
// KBChunk is a piece of a document and its embedding.
|
||||
type KBChunk struct {
|
||||
Body string
|
||||
Embedding []float32
|
||||
}
|
||||
|
||||
// KBHit is a chunk a search returned, best first.
|
||||
type KBHit struct {
|
||||
Collection string
|
||||
DocID string
|
||||
Document string
|
||||
Ord int
|
||||
Body string
|
||||
// Score is the fused rank (reciprocal rank fusion over the vector and
|
||||
// text searches): only the order means anything.
|
||||
Score float64
|
||||
}
|
||||
|
||||
// KnowledgeAvailable says whether this store can hold knowledge bases, and
|
||||
// why not when it cannot.
|
||||
func (s *DB) KnowledgeAvailable() error {
|
||||
if s.dialect != PostgresDialect {
|
||||
return fmt.Errorf("%w: they need the PostgreSQL store (with pgvector), and this one is SQLite", ErrKnowledgeOff)
|
||||
}
|
||||
return s.kbErr
|
||||
}
|
||||
|
||||
// ensureKnowledge creates kb_chunks when the vector extension allows it.
|
||||
//
|
||||
// At open rather than in a migration: a migration that needs an extension the
|
||||
// database servers do not have would stop the control plane from starting,
|
||||
// and one that skipped it would be recorded as applied and never run again
|
||||
// once the extension arrived.
|
||||
func (s *DB) ensureKnowledge(ctx context.Context) error {
|
||||
var have bool
|
||||
if err := s.db.QueryRowContext(ctx,
|
||||
`SELECT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'vector')`).Scan(&have); err != nil {
|
||||
return fmt.Errorf("%w: reading the database's extensions: %v", ErrKnowledgeOff, err)
|
||||
}
|
||||
if !have {
|
||||
var available bool
|
||||
if err := s.db.QueryRowContext(ctx,
|
||||
`SELECT EXISTS (SELECT 1 FROM pg_available_extensions WHERE name = 'vector')`).Scan(&available); err != nil {
|
||||
return fmt.Errorf("%w: reading the database's extensions: %v", ErrKnowledgeOff, err)
|
||||
}
|
||||
if !available {
|
||||
return fmt.Errorf("%w: pgvector is not installed on the database servers "+
|
||||
"(package postgresql-<version>-pgvector on every one; llmgw's database setup installs it)", ErrKnowledgeOff)
|
||||
}
|
||||
// Works only where the extension is trusted or llmgw's role may
|
||||
// create it; elsewhere a superuser has to, once.
|
||||
if _, err := s.db.ExecContext(ctx, `CREATE EXTENSION IF NOT EXISTS vector`); err != nil {
|
||||
return fmt.Errorf("%w: pgvector is installed but not enabled in llmgw's database; "+
|
||||
"as postgres, run CREATE EXTENSION vector in it (%v)", ErrKnowledgeOff, err)
|
||||
}
|
||||
}
|
||||
var exists bool
|
||||
if err := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); err != nil {
|
||||
return fmt.Errorf("%w: %v", ErrKnowledgeOff, err)
|
||||
}
|
||||
if exists {
|
||||
return nil
|
||||
}
|
||||
err := s.inTx(ctx, func(c conn) error {
|
||||
for _, q := range []string{
|
||||
// Partitioned by organization: each one's indexes stay its own
|
||||
// size, and an organization's chunks go with one DROP.
|
||||
`CREATE TABLE kb_chunks (
|
||||
org_id TEXT NOT NULL,
|
||||
collection TEXT NOT NULL,
|
||||
doc_id TEXT NOT NULL REFERENCES kb_documents(id) ON DELETE CASCADE,
|
||||
ord INTEGER NOT NULL,
|
||||
body TEXT NOT NULL,
|
||||
tsv tsvector GENERATED ALWAYS AS (to_tsvector('simple', body)) STORED,
|
||||
-- No dimension here: collections differ. Each collection's
|
||||
-- index casts to its own (kbVectorIndex).
|
||||
embedding vector NOT NULL,
|
||||
PRIMARY KEY (org_id, doc_id, ord)
|
||||
) PARTITION BY LIST (org_id)`,
|
||||
`CREATE INDEX kb_chunks_tsv ON kb_chunks USING gin (tsv)`,
|
||||
`ALTER TABLE kb_chunks ENABLE ROW LEVEL SECURITY`,
|
||||
`ALTER TABLE kb_chunks FORCE ROW LEVEL SECURITY`,
|
||||
`CREATE POLICY kb_scope ON kb_chunks
|
||||
USING (org_id = current_setting('llmgw.kb_org', true)
|
||||
AND collection = ANY (string_to_array(current_setting('llmgw.kb_collections', true), ',')))
|
||||
WITH CHECK (org_id = current_setting('llmgw.kb_org', true)
|
||||
AND collection = ANY (string_to_array(current_setting('llmgw.kb_collections', true), ',')))`,
|
||||
} {
|
||||
if _, err := c.ExecContext(ctx, q); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
// Another control plane opening at the same moment made it first.
|
||||
if e2 := s.db.QueryRowContext(ctx, `SELECT to_regclass('kb_chunks') IS NOT NULL`).Scan(&exists); e2 == nil && exists {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("%w: creating kb_chunks: %v", ErrKnowledgeOff, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// kbIDPattern is what a collection id looks like (NewID("kb_")); ids are put
|
||||
// into DDL and partial-index predicates, so nothing else is let through.
|
||||
var kbIDPattern = regexp.MustCompile(`^kb_[a-z2-7]{16}$`)
|
||||
|
||||
// kbPartition is the chunk partition of an organization. Hashed, because old
|
||||
// organization ids were slugs of names and could hold anything.
|
||||
func kbPartition(orgID string) string {
|
||||
h := sha256.Sum256([]byte(orgID))
|
||||
return "kb_chunks_" + hex.EncodeToString(h[:12])
|
||||
}
|
||||
|
||||
// kbVectorIndex is the name of a collection's vector index.
|
||||
func kbVectorIndex(collection string) string {
|
||||
return "kb_vec_" + strings.TrimPrefix(collection, "kb_")
|
||||
}
|
||||
|
||||
func sqlLiteral(v string) string { return "'" + strings.ReplaceAll(v, "'", "''") + "'" }
|
||||
|
||||
// kbScope sets, for this transaction, which rows of kb_chunks it may touch.
|
||||
func kbScope(ctx context.Context, c conn, orgID string, collections []string) error {
|
||||
_, err := c.ExecContext(ctx, `SELECT set_config('llmgw.kb_org', ?, true), set_config('llmgw.kb_collections', ?, true)`,
|
||||
orgID, strings.Join(collections, ","))
|
||||
return err
|
||||
}
|
||||
|
||||
// orgOfScope is the organization a scope sits in. A person has none of their
|
||||
// own, so a person is checked against org instead: they must be a member of
|
||||
// it, or of something in it.
|
||||
func (s *DB) orgOfScope(ctx context.Context, c conn, scope Scope, id, org string) (string, error) {
|
||||
var q string
|
||||
switch scope {
|
||||
case ScopeOrg:
|
||||
q = `SELECT id FROM organizations WHERE id = ?`
|
||||
case ScopeTeam:
|
||||
q = `SELECT org_id FROM teams WHERE id = ?`
|
||||
case ScopeProject:
|
||||
q = `SELECT t.org_id FROM projects p JOIN teams t ON t.id = p.team_id WHERE p.id = ?`
|
||||
case ScopeKey:
|
||||
q = `SELECT t.org_id FROM api_keys k JOIN projects p ON p.id = k.project_id
|
||||
JOIN teams t ON t.id = p.team_id WHERE k.id = ?`
|
||||
case ScopeUser:
|
||||
var n int
|
||||
err := c.QueryRowContext(ctx, `SELECT COUNT(*) FROM memberships m WHERE m.user_id = ? AND (
|
||||
(m.scope_type = 'org' AND m.scope_id = ?)
|
||||
OR (m.scope_type = 'team' AND m.scope_id IN (SELECT id FROM teams WHERE org_id = ?))
|
||||
OR (m.scope_type = 'project' AND m.scope_id IN (
|
||||
SELECT p.id FROM projects p JOIN teams t ON t.id = p.team_id WHERE t.org_id = ?)))`,
|
||||
id, org, org, org).Scan(&n)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if n == 0 {
|
||||
return "", fmt.Errorf("%s is not a member of anything in that organization", id)
|
||||
}
|
||||
return org, nil
|
||||
default:
|
||||
return "", fmt.Errorf("a collection cannot be owned by or granted to a %s", scope)
|
||||
}
|
||||
var got string
|
||||
err := c.QueryRowContext(ctx, q, id).Scan(&got)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
if scope == ScopeKey {
|
||||
return "", fmt.Errorf("no key %s in a project (a personal key reads no collection)", id)
|
||||
}
|
||||
return "", fmt.Errorf("no %s %s: %w", scope, id, ErrNotFound)
|
||||
}
|
||||
return got, err
|
||||
}
|
||||
|
||||
// CreateCollection makes an empty collection, its organization's chunk
|
||||
// partition when this is the first, and its vector index. c.OrgID is needed
|
||||
// only when a person owns it; otherwise it comes from the owner.
|
||||
func (s *DB) CreateCollection(ctx context.Context, c KBCollection) (KBCollection, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return KBCollection{}, err
|
||||
}
|
||||
c.Name = strings.TrimSpace(c.Name)
|
||||
switch {
|
||||
case c.Name == "":
|
||||
return KBCollection{}, errors.New("a collection needs a name")
|
||||
case c.EmbedModel == "":
|
||||
return KBCollection{}, errors.New("a collection needs the embedding model its documents are embedded with")
|
||||
case c.Dims < 1 || c.Dims > MaxEmbeddingDims:
|
||||
return KBCollection{}, fmt.Errorf("%d dimensions: an indexed embedding has 1 to %d", c.Dims, MaxEmbeddingDims)
|
||||
case c.OwnerScope == ScopeUser && c.OrgID == "":
|
||||
return KBCollection{}, errors.New("a person's collection names the organization it belongs to")
|
||||
case c.OwnerScope == ScopeKey:
|
||||
return KBCollection{}, errors.New("a key cannot own a collection; its project or its person can")
|
||||
}
|
||||
c.ID = NewID("kb_")
|
||||
if c.CreatedAt.IsZero() {
|
||||
c.CreatedAt = time.Now()
|
||||
}
|
||||
err := s.inTx(ctx, func(tx conn) error {
|
||||
org, err := s.orgOfScope(ctx, tx, c.OwnerScope, c.OwnerID, c.OrgID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if c.OrgID != "" && c.OrgID != org {
|
||||
return fmt.Errorf("the owner is in organization %s, not %s", org, c.OrgID)
|
||||
}
|
||||
c.OrgID = org
|
||||
var n int
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM kb_collections WHERE org_id = ? AND name = ?`,
|
||||
c.OrgID, c.Name).Scan(&n); err != nil {
|
||||
return err
|
||||
}
|
||||
if n > 0 {
|
||||
return fmt.Errorf("%w: collection %q", ErrDuplicateName, c.Name)
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO kb_collections
|
||||
(id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
c.ID, c.OrgID, string(c.OwnerScope), c.OwnerID, c.Name, c.EmbedModel, c.Dims,
|
||||
millis(c.CreatedAt), c.CreatedBy); err != nil {
|
||||
return err
|
||||
}
|
||||
part := kbPartition(c.OrgID)
|
||||
if _, err := tx.ExecContext(ctx, `CREATE TABLE IF NOT EXISTS `+part+
|
||||
` PARTITION OF kb_chunks FOR VALUES IN (`+sqlLiteral(c.OrgID)+`)`); err != nil {
|
||||
return err
|
||||
}
|
||||
// A partial index per collection, on its organization's partition
|
||||
// only: the dimension is the collection's, and a search, which
|
||||
// names one collection at a time, uses exactly this one.
|
||||
_, err = tx.ExecContext(ctx, `CREATE INDEX `+kbVectorIndex(c.ID)+` ON `+part+
|
||||
` USING hnsw ((embedding::vector(`+strconv.Itoa(c.Dims)+`)) vector_cosine_ops)`+
|
||||
` WHERE collection = `+sqlLiteral(c.ID))
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return KBCollection{}, err
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
const kbCollectionCols = `id, org_id, owner_scope, owner_id, name, embed_model, dims, created_at, created_by`
|
||||
|
||||
func scanCollection(sc scanner) (KBCollection, error) {
|
||||
var c KBCollection
|
||||
var scope string
|
||||
var at int64
|
||||
err := sc.Scan(&c.ID, &c.OrgID, &scope, &c.OwnerID, &c.Name, &c.EmbedModel, &c.Dims, &at, &c.CreatedBy)
|
||||
c.OwnerScope, c.CreatedAt = Scope(scope), fromMillis(at)
|
||||
return c, err
|
||||
}
|
||||
|
||||
// Collections lists every collection, by organization and name.
|
||||
func (s *DB) Collections(ctx context.Context) ([]KBCollection, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return queryAll(ctx, s.db, `SELECT `+kbCollectionCols+` FROM kb_collections ORDER BY org_id, name`, nil,
|
||||
func(r *sql.Rows) (KBCollection, error) { return scanCollection(r) })
|
||||
}
|
||||
|
||||
// Collection reads one collection.
|
||||
func (s *DB) Collection(ctx context.Context, id string) (KBCollection, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return KBCollection{}, err
|
||||
}
|
||||
c, err := scanCollection(s.db.QueryRowContext(ctx, `SELECT `+kbCollectionCols+` FROM kb_collections WHERE id = ?`, id))
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return KBCollection{}, ErrNotFound
|
||||
}
|
||||
return c, err
|
||||
}
|
||||
|
||||
// DeleteCollection removes a collection, its documents, chunks, grants and
|
||||
// index.
|
||||
func (s *DB) DeleteCollection(ctx context.Context, id string) error {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return err
|
||||
}
|
||||
if !kbIDPattern.MatchString(id) {
|
||||
return ErrNotFound
|
||||
}
|
||||
return s.inTx(ctx, func(c conn) error {
|
||||
// Chunks go with their documents (a foreign-key cascade, which row
|
||||
// security does not stand in the way of), documents and grants with
|
||||
// the collection.
|
||||
res, err := c.ExecContext(ctx, `DELETE FROM kb_collections WHERE id = ?`, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n, _ := res.RowsAffected(); n == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
_, err = c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(id))
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
// Grant lets a scope read or write a collection; granting again replaces the
|
||||
// permission. The scope must be in the collection's organization.
|
||||
func (s *DB) Grant(ctx context.Context, g KBGrant) error {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return err
|
||||
}
|
||||
if g.Perm != KBRead && g.Perm != KBWrite {
|
||||
return fmt.Errorf("permission %q: read or write", g.Perm)
|
||||
}
|
||||
if g.AddedAt.IsZero() {
|
||||
g.AddedAt = time.Now()
|
||||
}
|
||||
return s.inTx(ctx, func(c conn) error {
|
||||
var org string
|
||||
err := c.QueryRowContext(ctx, `SELECT org_id FROM kb_collections WHERE id = ?`, g.Collection).Scan(&org)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return fmt.Errorf("collection %s: %w", g.Collection, ErrNotFound)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
got, err := s.orgOfScope(ctx, c, g.Scope, g.ScopeID, org)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if got != org {
|
||||
return fmt.Errorf("%s %s is in another organization; a collection is never granted outside its own", g.Scope, g.ScopeID)
|
||||
}
|
||||
_, err = c.ExecContext(ctx, `INSERT INTO kb_grants (scope, scope_id, collection, perm, added_at, added_by)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (scope, scope_id, collection) DO UPDATE SET perm = excluded.perm,
|
||||
added_at = excluded.added_at, added_by = excluded.added_by`,
|
||||
string(g.Scope), g.ScopeID, g.Collection, string(g.Perm), millis(g.AddedAt), g.AddedBy)
|
||||
return err
|
||||
})
|
||||
}
|
||||
|
||||
// Revoke removes a grant, reporting whether it was there.
|
||||
func (s *DB) Revoke(ctx context.Context, scope Scope, scopeID, collection string) (bool, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return false, err
|
||||
}
|
||||
res, err := s.db.ExecContext(ctx, `DELETE FROM kb_grants WHERE scope = ? AND scope_id = ? AND collection = ?`,
|
||||
string(scope), scopeID, collection)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// Grants lists a collection's grants, or every grant when collection is "".
|
||||
func (s *DB) Grants(ctx context.Context, collection string) ([]KBGrant, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
q, args := `SELECT scope, scope_id, collection, perm, added_at, added_by FROM kb_grants`, []any(nil)
|
||||
if collection != "" {
|
||||
q, args = q+` WHERE collection = ?`, []any{collection}
|
||||
}
|
||||
return queryAll(ctx, s.db, q+` ORDER BY collection, scope, scope_id`, args, func(r *sql.Rows) (KBGrant, error) {
|
||||
var g KBGrant
|
||||
var scope, perm string
|
||||
var at int64
|
||||
err := r.Scan(&scope, &g.ScopeID, &g.Collection, &perm, &at, &g.AddedBy)
|
||||
g.Scope, g.Perm, g.AddedAt = Scope(scope), KBPerm(perm), fromMillis(at)
|
||||
return g, err
|
||||
})
|
||||
}
|
||||
|
||||
// chainScopes is a caller's place in the tree, widest first.
|
||||
func chainScopes(e Effective) [][2]string {
|
||||
var out [][2]string
|
||||
add := func(s Scope, id string) {
|
||||
if id != "" {
|
||||
out = append(out, [2]string{string(s), id})
|
||||
}
|
||||
}
|
||||
add(ScopeOrg, e.OrgID)
|
||||
add(ScopeTeam, e.TeamID)
|
||||
add(ScopeProject, e.ProjectID)
|
||||
add(ScopeUser, e.UserID)
|
||||
// The chat page's caller is a session, not a key row; its id matches no
|
||||
// key grant, which is as it should be.
|
||||
add(ScopeKey, e.KeyID)
|
||||
return out
|
||||
}
|
||||
|
||||
// CollectionsFor lists the collections a caller may read, or with write
|
||||
// those it may change. A caller outside any organization — a personal key
|
||||
// that named no project — may read none: joining a second team must not
|
||||
// widen what every key of that person reads.
|
||||
func (s *DB) CollectionsFor(ctx context.Context, e Effective, write bool) ([]KBCollection, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if e.OrgID == "" {
|
||||
return nil, nil
|
||||
}
|
||||
chain := chainScopes(e)
|
||||
var match []string
|
||||
args := []any{e.OrgID}
|
||||
for _, sc := range chain {
|
||||
match = append(match, `(c.owner_scope = ? AND c.owner_id = ?)`)
|
||||
args = append(args, sc[0], sc[1])
|
||||
}
|
||||
var grant []string
|
||||
for _, sc := range chain {
|
||||
grant = append(grant, `(g.scope = ? AND g.scope_id = ?)`)
|
||||
args = append(args, sc[0], sc[1])
|
||||
}
|
||||
perm := ""
|
||||
if write {
|
||||
perm = ` AND g.perm = 'write'`
|
||||
}
|
||||
q := `SELECT ` + prefixCols("c.", kbCollectionCols) + ` FROM kb_collections c WHERE c.org_id = ? AND (` +
|
||||
strings.Join(match, " OR ") + ` OR EXISTS (SELECT 1 FROM kb_grants g WHERE g.collection = c.id` + perm +
|
||||
` AND (` + strings.Join(grant, " OR ") + `))) ORDER BY c.name`
|
||||
return queryAll(ctx, s.db, q, args, func(r *sql.Rows) (KBCollection, error) { return scanCollection(r) })
|
||||
}
|
||||
|
||||
func prefixCols(p, cols string) string {
|
||||
parts := strings.Split(cols, ", ")
|
||||
for i := range parts {
|
||||
parts[i] = p + parts[i]
|
||||
}
|
||||
return strings.Join(parts, ", ")
|
||||
}
|
||||
|
||||
// PutDocument adds a document to a collection, or replaces the one of the
|
||||
// same name, in one transaction: it is searchable whole or not at all. The
|
||||
// same digest again changes nothing, and reports so.
|
||||
func (s *DB) PutDocument(ctx context.Context, c KBCollection, d KBDocument, chunks []KBChunk) (KBDocument, bool, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return KBDocument{}, false, err
|
||||
}
|
||||
if !kbIDPattern.MatchString(c.ID) {
|
||||
return KBDocument{}, false, fmt.Errorf("collection %s: %w", c.ID, ErrNotFound)
|
||||
}
|
||||
for i, ch := range chunks {
|
||||
if len(ch.Embedding) != c.Dims {
|
||||
return KBDocument{}, false, fmt.Errorf("chunk %d has %d dimensions; collection %s takes %d",
|
||||
i, len(ch.Embedding), c.Name, c.Dims)
|
||||
}
|
||||
}
|
||||
d.Collection, d.Chunks = c.ID, len(chunks)
|
||||
if d.AddedAt.IsZero() {
|
||||
d.AddedAt = time.Now()
|
||||
}
|
||||
changed := true
|
||||
err := s.inTx(ctx, func(tx conn) error {
|
||||
var oldID, oldDigest string
|
||||
err := tx.QueryRowContext(ctx, `SELECT id, digest FROM kb_documents WHERE collection = ? AND name = ?`,
|
||||
c.ID, d.Name).Scan(&oldID, &oldDigest)
|
||||
switch {
|
||||
case err == nil && oldDigest == d.Digest:
|
||||
changed = false
|
||||
return nil
|
||||
case err == nil:
|
||||
if _, err := tx.ExecContext(ctx, `DELETE FROM kb_documents WHERE id = ?`, oldID); err != nil {
|
||||
return err
|
||||
}
|
||||
case !errors.Is(err, sql.ErrNoRows):
|
||||
return err
|
||||
}
|
||||
d.ID = NewID("doc_")
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO kb_documents
|
||||
(id, collection, name, digest, bytes, chunks, added_at, added_by) VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
|
||||
d.ID, c.ID, d.Name, d.Digest, d.Bytes, d.Chunks, millis(d.AddedAt), d.AddedBy); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := kbScope(ctx, tx, c.OrgID, []string{c.ID}); err != nil {
|
||||
return err
|
||||
}
|
||||
const batch = 100
|
||||
for start := 0; start < len(chunks); start += batch {
|
||||
end := min(start+batch, len(chunks))
|
||||
var vals []string
|
||||
var args []any
|
||||
for i := start; i < end; i++ {
|
||||
vals = append(vals, `(?, ?, ?, ?, ?, ?::vector)`)
|
||||
args = append(args, c.OrgID, c.ID, d.ID, i, chunks[i].Body, vectorText(chunks[i].Embedding))
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `INSERT INTO kb_chunks (org_id, collection, doc_id, ord, body, embedding) VALUES `+
|
||||
strings.Join(vals, ", "), args...); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return KBDocument{}, false, err
|
||||
}
|
||||
return d, changed, nil
|
||||
}
|
||||
|
||||
// vectorText is pgvector's text form of an embedding.
|
||||
func vectorText(v []float32) string {
|
||||
var b strings.Builder
|
||||
b.Grow(len(v) * 10)
|
||||
b.WriteByte('[')
|
||||
for i, f := range v {
|
||||
if i > 0 {
|
||||
b.WriteByte(',')
|
||||
}
|
||||
b.WriteString(strconv.FormatFloat(float64(f), 'g', -1, 32))
|
||||
}
|
||||
b.WriteByte(']')
|
||||
return b.String()
|
||||
}
|
||||
|
||||
// Documents lists a collection's documents, by name.
|
||||
func (s *DB) Documents(ctx context.Context, collection string) ([]KBDocument, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return queryAll(ctx, s.db, `SELECT id, collection, name, digest, bytes, chunks, added_at, added_by
|
||||
FROM kb_documents WHERE collection = ? ORDER BY name`, []any{collection},
|
||||
func(r *sql.Rows) (KBDocument, error) {
|
||||
var d KBDocument
|
||||
var at int64
|
||||
err := r.Scan(&d.ID, &d.Collection, &d.Name, &d.Digest, &d.Bytes, &d.Chunks, &at, &d.AddedBy)
|
||||
d.AddedAt = fromMillis(at)
|
||||
return d, err
|
||||
})
|
||||
}
|
||||
|
||||
// DeleteDocument removes a document and its chunks, reporting whether it was
|
||||
// there.
|
||||
func (s *DB) DeleteDocument(ctx context.Context, collection, name string) (bool, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return false, err
|
||||
}
|
||||
res, err := s.db.ExecContext(ctx, `DELETE FROM kb_documents WHERE collection = ? AND name = ?`, collection, name)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
n, _ := res.RowsAffected()
|
||||
return n > 0, nil
|
||||
}
|
||||
|
||||
// KBSearch is one search: the caller's organization, the collections it may
|
||||
// read and asked for, the query embedded by each collection's model, and the
|
||||
// query's text for the full-text arm.
|
||||
type KBSearch struct {
|
||||
OrgID string
|
||||
Collections []KBCollection
|
||||
// Embeddings is the query embedded by each model, keyed by model name.
|
||||
Embeddings map[string][]float32
|
||||
Text string
|
||||
K int
|
||||
}
|
||||
|
||||
// rrfK is reciprocal rank fusion's constant: 60, as in the paper and nearly
|
||||
// everywhere it is used.
|
||||
const rrfK = 60
|
||||
|
||||
// SearchChunks finds the best chunks for a query across collections: a
|
||||
// nearest-neighbour search in each collection and a full-text search across
|
||||
// them, fused by rank. Every collection must be in OrgID; the caller decides
|
||||
// which ones the requester may read, and the table refuses rows outside them
|
||||
// regardless.
|
||||
func (s *DB) SearchChunks(ctx context.Context, q KBSearch) ([]KBHit, error) {
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if q.K <= 0 || len(q.Collections) == 0 || q.OrgID == "" {
|
||||
return nil, nil
|
||||
}
|
||||
ids := make([]string, 0, len(q.Collections))
|
||||
quoted := make([]string, 0, len(q.Collections))
|
||||
for _, c := range q.Collections {
|
||||
if c.OrgID != q.OrgID || !kbIDPattern.MatchString(c.ID) {
|
||||
return nil, fmt.Errorf("collection %s is not in organization %s", c.ID, q.OrgID)
|
||||
}
|
||||
ids = append(ids, c.ID)
|
||||
quoted = append(quoted, sqlLiteral(c.ID))
|
||||
}
|
||||
type key struct {
|
||||
doc string
|
||||
ord int
|
||||
}
|
||||
hits := map[key]*KBHit{}
|
||||
add := func(h KBHit, rank int) {
|
||||
k := key{h.DocID, h.Ord}
|
||||
if have, ok := hits[k]; ok {
|
||||
have.Score += 1.0 / float64(rrfK+rank)
|
||||
return
|
||||
}
|
||||
h.Score = 1.0 / float64(rrfK+rank)
|
||||
hits[k] = &h
|
||||
}
|
||||
scan := func(rows *sql.Rows) ([]KBHit, error) {
|
||||
defer rows.Close()
|
||||
var out []KBHit
|
||||
for rows.Next() {
|
||||
var h KBHit
|
||||
if err := rows.Scan(&h.Collection, &h.DocID, &h.Document, &h.Ord, &h.Body); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, h)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
err := s.inTx(ctx, func(tx conn) error {
|
||||
if err := kbScope(ctx, tx, q.OrgID, ids); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, c := range q.Collections {
|
||||
v, ok := q.Embeddings[c.EmbedModel]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if len(v) != c.Dims {
|
||||
return fmt.Errorf("the query has %d dimensions; collection %s takes %d", len(v), c.Name, c.Dims)
|
||||
}
|
||||
// The literal collection and cast are what let the planner use
|
||||
// that collection's partial index.
|
||||
cast := `::vector(` + strconv.Itoa(c.Dims) + `)`
|
||||
rows, err := tx.QueryContext(ctx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
|
||||
FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id
|
||||
WHERE k.org_id = ? AND k.collection = `+sqlLiteral(c.ID)+`
|
||||
ORDER BY k.embedding`+cast+` <=> ?`+cast+` LIMIT ?`,
|
||||
q.OrgID, vectorText(v), q.K)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
got, err := scan(rows)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i, h := range got {
|
||||
add(h, i+1)
|
||||
}
|
||||
}
|
||||
if tsq := orQuery(q.Text); tsq != "" {
|
||||
rows, err := tx.QueryContext(ctx, `SELECT k.collection, k.doc_id, d.name, k.ord, k.body
|
||||
FROM kb_chunks k JOIN kb_documents d ON d.id = k.doc_id, to_tsquery('simple', ?) q
|
||||
WHERE k.org_id = ? AND k.collection IN (`+strings.Join(quoted, ", ")+`) AND k.tsv @@ q
|
||||
ORDER BY ts_rank(k.tsv, q) DESC LIMIT ?`,
|
||||
tsq, q.OrgID, q.K)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
got, err := scan(rows)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for i, h := range got {
|
||||
add(h, i+1)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]KBHit, 0, len(hits))
|
||||
for _, h := range hits {
|
||||
out = append(out, *h)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Score != out[j].Score {
|
||||
return out[i].Score > out[j].Score
|
||||
}
|
||||
if out[i].DocID != out[j].DocID {
|
||||
return out[i].DocID < out[j].DocID
|
||||
}
|
||||
return out[i].Ord < out[j].Ord
|
||||
})
|
||||
if len(out) > q.K {
|
||||
out = out[:q.K]
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// orQuery turns a question into a tsquery matching any of its words. All of
|
||||
// them (plainto_tsquery) would match almost nothing a person types as a
|
||||
// question; the rank then prefers chunks with more of them.
|
||||
func orQuery(text string) string {
|
||||
seen := map[string]bool{}
|
||||
var words []string
|
||||
// Not lowercased here: to_tsquery does that with the database's own
|
||||
// rules, the same ones to_tsvector used on the chunks — Go's lower case
|
||||
// of a Turkish İ is not the database's.
|
||||
for _, w := range strings.FieldsFunc(text, func(r rune) bool {
|
||||
return !unicode.IsLetter(r) && !unicode.IsDigit(r) && !unicode.IsMark(r)
|
||||
}) {
|
||||
if len([]rune(w)) < 2 || seen[w] {
|
||||
continue
|
||||
}
|
||||
seen[w] = true
|
||||
words = append(words, w)
|
||||
if len(words) == 32 {
|
||||
break
|
||||
}
|
||||
}
|
||||
return strings.Join(words, " | ")
|
||||
}
|
||||
|
||||
// kbOwnedBy counts the collections a scope owns, for refusing to delete it
|
||||
// from under them. Zero on a store without knowledge bases.
|
||||
func (s *DB) kbOwnedBy(ctx context.Context, c conn, scope Scope, id string) (int, error) {
|
||||
if s.dialect != PostgresDialect {
|
||||
return 0, nil
|
||||
}
|
||||
var n int
|
||||
q := `SELECT COUNT(*) FROM kb_collections WHERE owner_scope = ? AND owner_id = ?`
|
||||
args := []any{string(scope), id}
|
||||
if scope == ScopeOrg {
|
||||
q, args = `SELECT COUNT(*) FROM kb_collections WHERE org_id = ?`, []any{id}
|
||||
}
|
||||
err := c.QueryRowContext(ctx, q, args...).Scan(&n)
|
||||
return n, err
|
||||
}
|
||||
|
||||
// kbForgetScope removes what a deleted scope had: its grants, and for a
|
||||
// person the collections they owned.
|
||||
func (s *DB) kbForgetScope(ctx context.Context, c conn, scope Scope, id string) error {
|
||||
if s.dialect != PostgresDialect {
|
||||
return nil
|
||||
}
|
||||
if _, err := c.ExecContext(ctx, `DELETE FROM kb_grants WHERE scope = ? AND scope_id = ?`, string(scope), id); err != nil {
|
||||
return err
|
||||
}
|
||||
if scope != ScopeUser {
|
||||
return nil
|
||||
}
|
||||
owned, err := queryAll(ctx, c, `SELECT id FROM kb_collections WHERE owner_scope = 'user' AND owner_id = ?`,
|
||||
[]any{id}, func(r *sql.Rows) (string, error) {
|
||||
var v string
|
||||
return v, r.Scan(&v)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, cid := range owned {
|
||||
if _, err := c.ExecContext(ctx, `DELETE FROM kb_collections WHERE id = ?`, cid); err != nil {
|
||||
return err
|
||||
}
|
||||
if kbIDPattern.MatchString(cid) {
|
||||
if _, err := c.ExecContext(ctx, `DROP INDEX IF EXISTS `+kbVectorIndex(cid)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// kbDropPartition drops a deleted organization's chunk partition, empty by
|
||||
// then: an organization with collections is not deleted.
|
||||
func (s *DB) kbDropPartition(ctx context.Context, c conn, scope Scope, id string) error {
|
||||
if s.dialect != PostgresDialect || scope != ScopeOrg || s.kbErr != nil {
|
||||
return nil
|
||||
}
|
||||
_, err := c.ExecContext(ctx, `DROP TABLE IF EXISTS `+kbPartition(id))
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,362 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"math"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestKnowledgeIsOffOnSQLite(t *testing.T) {
|
||||
s := openDB(t)
|
||||
tree(t, s)
|
||||
err := s.KnowledgeAvailable()
|
||||
if !errors.Is(err, ErrKnowledgeOff) || !strings.Contains(err.Error(), "SQLite") {
|
||||
t.Fatalf("SQLite store says %v", err)
|
||||
}
|
||||
if _, err := s.CreateCollection(context.Background(), KBCollection{
|
||||
OwnerScope: ScopeProject, OwnerID: "proj1", Name: "x", EmbedModel: "m", Dims: 3,
|
||||
}); !errors.Is(err, ErrKnowledgeOff) {
|
||||
t.Errorf("creating a collection on SQLite gave %v", err)
|
||||
}
|
||||
// Deleting from the tree still works without the tables.
|
||||
if err := s.DeleteProject(context.Background(), "proj1"); err != nil {
|
||||
t.Errorf("deleting a project on SQLite: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// kbTwoOrgs is org1 (team1/proj1, team2/proj2) and org2 (team3/proj3).
|
||||
func kbTwoOrgs(t *testing.T, s *DB) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
tree(t, s)
|
||||
for _, err := range []error{
|
||||
s.PutTeam(ctx, Team{ID: "team2", OrgID: "org1", Name: "Finance"}),
|
||||
s.PutProject(ctx, Project{ID: "proj2", TeamID: "team2", Name: "Payroll"}),
|
||||
s.PutOrg(ctx, Org{ID: "org2", Name: "Other"}),
|
||||
s.PutTeam(ctx, Team{ID: "team3", OrgID: "org2", Name: "Research"}),
|
||||
s.PutProject(ctx, Project{ID: "proj3", TeamID: "team3", Name: "Chat"}),
|
||||
} {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// unit is a 3-dimensional unit vector pointing mostly along axis i.
|
||||
func unit(i int) []float32 {
|
||||
v := []float32{0.1, 0.1, 0.1}
|
||||
v[i] = 1
|
||||
n := float32(math.Sqrt(float64(v[0]*v[0] + v[1]*v[1] + v[2]*v[2])))
|
||||
for j := range v {
|
||||
v[j] /= n
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func mustCollection(t *testing.T, s *DB, scope Scope, owner, org, name string) KBCollection {
|
||||
t.Helper()
|
||||
c, err := s.CreateCollection(context.Background(), KBCollection{
|
||||
OwnerScope: scope, OwnerID: owner, OrgID: org, Name: name, EmbedModel: "emb", Dims: 3,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("creating %s: %v", name, err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func mustDoc(t *testing.T, s *DB, c KBCollection, name string, bodies ...string) {
|
||||
t.Helper()
|
||||
var chunks []KBChunk
|
||||
for i, b := range bodies {
|
||||
chunks = append(chunks, KBChunk{Body: b, Embedding: unit(i % 3)})
|
||||
}
|
||||
if _, _, err := s.PutDocument(context.Background(), c, KBDocument{Name: name, Digest: name + "-1", Bytes: 10}, chunks); err != nil {
|
||||
t.Fatalf("adding %s: %v", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
func openKB(t *testing.T) *DB {
|
||||
t.Helper()
|
||||
s := openPG(t)
|
||||
if err := s.KnowledgeAvailable(); err != nil {
|
||||
t.Skipf("the test database has no pgvector: %v", err)
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
func names(cs []KBCollection) string {
|
||||
var out []string
|
||||
for _, c := range cs {
|
||||
out = append(out, c.Name)
|
||||
}
|
||||
return strings.Join(out, ",")
|
||||
}
|
||||
|
||||
func TestPostgresCollectionsReachOnlyTheirChain(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
mustCollection(t, s, ScopeOrg, "org1", "", "handbook")
|
||||
mustCollection(t, s, ScopeTeam, "team1", "", "research")
|
||||
pay := mustCollection(t, s, ScopeProject, "proj2", "", "payroll")
|
||||
mustCollection(t, s, ScopeProject, "proj3", "", "other-org")
|
||||
|
||||
read := func(e Effective) string {
|
||||
t.Helper()
|
||||
got, err := s.CollectionsFor(ctx, e, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return names(got)
|
||||
}
|
||||
inProj1 := Effective{KeyID: "k1", ProjectID: "proj1", TeamID: "team1", OrgID: "org1"}
|
||||
if got := read(inProj1); got != "handbook,research" {
|
||||
t.Errorf("proj1 reads %q", got)
|
||||
}
|
||||
if got := read(Effective{KeyID: "k3", ProjectID: "proj3", TeamID: "team3", OrgID: "org2"}); got != "other-org" {
|
||||
t.Errorf("org2 reads %q", got)
|
||||
}
|
||||
// A personal key with no project reads nothing, whatever its person
|
||||
// belongs to.
|
||||
if got := read(Effective{KeyID: "kp", UserID: "u1"}); got != "" {
|
||||
t.Errorf("a caller outside any organization reads %q", got)
|
||||
}
|
||||
|
||||
// A grant to a key adds that collection for that key alone.
|
||||
_, k := mustKey(t, s, "k1", "proj1")
|
||||
if err := s.Grant(ctx, KBGrant{Scope: ScopeKey, ScopeID: k.ID, Collection: pay.ID, Perm: KBRead}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := read(inProj1); got != "handbook,payroll,research" {
|
||||
t.Errorf("proj1 with a key grant reads %q", got)
|
||||
}
|
||||
if got := read(Effective{KeyID: "k-other", ProjectID: "proj1", TeamID: "team1", OrgID: "org1"}); got != "handbook,research" {
|
||||
t.Errorf("another key of proj1 reads %q", got)
|
||||
}
|
||||
// Read is not write.
|
||||
w, err := s.CollectionsFor(ctx, inProj1, true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if strings.Contains(names(w), "payroll") {
|
||||
t.Errorf("a read grant gave write: %s", names(w))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresGrantsNeverCrossOrganizations(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
|
||||
for _, g := range []KBGrant{
|
||||
{Scope: ScopeOrg, ScopeID: "org2"},
|
||||
{Scope: ScopeTeam, ScopeID: "team3"},
|
||||
{Scope: ScopeProject, ScopeID: "proj3"},
|
||||
} {
|
||||
g.Collection, g.Perm = c.ID, KBRead
|
||||
if err := s.Grant(ctx, g); err == nil || !strings.Contains(err.Error(), "another organization") {
|
||||
t.Errorf("granting to %s %s gave %v", g.Scope, g.ScopeID, err)
|
||||
}
|
||||
}
|
||||
_, k3 := mustKey(t, s, "k3", "proj3")
|
||||
if err := s.Grant(ctx, KBGrant{Scope: ScopeKey, ScopeID: k3.ID, Collection: c.ID, Perm: KBRead}); err == nil {
|
||||
t.Error("a key of another organization was granted a collection")
|
||||
}
|
||||
// A person who belongs to nothing in the organization cannot own or be
|
||||
// granted one there.
|
||||
if err := s.CreateUser(ctx, User{ID: "u1", Email: "u1@example.com", Name: "u", PasswordHash: "x",
|
||||
State: UserActive, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Grant(ctx, KBGrant{Scope: ScopeUser, ScopeID: "u1", Collection: c.ID, Perm: KBRead}); err == nil {
|
||||
t.Error("a person with no membership in the organization was granted a collection")
|
||||
}
|
||||
if err := s.AddMembership(ctx, "u1", ScopeProject, "proj2", "member", "t", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Grant(ctx, KBGrant{Scope: ScopeUser, ScopeID: "u1", Collection: c.ID, Perm: KBRead}); err != nil {
|
||||
t.Errorf("a member of a project in the organization: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresSearchReturnsOnlyAllowedChunks(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
mine := mustCollection(t, s, ScopeProject, "proj1", "", "mine")
|
||||
secret := mustCollection(t, s, ScopeProject, "proj2", "", "secret")
|
||||
foreign := mustCollection(t, s, ScopeProject, "proj3", "", "foreign")
|
||||
mustDoc(t, s, mine, "a.md", "vacation policy is twenty days", "expenses go through finance")
|
||||
mustDoc(t, s, secret, "salaries.md", "vacation salaries are confidential")
|
||||
mustDoc(t, s, foreign, "theirs.md", "vacation at the other company")
|
||||
|
||||
hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{mine},
|
||||
Embeddings: map[string][]float32{"emb": unit(0)}, Text: "vacation policy", K: 5})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(hits) == 0 || hits[0].Body != "vacation policy is twenty days" || hits[0].Document != "a.md" {
|
||||
t.Fatalf("best hit %+v", hits)
|
||||
}
|
||||
for _, h := range hits {
|
||||
if h.Collection != mine.ID {
|
||||
t.Errorf("a search of %s returned a chunk of %s: %q", mine.Name, h.Collection, h.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// A collection of another organization is refused, not searched.
|
||||
if _, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{mine, foreign},
|
||||
Embeddings: map[string][]float32{"emb": unit(0)}, Text: "vacation", K: 5}); err == nil {
|
||||
t.Error("a search across organizations was run")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresRowSecurityHoldsWithoutTheFilter(t *testing.T) {
|
||||
// The second line: a query that forgets its WHERE gets nothing outside
|
||||
// the scope its transaction set, and nothing at all without one.
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
mine := mustCollection(t, s, ScopeProject, "proj1", "", "mine")
|
||||
other := mustCollection(t, s, ScopeProject, "proj2", "", "other")
|
||||
mustDoc(t, s, mine, "a.md", "one", "two")
|
||||
mustDoc(t, s, other, "b.md", "three")
|
||||
|
||||
count := func(org string, colls ...string) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
err := s.inTx(ctx, func(c conn) error {
|
||||
if org != "" {
|
||||
if err := kbScope(ctx, c, org, colls); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return c.QueryRowContext(ctx, `SELECT COUNT(*) FROM kb_chunks`).Scan(&n)
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
if n := count(""); n != 0 {
|
||||
t.Errorf("with no scope set, %d chunks are visible", n)
|
||||
}
|
||||
if n := count("org1", mine.ID); n != 2 {
|
||||
t.Errorf("scoped to one collection, %d chunks are visible", n)
|
||||
}
|
||||
if n := count("org2", mine.ID); n != 0 {
|
||||
t.Errorf("a collection named under the wrong organization shows %d chunks", n)
|
||||
}
|
||||
// Nor can a write put a chunk outside its scope.
|
||||
err := s.inTx(ctx, func(c conn) error {
|
||||
if err := kbScope(ctx, c, "org1", []string{mine.ID}); err != nil {
|
||||
return err
|
||||
}
|
||||
docs, _ := s.Documents(ctx, other.ID)
|
||||
_, err := c.ExecContext(ctx, `INSERT INTO kb_chunks (org_id, collection, doc_id, ord, body, embedding)
|
||||
VALUES ('org1', ?, ?, 9, 'x', '[1,0,0]')`, other.ID, docs[0].ID)
|
||||
return err
|
||||
})
|
||||
if err == nil || !strings.Contains(err.Error(), "row-level security") {
|
||||
t.Errorf("writing into another collection gave %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresDocumentsReplaceWholeAndSkipTheSame(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
c := mustCollection(t, s, ScopeProject, "proj1", "", "docs")
|
||||
put := func(digest string, bodies ...string) bool {
|
||||
t.Helper()
|
||||
var chunks []KBChunk
|
||||
for _, b := range bodies {
|
||||
chunks = append(chunks, KBChunk{Body: b, Embedding: unit(0)})
|
||||
}
|
||||
_, changed, err := s.PutDocument(ctx, c, KBDocument{Name: "a.md", Digest: digest}, chunks)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return changed
|
||||
}
|
||||
if !put("d1", "old one", "old two") {
|
||||
t.Error("a new document was not a change")
|
||||
}
|
||||
if put("d1", "ignored") {
|
||||
t.Error("the same digest again was a change")
|
||||
}
|
||||
if !put("d2", "new") {
|
||||
t.Error("a new digest was not a change")
|
||||
}
|
||||
hits, err := s.SearchChunks(ctx, KBSearch{OrgID: "org1", Collections: []KBCollection{c},
|
||||
Embeddings: map[string][]float32{"emb": unit(0)}, K: 10})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(hits) != 1 || hits[0].Body != "new" {
|
||||
t.Errorf("after replacing, the collection holds %+v", hits)
|
||||
}
|
||||
if _, _, err := s.PutDocument(ctx, c, KBDocument{Name: "b.md", Digest: "x"},
|
||||
[]KBChunk{{Body: "x", Embedding: []float32{1, 0}}}); err == nil {
|
||||
t.Error("a chunk of the wrong dimension was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresScopesWithCollectionsAreNotDeleted(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
c := mustCollection(t, s, ScopeProject, "proj2", "", "docs")
|
||||
mustDoc(t, s, c, "a.md", "x")
|
||||
if err := s.DeleteProject(ctx, "proj2"); !errors.Is(err, ErrNotEmpty) || !strings.Contains(err.Error(), "collection") {
|
||||
t.Errorf("deleting a project with a collection gave %v", err)
|
||||
}
|
||||
if err := s.DeleteCollection(ctx, c.ID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.DeleteProject(ctx, "proj2"); err != nil {
|
||||
t.Errorf("deleting the project once its collection is gone: %v", err)
|
||||
}
|
||||
var n int
|
||||
if err := s.raw.QueryRowContext(ctx, `SELECT COUNT(*) FROM pg_indexes WHERE indexname = $1`,
|
||||
kbVectorIndex(c.ID)).Scan(&n); err != nil || n != 0 {
|
||||
t.Errorf("the collection's index is left behind (%d, %v)", n, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostgresAPersonsCollectionsGoWithTheAccount(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
s := openKB(t)
|
||||
kbTwoOrgs(t, s)
|
||||
if err := s.CreateUser(ctx, User{ID: "u1", Email: "u1@example.com", Name: "u", PasswordHash: "x",
|
||||
State: UserActive, CreatedAt: time.Now()}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.CreateCollection(ctx, KBCollection{OwnerScope: ScopeUser, OwnerID: "u1", OrgID: "org1",
|
||||
Name: "notes", EmbedModel: "emb", Dims: 3}); err == nil {
|
||||
t.Error("a person outside the organization made a collection in it")
|
||||
}
|
||||
if err := s.AddMembership(ctx, "u1", ScopeTeam, "team1", "member", "t", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := mustCollection(t, s, ScopeUser, "u1", "org1", "notes")
|
||||
mustDoc(t, s, c, "n.md", "x")
|
||||
if err := s.DeleteUser(ctx, "u1", time.Now()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := s.Collection(ctx, c.ID); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("the person's collection outlived the account: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOrQueryMatchesAnyWord(t *testing.T) {
|
||||
if got := orQuery("Yıllık izin kaç gün? İzin & 'x' -- a"); got != "Yıllık | izin | kaç | gün | İzin" {
|
||||
t.Errorf("orQuery gave %q", got)
|
||||
}
|
||||
if got := orQuery("?!"); got != "" {
|
||||
t.Errorf("no words gave %q", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
-- Knowledge bases: collections of a company's documents that requests are
|
||||
-- answered from (BACKLOG F). PostgreSQL only: there is no SQLite file of this
|
||||
-- number, and on SQLite the feature is off.
|
||||
--
|
||||
-- The chunks themselves are not here. They need the pgvector extension, which
|
||||
-- only a superuser can create and which a cluster may not have installed; a
|
||||
-- migration that failed for want of it would stop the control plane from
|
||||
-- starting at all. The store creates kb_chunks when it opens, if it can
|
||||
-- (knowledge.go), and says why not when it cannot.
|
||||
|
||||
-- A collection has one owner in the tenancy tree and lives inside one
|
||||
-- organization, which no grant can cross.
|
||||
CREATE TABLE kb_collections (
|
||||
id TEXT PRIMARY KEY,
|
||||
org_id TEXT NOT NULL REFERENCES organizations(id),
|
||||
owner_scope TEXT NOT NULL,
|
||||
owner_id TEXT NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
-- Vectors from two models are not comparable, so a collection is embedded
|
||||
-- by one model and its dimension is fixed with it.
|
||||
embed_model TEXT NOT NULL,
|
||||
dims INTEGER NOT NULL,
|
||||
created_at BIGINT NOT NULL,
|
||||
created_by TEXT NOT NULL DEFAULT '',
|
||||
UNIQUE (org_id, name)
|
||||
);
|
||||
CREATE INDEX kb_collections_by_owner ON kb_collections(owner_scope, owner_id);
|
||||
|
||||
-- Who else may read or write a collection. Like guard_attachments, keyed by
|
||||
-- scope with everything below the scope included; unlike guard policies,
|
||||
-- grants add up rather than narrow.
|
||||
CREATE TABLE kb_grants (
|
||||
scope TEXT NOT NULL,
|
||||
scope_id TEXT NOT NULL,
|
||||
collection TEXT NOT NULL REFERENCES kb_collections(id) ON DELETE CASCADE,
|
||||
perm TEXT NOT NULL,
|
||||
added_at BIGINT NOT NULL,
|
||||
added_by TEXT NOT NULL DEFAULT '',
|
||||
PRIMARY KEY (scope, scope_id, collection)
|
||||
);
|
||||
CREATE INDEX kb_grants_by_collection ON kb_grants(collection);
|
||||
|
||||
-- A document as added, by name within its collection. The digest is what
|
||||
-- decides whether adding it again changes anything.
|
||||
CREATE TABLE kb_documents (
|
||||
id TEXT PRIMARY KEY,
|
||||
collection TEXT NOT NULL REFERENCES kb_collections(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
digest TEXT NOT NULL,
|
||||
bytes BIGINT NOT NULL,
|
||||
chunks INTEGER NOT NULL,
|
||||
added_at BIGINT NOT NULL,
|
||||
added_by TEXT NOT NULL DEFAULT '',
|
||||
UNIQUE (collection, name)
|
||||
);
|
||||
@@ -55,9 +55,13 @@ func OpenPostgres(ctx context.Context, dsn string) (*DB, error) {
|
||||
_ = db.Close()
|
||||
return nil, err
|
||||
}
|
||||
return &DB{
|
||||
s := &DB{
|
||||
raw: db,
|
||||
dialect: PostgresDialect,
|
||||
db: bound{c: PostgresDialect, to: db},
|
||||
}, nil
|
||||
}
|
||||
// Not fatal: a store without pgvector serves everything but knowledge
|
||||
// bases, and says why when somebody asks for one.
|
||||
s.kbErr = s.ensureKnowledge(ctx)
|
||||
return s, nil
|
||||
}
|
||||
@@ -48,6 +48,10 @@ type DB struct {
|
||||
// personal is what a person's own use is held to when nobody wrote a
|
||||
// limit for them; see SetPersonalDefaults.
|
||||
personal *Limits
|
||||
|
||||
// kbErr is why this store cannot hold knowledge bases, nil when it can;
|
||||
// set once at open (knowledge.go).
|
||||
kbErr error
|
||||
}
|
||||
|
||||
// OpenSQLite opens (creating if needed) a database and brings it up to date.
|
||||
|
||||
Reference in new issue
Block a user