Knowledge bases in the store: collections, grants and chunks kept apart per tenant
Collections owned by an organization, team, project or person; grants that add up down the tree and never cross an organization; chunks in PostgreSQL with pgvector, partitioned by organization, an HNSW index per collection, and row-level security as a second line behind each query's filter. Search fuses nearest-neighbour and full-text ranks. Off on SQLite and without pgvector, saying why. server init and the HA database setup install and enable it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
0a7fb821d8
commit
d35630defe
11 files changed
+1408
-25
No files matched your search
+35
-19
@@ -829,25 +829,41 @@ ACL yet); a personal key or session names a project or team and gets that
|
||||
chain only, never the union of its person's memberships; PostgreSQL store
|
||||
only, off on SQLite with a sentence saying why.
|
||||
|
||||
### F1. Store: collections, grants, chunks (M)
|
||||
- `pgvector` ≥ 0.8 installed on every Patroni member by `server init`
|
||||
(`internal/cli/serverinit_postgres.go`), `CREATE EXTENSION` in the
|
||||
migration; the SQLite migration has no tables, and the store reports the
|
||||
feature unavailable.
|
||||
- Tables: `kb_collections` (id, org_id, owner `team:…`/`project:…`/`user:…`,
|
||||
name, embed_model, dims, created_at), `kb_grants` (scope, scope_id,
|
||||
collection, perm read|write — like `guard_attachments`), `kb_documents`
|
||||
(name, digest, bytes, state, chunk count), `kb_chunks` (org_id,
|
||||
collection, doc_id, ord, body, tsv, embedding) `PARTITION BY LIST
|
||||
(org_id)`, a partition per organization, HNSW and GIN indexes per
|
||||
partition.
|
||||
- Row-level security on `kb_chunks` against `current_setting('llmgw.org')`
|
||||
and `llmgw.collections`, retrieval through its own role.
|
||||
- Rules, refused before anything is written: a grant never crosses
|
||||
organizations; owner and grants resolve to existing scopes; deleting a
|
||||
project/team/org and `users forget` delete their collections (`forget.go`).
|
||||
- Done: store tests on PostgreSQL, including an RLS test that a query
|
||||
without the settings returns no rows.
|
||||
### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
|
||||
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
|
||||
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
|
||||
(read|write, any scope, never across organizations; a key needs a
|
||||
project; a person needs a membership in the organization),
|
||||
`kb_documents` (digest: the same again is no change, a new one replaces
|
||||
in one transaction).
|
||||
- `kb_chunks` is created when the store opens, not in a migration (a
|
||||
migration needing an extension would stop the control plane; one that
|
||||
skipped it would never run again): partitioned by organization (hashed
|
||||
partition name), `tsv` generated (`simple`), GIN on the parent, one
|
||||
partial HNSW index per collection on its organization's partition
|
||||
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
|
||||
20k rows: partition pruned, that index used, policy applied.
|
||||
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
|
||||
(set with set_config per transaction), one role — the table owner — not
|
||||
a separate one. Chunks are deleted by the foreign-key cascade from their
|
||||
document, which RLS does not block.
|
||||
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
|
||||
no organization (personal key without a project) reads nothing.
|
||||
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
|
||||
the partial index is used) plus a full-text arm with any word
|
||||
(`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
|
||||
- Deleting a project/team/org that owns collections is refused like one
|
||||
with members (changed from "deleted with them": every other delete in the
|
||||
tree refuses); DeleteUser/ForgetUser delete the person's collections;
|
||||
deleting an organization drops its partition; grants to a deleted scope go.
|
||||
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
|
||||
it; pgha installs it on every member at setup and enables it on the
|
||||
leader on every run. Neither stops on failure.
|
||||
Left: a cluster set up before (the PoC) gets no package from a rerun of
|
||||
pgha on a running cluster — install by hand or add an upgrade step;
|
||||
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
|
||||
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
|
||||
pgvector database as a non-superuser (a superuser bypasses RLS).
|
||||
|
||||
### F2. Ingest (M)
|
||||
- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add
|
||||
|
||||
Reference in new issue
Block a user