Knowledge bases in the store: collections, grants and chunks kept apart per tenant

Collections owned by an organization, team, project or person; grants that
add up down the tree and never cross an organization; chunks in PostgreSQL
with pgvector, partitioned by organization, an HNSW index per collection, and
row-level security as a second line behind each query's filter. Search fuses
nearest-neighbour and full-text ranks. Off on SQLite and without pgvector,
saying why. server init and the HA database setup install and enable it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 12:47:57 +02:00
1 parent 0a7fb821d8
commit d35630defe
11 files changed
+1408 -25

No files matched your search

+35 -19
View File
@@ -829,25 +829,41 @@ ACL yet); a personal key or session names a project or team and gets that
chain only, never the union of its person's memberships; PostgreSQL store
only, off on SQLite with a sentence saying why.
### F1. Store: collections, grants, chunks (M)
- `pgvector` ≥ 0.8 installed on every Patroni member by `server init`
(`internal/cli/serverinit_postgres.go`), `CREATE EXTENSION` in the
migration; the SQLite migration has no tables, and the store reports the
feature unavailable.
- Tables: `kb_collections` (id, org_id, owner `team:…`/`project:…`/`user:…`,
name, embed_model, dims, created_at), `kb_grants` (scope, scope_id,
collection, perm read|write — like `guard_attachments`), `kb_documents`
(name, digest, bytes, state, chunk count), `kb_chunks` (org_id,
collection, doc_id, ord, body, tsv, embedding) `PARTITION BY LIST
(org_id)`, a partition per organization, HNSW and GIN indexes per
partition.
- Row-level security on `kb_chunks` against `current_setting('llmgw.org')`
and `llmgw.collections`, retrieval through its own role.
- Rules, refused before anything is written: a grant never crosses
organizations; owner and grants resolve to existing scopes; deleting a
project/team/org and `users forget` delete their collections (`forget.go`).
- Done: store tests on PostgreSQL, including an RLS test that a query
without the settings returns no rows.
### F1. Store: collections, grants, chunks (M) — done 2026-10-08 (branch `rag`)
Built (`internal/store/knowledge.go`, migration 0035 PostgreSQL only):
- `kb_collections` (owner scope/id, org, embed_model, dims), `kb_grants`
(read|write, any scope, never across organizations; a key needs a
project; a person needs a membership in the organization),
`kb_documents` (digest: the same again is no change, a new one replaces
in one transaction).
- `kb_chunks` is created when the store opens, not in a migration (a
migration needing an extension would stop the control plane; one that
skipped it would never run again): partitioned by organization (hashed
partition name), `tsv` generated (`simple`), GIN on the parent, one
partial HNSW index per collection on its organization's partition
(`embedding::vector(dims)`, `WHERE collection = '<id>'`). EXPLAIN on
20k rows: partition pruned, that index used, policy applied.
- RLS FORCEd on `kb_chunks` against `llmgw.kb_org`/`llmgw.kb_collections`
(set with set_config per transaction), one role — the table owner — not
a separate one. Chunks are deleted by the foreign-key cascade from their
document, which RLS does not block.
- `CollectionsFor(Effective, write)`: owner or grant anywhere on the chain;
no organization (personal key without a project) reads nothing.
- `SearchChunks`: per collection nearest-neighbour (literal id and cast so
the partial index is used) plus a full-text arm with any word
(`a | b`, lowercased by the database, not Go), fused by reciprocal rank.
- Deleting a project/team/org that owns collections is refused like one
with members (changed from "deleted with them": every other delete in the
tree refuses); DeleteUser/ForgetUser delete the person's collections;
deleting an organization drops its partition; grants to a deleted scope go.
- `server init` installs `postgresql-N-pgvector` / `pgvector_N` and enables
it; pgha installs it on every member at setup and enables it on the
leader on every run. Neither stops on failure.
Left: a cluster set up before (the PoC) gets no package from a rerun of
pgha on a running cluster — install by hand or add an upgrade step;
`server restore`/`CopyTo` do not carry kb tables (SQLite → PostgreSQL only,
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).
### F2. Ingest (M)
- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add