llmgw/docs/knowledge.md

Knowledge bases

A knowledge base is a collection of a company's documents that the cluster can answer from. People put their documents into it from their own machines with llmgw-kb. The tool reads the files on that machine, cuts them into passages ("chunks"), has the gateway embed each passage with the person's own key, and sends the text and the vectors. The files themselves never leave the machine.

Status. Collections, access and filling them work now. Answering requests from them (the gateway finding the passages a question needs and putting them in front of the model) is the next step and is not in this build yet. Until then a collection can be searched by an administrator (llmgw ctl kb search) to check what it holds.

Why retrieval, and not training

Retrieval is the tool for knowledge: policies, procedures, contracts. The answer can name its source, a changed document is sent again in minutes, and a passage only ever reaches people allowed to read its collection.

Training a model on the documents (LoRA) changes how it writes, not what it reliably knows. It has no access control: whatever goes into the weights is available to everyone who may use them.

What it needs

• The PostgreSQL store with pgvector. On a SQLite store knowledge bases are off, and llmgw ctl kb list and the console say why.

  • llmgw server init and the high-availability database setup install pgvector and enable it.
  • A cluster set up before this build needs it by hand, once:
    1. Install the package on every database machine: sudo apt-get install postgresql-16-pgvector (use the server's version number).
    2. On the leader, enable it in llmgw's database: sudo -u postgres psql -d llmgw -c 'CREATE EXTENSION vector'.
    3. Restart the control planes: llmgw restart.

• An embedding model in the catalog, with capabilities.embeddings. Two good choices handle Turkish and run on CPUs: BAAI/bge-m3 (1024 dimensions) and Qwen/Qwen3-Embedding-0.6B.

  Add it as any model is added (Serve a model in the console, or the catalog), with its entry saying that it embeds:

  capabilities:
    embeddings: true
  

  A collection stays with the model it was made with: vectors from two models cannot be compared.

For administrators

Make a collection

A collection belongs to one part of the tenancy tree: an organization, a team, a project or a person. Whoever is in the owner, at any level below it, may read it and add to it.

llmgw ctl kb create team:Acme/hr policies bge-m3
llmgw ctl kb create user:USER-ID notes bge-m3 --org Acme

The model's dimension is asked of the model itself, so the model has to be able to serve when the collection is made. --dims 1024 gives it instead.

In the console: Knowledge, then A.

Who may read and add

The owner, and everything below it, may read and add. A grant gives the same to another part of the same organization, to a person, or to one key:

llmgw ctl kb grant Acme/policies project:Acme/sales/crm           # read
llmgw ctl kb grant Acme/policies team:Acme/legal --write          # read and add
llmgw ctl kb revoke Acme/policies project:Acme/sales/crm

How access works:

• Grants add up down the tree. A grant to a team reaches its projects and their keys.
• Nothing crosses organizations. A grant to another organization's team, project, key or person is refused.
• A person's own key (a key with no project) reads nothing by itself. The person names the project they act in (llmgw-kb login asks for it, or --project) and must be a member of it. Joining a second team therefore does not quietly widen what every key of theirs can read.
• No access looks like no collection. A collection a key may not read is answered exactly as one that does not exist, so even its name does not leak.

Look after it

llmgw ctl kb list                          # collections, owners, documents, grants
llmgw ctl kb docs Acme/policies            # its documents, who added each and when
llmgw ctl kb rm Acme/policies old/form.pdf # one document out
llmgw ctl kb search Acme/policies how many days of annual leave
llmgw ctl kb delete Acme/policies          # the collection, its documents and grants

Rules that apply:

• A project, team or organization that owns a collection is not deleted until its collections are. This is the same rule as for one with members.
• A person's own collections go with their account (users delete, users forget).
• Every document added or removed is in the audit log (llmgw ctl audit), by name. The text is never recorded.

For everyone: llmgw-kb

Get it

Download llmgw-kb for your system from the release page, next to llmgw:

System	File
Windows (Intel/AMD)	llmgw-kb-VERSION-windows-amd64.exe
Windows (ARM)	llmgw-kb-VERSION-windows-arm64.exe
macOS (Apple silicon)	llmgw-kb-VERSION-darwin-arm64
macOS (Intel)	llmgw-kb-VERSION-darwin-amd64
Linux	llmgw-kb-VERSION-linux-amd64 / -arm64

Rename it to llmgw-kb (llmgw-kb.exe on Windows) and put it somewhere on your PATH. You do not need to install anything else.

On macOS, a downloaded program that is not signed is stopped the first time. Either right-click it and choose Open, or run xattr -d com.apple.quarantine llmgw-kb.

Log in once

llmgw-kb login

It asks for three things, checks them, and saves them where only you can read them:

• The gateway's address.
• Your API key. It is typed without being shown.
• A project. Needed only for a personal key.

Instead of logging in you can set LLMGW_URL, LLMGW_API_KEY and LLMGW_PROJECT.

llmgw-kb collections        # what your key may read, and which you may add to

Send your documents

llmgw-kb sync policies ~/Documents/HR
llmgw-kb sync policies "C:\Users\ayse\Documents\İK" --include "*.pdf,*.docx"
llmgw-kb sync policies ./handbook ./forms --exclude "drafts/**"

Run it as often as you like. The first run sends everything. Later runs send only what changed, and do not even read files whose size and time have not changed.

Option	What it does
--dry-run	Reads and cuts everything, sends nothing, and lists what would leave the machine and how much.
--delete	Also removes documents whose files are gone. Only those this machine sent, from these folders; never anything someone else sent.
--retry	Reads again files skipped before, for example after installing LibreOffice.
--include, --exclude	Glob patterns. *.pdf matches a file name; reports/**/*.xlsx matches a path.
--hidden	Reads hidden files and folders too (left out by default).
-v	Lists unchanged files too.

Documents are named by their path under the folder, with forward slashes, on every system. With several folders, each folder's own name comes first.

llmgw-kb status policies    # what this machine sent and skipped, and why
llmgw-kb history            # the latest runs
llmgw-kb formats            # what can be read here

What it reads

Read here	Formats
Documents	PDF (text), Word (.docx), Excel (.xlsx), PowerPoint (.pptx, with speaker notes), OpenDocument (.odt, .ods, .odp), RTF, EPUB
Web and mail	HTML, MHTML, e-mail (.eml, with its attachments)
Text	Markdown, plain text, CSV, JSON, YAML, XML, subtitles, notebooks (.ipynb), source code
Archives	ZIP, with every file inside it read in turn

Read with a helper on your machine	Helper
Word, Excel and PowerPoint 97–2003 (.doc, .xls, .ppt), Pages, Numbers, Keynote and others	LibreOffice
Images and scanned PDFs	Tesseract (with the Turkish language data), plus Poppler's pdftoppm for PDFs
PDFs the built-in reader cannot parse	Poppler's pdftotext

llmgw-kb formats says which helpers it found. Text in UTF-8, UTF-16, or the Windows Turkish and Western code pages is read correctly.

A file that cannot be read is skipped with the reason, and the run goes on. Reasons include a locked Office file, a format with no helper, and a scanned page with no OCR.

What leaves your machine

For each new or changed document, two things leave the machine, both over the gateway's TLS connection:

• The passages, to be embedded. They go through /v1/embeddings like any request of yours. Your key's limits, budget and guardrails apply.
• The document itself. Its name, its file's SHA-256 digest and size, and the passages with their vectors.

The file itself is never sent. --dry-run shows exactly what would be.

The ledger

llmgw-kb keeps a small SQLite database of what it did with every document: the file it came from, its size, time and digest, and whether it was sent, skipped or failed, with the reason.

System	Where it is
Windows	%AppData%\llmgw-kb\ledger.db
macOS	~/Library/Application Support/llmgw-kb/ledger.db
Linux	~/.config/llmgw-kb/ledger.db

Deleting the ledger is safe: the next run hashes every file once, finds what the collection already has, and sends nothing that is unchanged. --ledger uses another file.

How it is kept apart

Passages live in PostgreSQL, partitioned by organization, with an index per collection. Every search and every write names the organization and the collections the caller may use. On top of that, row-level security on the table checks the same thing again, so a query that forgot its filter returns nothing rather than another tenant's passages.