llmgw/docs/knowledge.md

# Knowledge bases

A knowledge base is a collection of a company's documents that the cluster
can answer from. People put their documents into it from their own machines
with `llmgw-kb`. The tool reads the files on that machine, cuts them into
passages ("chunks"), has the gateway embed each passage with the person's own
key, and sends the text and the vectors. The files themselves never leave
the machine.

## Why retrieval, and not training

Retrieval is the tool for *knowledge*: policies, procedures, contracts. The
answer can name its source, a changed document is sent again in minutes, and
a passage only ever reaches people allowed to read its collection.

Training a model on the documents (LoRA) changes how it *writes*, not what it
reliably *knows*. It has no access control: whatever goes into the weights
is available to everyone who may use them.

## What it needs

- **The PostgreSQL store with pgvector.** On a SQLite store knowledge bases
  are off, and `llmgw ctl kb list` and the console say why.
  - `llmgw server init` and the high-availability database setup install
    pgvector and enable it.
  - A cluster set up before this build needs it by hand, once:
    1. Install the package on every database machine:
       `sudo apt-get install postgresql-16-pgvector` (use the server's
       version number).
    2. On the leader, enable it in llmgw's database:
       `sudo -u postgres psql -d llmgw -c 'CREATE EXTENSION vector'`.
    3. Restart the control planes: `llmgw restart`.
- **An embedding model in the catalog**, with `capabilities.embeddings`.
  Two good choices handle Turkish and run on CPUs: `BAAI/bge-m3` (1024
  dimensions) and `Qwen/Qwen3-Embedding-0.6B`.

  Add it as any model is added (**Serve a model** in the console, or the
  catalog), with its entry saying that it embeds:

  ```yaml
  capabilities:
    embeddings: true
  ```

  A collection stays with the model it was made with: vectors from two
  models cannot be compared.

## For administrators

### Make a collection

A collection belongs to one part of the tenancy tree: an organization, a
team, a project or a person. Whoever is in the owner, at any level below it,
may read it and add to it.

```sh
llmgw ctl kb create team:Acme/hr policies bge-m3
llmgw ctl kb create user:USER-ID notes bge-m3 --org Acme
```

The model's dimension is asked of the model itself, so the model has to be
able to serve when the collection is made. `--dims 1024` gives it instead.

In the console: **Knowledge**, then **A**.

### Who may read and add

The owner, and everything below it, may read and add. A **grant** gives the
same to another part of the same organization, to a person, or to one key:

```sh
llmgw ctl kb grant Acme/policies project:Acme/sales/crm           # read
llmgw ctl kb grant Acme/policies team:Acme/legal --write          # read and add
llmgw ctl kb revoke Acme/policies project:Acme/sales/crm
```

How access works:

- **Grants add up down the tree.** A grant to a team reaches its projects
  and their keys.
- **Nothing crosses organizations.** A grant to another organization's
  team, project, key or person is refused.
- **A person's own key** (a key with no project) reads nothing by itself.
  The person names the project they act in (`llmgw-kb login` asks for it,
  or `--project`) and must be a member of it. Joining a second team
  therefore does not quietly widen what every key of theirs can read.
- **No access looks like no collection.** A collection a key may not read
  is answered exactly as one that does not exist, so even its name does not
  leak.

### Look after it

```sh
llmgw ctl kb list                          # collections, owners, documents, grants
llmgw ctl kb docs Acme/policies            # its documents, who added each and when
llmgw ctl kb rm Acme/policies old/form.pdf # one document out
llmgw ctl kb search Acme/policies how many days of annual leave
llmgw ctl kb delete Acme/policies          # the collection, its documents and grants
```

Rules that apply:

- A project, team or organization that owns a collection is not deleted
  until its collections are. This is the same rule as for one with members.
- A person's own collections go with their account (`users delete`,
  `users forget`).
- Every document added or removed is in the audit log (`llmgw ctl audit`),
  by name. The text is never recorded.

## For everyone: `llmgw-kb`

### Get it

Download `llmgw-kb` for your system from the release page, next to `llmgw`:

| System | File |
|---|---|
| Windows (Intel/AMD) | `llmgw-kb-VERSION-windows-amd64.exe` |
| Windows (ARM) | `llmgw-kb-VERSION-windows-arm64.exe` |
| macOS (Apple silicon) | `llmgw-kb-VERSION-darwin-arm64` |
| macOS (Intel) | `llmgw-kb-VERSION-darwin-amd64` |
| Linux | `llmgw-kb-VERSION-linux-amd64` / `-arm64` |

Rename it to `llmgw-kb` (`llmgw-kb.exe` on Windows) and put it somewhere on
your PATH. You do not need to install anything else.

On macOS, a downloaded program that is not signed is stopped the first time.
Either right-click it and choose Open, or run
`xattr -d com.apple.quarantine llmgw-kb`.

### Log in once

```sh
llmgw-kb login
```

It asks for three things, checks them, and saves them where only you can
read them:

- **The gateway's address.**
- **Your API key.** It is typed without being shown.
- **A project.** Needed only for a personal key.

Instead of logging in you can set `LLMGW_URL`, `LLMGW_API_KEY` and
`LLMGW_PROJECT`.

```sh
llmgw-kb collections        # what your key may read, and which you may add to
```

### Send your documents

```sh
llmgw-kb sync policies ~/Documents/HR
llmgw-kb sync policies "C:\Users\ayse\Documents\İK" --include "*.pdf,*.docx"
llmgw-kb sync policies ./handbook ./forms --exclude "drafts/**"
```

Run it as often as you like. The first run sends everything. Later runs
send only what changed, and do not even read files whose size and time have
not changed.

| Option | What it does |
|---|---|
| `--dry-run` | Reads and cuts everything, sends nothing, and lists what would leave the machine and how much. |
| `--delete` | Also removes documents whose files are gone. Only those this machine sent, from these folders; never anything someone else sent. |
| `--retry` | Reads again files skipped before, for example after installing LibreOffice. |
| `--include`, `--exclude` | Glob patterns. `*.pdf` matches a file name; `reports/**/*.xlsx` matches a path. |
| `--hidden` | Reads hidden files and folders too (left out by default). |
| `-v` | Lists unchanged files too. |

Documents are named by their path under the folder, with forward slashes,
on every system. With several folders, each folder's own name comes first.

```sh
llmgw-kb status policies    # what this machine sent and skipped, and why
llmgw-kb history            # the latest runs
llmgw-kb formats            # what can be read here
```

### What it reads

| Read here | Formats |
|---|---|
| Documents | PDF (text), Word (`.docx`), Excel (`.xlsx`), PowerPoint (`.pptx`, with speaker notes), OpenDocument (`.odt`, `.ods`, `.odp`), RTF, EPUB |
| Web and mail | HTML, MHTML, e-mail (`.eml`, with its attachments) |
| Text | Markdown, plain text, CSV, JSON, YAML, XML, subtitles, notebooks (`.ipynb`), source code |
| Archives | ZIP, with every file inside it read in turn |

| Read with a helper on your machine | Helper |
|---|---|
| Word, Excel and PowerPoint 97–2003 (`.doc`, `.xls`, `.ppt`), Pages, Numbers, Keynote and others | [LibreOffice](https://www.libreoffice.org) |
| Images and scanned PDFs | [Tesseract](https://tesseract-ocr.github.io) (with the Turkish language data), plus Poppler's `pdftoppm` for PDFs |
| PDFs the built-in reader cannot parse | Poppler's `pdftotext` |

`llmgw-kb formats` says which helpers it found. Text in UTF-8, UTF-16, or
the Windows Turkish and Western code pages is read correctly.

A file that cannot be read is skipped with the reason, and the run goes on.
Reasons include a locked Office file, a format with no helper, and a
scanned page with no OCR.

### What leaves your machine

For each new or changed document, two things leave the machine, both over
the gateway's TLS connection:

- **The passages, to be embedded.** They go through `/v1/embeddings` like
  any request of yours. Your key's limits, budget and guardrails apply.
- **The document itself.** Its name, its file's SHA-256 digest and size,
  and the passages with their vectors.

The file itself is never sent. `--dry-run` shows exactly what would be.

### The ledger

`llmgw-kb` keeps a small SQLite database of what it did with every
document: the file it came from, its size, time and digest, and whether it
was sent, skipped or failed, with the reason.

| System | Where it is |
|---|---|
| Windows | `%AppData%\llmgw-kb\ledger.db` |
| macOS | `~/Library/Application Support/llmgw-kb/ledger.db` |
| Linux | `~/.config/llmgw-kb/ledger.db` |

Deleting the ledger is safe: the next run hashes every file once, finds what
the collection already has, and sends nothing that is unchanged. `--ledger`
uses another file.

## Asking questions of a collection

A chat request names the collections to answer from in a header. Any client
that can add a header can do it: the OpenAI and Anthropic SDKs, Continue, or
curl.

```sh
curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "X-Llmgw-Knowledge: politikalar" \
  -d '{"model": "qwen2.5-coder-14b-instruct",
       "messages": [{"role": "user", "content": "Kaç gün yıllık iznim var?"}]}'
```

```python
client.chat.completions.create(
    model="qwen2.5-coder-14b-instruct",
    messages=[{"role": "user", "content": "Kaç gün yıllık iznim var?"}],
    extra_headers={"X-Llmgw-Knowledge": "politikalar"},
)
```

What the gateway does with such a request:

1. **It finds the collections.** It looks up the ones named, by name or ID,
   among those the key may read. A collection it may not read is answered
   404, the same as one that does not exist.
2. **It embeds the question.** The last user message is embedded by the
   collection's model, on the caller's own account: their limits, budget and
   guardrails apply.
3. **It searches.** The collections are searched by meaning and by words
   (below).
4. **It adds the passages.** The best passages are numbered and added to the
   system message, with an instruction to answer from them, cite them as
   [1], [2], and say so when they do not hold the answer. The system message
   is created if the request has none. Passages that would not fit the
   model's context are left out, the weakest first.
5. **It judges and sends the request.** The request, passages included, then
   goes through the guardrails and to the model as usual. A document
   carrying instructions aimed at its reader is judged like anything else
   the model reads.

How it is shaped:

| | |
|---|---|
| `X-Llmgw-Knowledge` | Collections, comma separated. |
| `X-Llmgw-Knowledge-K` | How many passages, 1 to 20; 5 by default. |
| `X-Llmgw-Project` | For a personal key: the project it acts in. |
| `x-llmgw-sources` (answer) | The passages given, as `document#chunk`, in their numbered order; empty when nothing matched. |

Behaviour to know about:

- **No passages matched.** The model is told that nothing in the documents
  answers the question, and asked to say so rather than answer from general
  knowledge.
- **Other APIs.** Requests through `/v1/responses` and `/v1/messages`
  (Codex, Claude Code) work the same way, since they are answered as chat
  completions.
- **No answer cache.** Answers from knowledge bases are never served from
  the answer cache, because two keys of one project can read different
  collections.

### How a passage is chosen

Each passage scores the following, and the highest scores win:

- its similarity in meaning to the question (cosine, 0 to 1), plus
- up to 0.3 for the share of the question's words it contains.

Words almost every passage has ("ne", "için", "the", "of") are not counted.
The meaning does most of the work: "how much leave do fathers get" finds the
paternity page, not the annual-leave page that shares more of its words.
The word match helps with what meaning cannot carry, such as a form number,
a product code or a name.

Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
`llmgw_kb_retrieval_seconds`.

## How it is kept apart

Passages live in PostgreSQL, partitioned by organization, with an index per
collection. Every search and every write names the organization and the
collections the caller may use. On top of that, row-level security on the
table checks the same thing again, so a query that forgot its filter
returns nothing rather than another tenant's passages.