llmgw/docs/knowledge.md

# Knowledge bases

A knowledge base is a collection of a company's documents that the cluster
can answer from. People put their documents into it from their own machines
with `llmgw-kb`. The tool reads the files on that machine, cuts them into
passages ("chunks") and sends their text; the gateway embeds each passage
with the person's own key and keeps the text and the vectors. The files
themselves never leave the machine.

## Why retrieval, and not training

Retrieval is the tool for *knowledge*: policies, procedures, contracts. The
answer can name its source, a changed document is sent again in minutes, and
a passage only ever reaches people allowed to read its collection.

Training a model on the documents (LoRA) changes how it *writes*, not what it
reliably *knows*. It has no access control: whatever goes into the weights
is available to everyone who may use them.

## What it needs

- **The PostgreSQL store with pgvector.** On a SQLite store knowledge bases
  are off, and `llmgw ctl kb list` and the console say why.
  - `llmgw server init` and the high-availability database setup install
    pgvector and enable it.
  - A cluster set up before this build needs it by hand, once:
    1. Install the package on every database machine:
       `sudo apt-get install postgresql-16-pgvector` (use the server's
       version number).
    2. On the leader, enable it in llmgw's database:
       `sudo -u postgres psql -d llmgw -c 'CREATE EXTENSION vector'`.
    3. Restart the control planes: `llmgw restart`.
- **An embedding model in the catalog**, with `capabilities.embeddings`.
  Two good choices handle Turkish and run on CPUs: `BAAI/bge-m3` (1024
  dimensions) and `Qwen/Qwen3-Embedding-0.6B`.

  Add it as any model is added (**Serve a model** in the console, or the
  catalog), with its entry saying that it embeds:

  ```yaml
  capabilities:
    embeddings: true
  ```

  A collection stays with the model it was made with: vectors from two
  models cannot be compared.

  Some embedding models are trained with an instruction before the
  question and none before the documents. Give it as the entry's
  `query_prefix`; llmgw puts it before questions only. For
  Qwen3-Embedding:

  ```yaml
  query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
  ```

  With pgvector 0.7 or later a collection's index is kept at half
  precision (`halfvec`), which halves its memory and takes models of up to
  4000 dimensions; with an older pgvector the limit is 2000. A collection
  keeps the kind it was made with.

## For administrators

### Make a collection

A collection belongs to one part of the tenancy tree: an organization, a
team, a project or a person. Whoever is in the owner, at any level below it,
may read it and add to it.

```sh
llmgw ctl kb create team:Acme/hr policies bge-m3
llmgw ctl kb create user:USER-ID notes bge-m3 --org Acme
```

The model's dimension is asked of the model itself, so the model has to be
able to serve when the collection is made. `--dims 1024` gives it instead.

In the console: **Knowledge**, then **A**.

### Who may read and add

The owner, and everything below it, may read and add. A **grant** gives the
same to another part of the same organization, to a person, or to one key:

```sh
llmgw ctl kb grant Acme/policies project:Acme/sales/crm           # read
llmgw ctl kb grant Acme/policies team:Acme/legal --write          # read and add
llmgw ctl kb revoke Acme/policies project:Acme/sales/crm
```

How access works:

- **Grants add up down the tree.** A grant to a team reaches its projects
  and their keys.
- **Nothing crosses organizations.** A grant to another organization's
  team, project, key or person is refused.
- **A person's own key** (a key with no project) reads nothing by itself.
  The person names the project they act in (`llmgw-kb login` asks for it,
  or `--project`) and must be a member of it. Joining a second team
  therefore does not quietly widen what every key of theirs can read.
- **No access looks like no collection.** A collection a key may not read
  is answered exactly as one that does not exist, so even its name does not
  leak.

### Answer from collections by default

Clients that cannot add a header still get answers from the documents when
their part of the tree has a default:

```sh
llmgw ctl kb default team:Acme/hr Acme/policies Acme/forms
llmgw ctl kb default key:KEY-ID Acme/policies
llmgw ctl kb default team:Acme/hr --clear
```

- **Which default applies.** The narrowest one: a key's over its project's,
  a project's over its team's, a team's over its organization's.
- **A request can override it.** A request that names collections uses
  those instead, and `X-Llmgw-Knowledge: none` turns the default off for
  that request.
- **A default never refuses a request.** A collection the caller may not
  read is left out for them. If the search fails (the embedding model is
  down, say), the request is answered without passages and the failure is
  logged.
- **Changes take effect within 30 seconds.**

In the console: **Knowledge**, Enter on a collection, then **Answer from it
by default…**.

### Look after it

```sh
llmgw ctl kb list                          # collections, owners, documents, grants
llmgw ctl kb docs Acme/policies            # its documents, who added each and when
llmgw ctl kb rm Acme/policies old/form.pdf # one document out
llmgw ctl kb search Acme/policies how many days of annual leave
llmgw ctl kb delete Acme/policies          # the collection, its documents and grants
```

Rules that apply:

- A project, team or organization that owns a collection is not deleted
  until its collections are. This is the same rule as for one with members.
- A person's own collections go with their account (`users delete`,
  `users forget`).
- Every document added or removed is in the audit log (`llmgw ctl audit`),
  by name. The text is never recorded.

### A large import

A few thousand documents ask the embedding model for far more than it can
answer at once. Give the key that sends them a band of its own, below the
people's, that may wait for room for minutes rather than the 30 seconds the
others wait:

```sh
llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
  {name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
llmgw keys create import --project Acme/docs --band bulk
```

The people's questions then go first, and the import waits behind them. The
gateway embeds a document's passages 32 at a time, four batches at once,
enough to fill an embedding replica's slots; a batch told to wait (`Retry-After`)
waits up to five minutes before the document is answered the same, and
`llmgw-kb` sends it again rather than counting it as failed. It gives up
after 15 minutes.

## For everyone: `llmgw-kb`

### Get it

Download `llmgw-kb` for your system from the release page, next to `llmgw`:

| System | File |
|---|---|
| Windows (Intel/AMD) | `llmgw-kb-VERSION-windows-amd64.exe` |
| Windows (ARM) | `llmgw-kb-VERSION-windows-arm64.exe` |
| macOS (Apple silicon) | `llmgw-kb-VERSION-darwin-arm64` |
| macOS (Intel) | `llmgw-kb-VERSION-darwin-amd64` |
| Linux | `llmgw-kb-VERSION-linux-amd64` / `-arm64` |

Rename it to `llmgw-kb` (`llmgw-kb.exe` on Windows) and put it somewhere on
your PATH. You do not need to install anything else.

On macOS, a downloaded program that is not signed is stopped the first time.
Either right-click it and choose Open, or run
`xattr -d com.apple.quarantine llmgw-kb`.

### Log in once

```sh
llmgw-kb login
```

It asks for three things, checks them, and saves them where only you can
read them:

- **The gateway's address.**
- **Your API key.** It is typed without being shown.
- **A project.** Needed only for a personal key.

Instead of logging in you can set `LLMGW_URL`, `LLMGW_API_KEY` and
`LLMGW_PROJECT`.

```sh
llmgw-kb collections        # what your key may read, and which you may add to
```

### Send your documents

```sh
llmgw-kb sync policies ~/Documents/HR
llmgw-kb sync policies "C:\Users\ayse\Documents\İK" --include "*.pdf,*.docx"
llmgw-kb sync policies ./handbook ./forms --exclude "drafts/**"
```

Run it as often as you like. The first run sends everything. Later runs
send only what changed, and do not even read files whose size and time have
not changed.

| Option | What it does |
|---|---|
| `--dry-run` | Reads and cuts everything, sends nothing, and lists what would leave the machine and how much. |
| `--delete` | Also removes documents whose files are gone. Only those this machine sent, from these folders; never anything someone else sent. |
| `--retry` | Reads again files skipped before, for example after installing LibreOffice. |
| `--include`, `--exclude` | Glob patterns. `*.pdf` matches a file name; `reports/**/*.xlsx` matches a path. |
| `--hidden` | Reads hidden files and folders too (left out by default). |
| `-v` | Lists unchanged files too. |
| `--embed-here` | Embeds the passages from this machine and sends the vectors. With a gateway that does not embed them itself it does so anyway. |

Documents are named by their path under the folder, with forward slashes,
on every system. With several folders, each folder's own name comes first.

```sh
llmgw-kb status policies    # what this machine sent and skipped, and why
llmgw-kb history            # the latest runs
llmgw-kb formats            # what can be read here
```

### What it reads

| Read here | Formats |
|---|---|
| Documents | PDF (text), Word (`.docx`), Excel (`.xlsx`), PowerPoint (`.pptx`, with speaker notes), OpenDocument (`.odt`, `.ods`, `.odp`), RTF, EPUB |
| Web and mail | HTML, MHTML, e-mail (`.eml`, with its attachments) |
| Text | Markdown, plain text, CSV, JSON, YAML, XML, subtitles, notebooks (`.ipynb`), source code |
| Archives | ZIP, with every file inside it read in turn |

| Read with a helper on your machine | Helper |
|---|---|
| Word, Excel and PowerPoint 97–2003 (`.doc`, `.xls`, `.ppt`), Pages, Numbers, Keynote and others | [LibreOffice](https://www.libreoffice.org) |
| Images and scanned PDFs | [Tesseract](https://tesseract-ocr.github.io) (with the Turkish language data), plus Poppler's `pdftoppm` for PDFs |
| PDFs the built-in reader cannot parse | Poppler's `pdftotext` |

`llmgw-kb formats` says which helpers it found. Text in UTF-8, UTF-16, or
the Windows Turkish and Western code pages is read correctly.

A file that cannot be read is skipped with the reason, and the run goes on.
Reasons include a locked Office file, a format with no helper, and a
scanned page with no OCR.

### What leaves your machine

For each new or changed document, one request leaves the machine, over the
gateway's TLS connection: its name, its file's SHA-256 digest and size, and
its passages' text. The gateway embeds the passages through `/v1/embeddings`
as you, so your key's limits, budget and guardrails apply as if you had
asked. That is about a kilobyte a passage; embedding here (`--embed-here`)
brings each vector down and sends it up again, some thirty times more.

The file itself is never sent. `--dry-run` shows exactly what would be.

### The ledger

`llmgw-kb` keeps a small SQLite database of what it did with every
document: the file it came from, its size, time and digest, and whether it
was sent, skipped or failed, with the reason.

| System | Where it is |
|---|---|
| Windows | `%AppData%\llmgw-kb\ledger.db` |
| macOS | `~/Library/Application Support/llmgw-kb/ledger.db` |
| Linux | `~/.config/llmgw-kb/ledger.db` |

Deleting the ledger is safe: the next run hashes every file once, finds what
the collection already has, and sends nothing that is unchanged. `--ledger`
uses another file.

## Asking questions of a collection

A chat request names the collections to answer from in a header. Any client
that can add a header can do it: the OpenAI and Anthropic SDKs, Continue, or
curl.

```sh
curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "X-Llmgw-Knowledge: politikalar" \
  -d '{"model": "qwen2.5-coder-14b-instruct",
       "messages": [{"role": "user", "content": "Kaç gün yıllık iznim var?"}]}'
```

```python
client.chat.completions.create(
    model="qwen2.5-coder-14b-instruct",
    messages=[{"role": "user", "content": "Kaç gün yıllık iznim var?"}],
    extra_headers={"X-Llmgw-Knowledge": "politikalar"},
)
```

What the gateway does with such a request:

1. **It finds the collections.** It looks up the ones named, by name or ID,
   among those the key may read. A collection it may not read is answered
   404, the same as one that does not exist.
2. **It embeds the question.** The last user message is embedded by the
   collection's model, on the caller's own account: their limits, budget and
   guardrails apply.
3. **It searches.** The collections are searched by meaning and by words
   (below).
4. **It adds the passages.** The best passages are numbered and added to the
   system message, with an instruction to answer from them, cite them as
   [1], [2], and say so when they do not hold the answer. The system message
   is created if the request has none. Passages that would not fit the
   model's context are left out, the weakest first.
5. **It judges and sends the request.** The request, passages included, then
   goes through the guardrails and to the model as usual. A document
   carrying instructions aimed at its reader is judged like anything else
   the model reads.

On the portal's chat page, **Documents** picks a collection the person may
read in the chosen project, the default, or none. Under each answer, the
sources it was given are listed by number.

How it is shaped:

| | |
|---|---|
| `X-Llmgw-Knowledge` | Collections, comma separated; `none` for no passages, whatever the default. |
| `X-Llmgw-Knowledge-K` | How many passages, 1 to 20; 5 by default. |
| `X-Llmgw-Project` | For a personal key: the project it acts in. |
| `x-llmgw-sources` (answer) | The passages given, as `document#chunk`, in their numbered order; empty when nothing matched. |

Behaviour to know about:

- **No passages matched.** The model is told that nothing in the documents
  answers the question, and asked to say so rather than answer from general
  knowledge.
- **Other APIs.** Requests through `/v1/responses` and `/v1/messages`
  (Codex, Claude Code) work the same way, since they are answered as chat
  completions.
- **Separate prefix caches.** On vLLM replicas, a request given passages
  carries a `cache_salt` derived from the collections they came from.
  Requests that could be given the same passages still share cached blocks,
  and no one else's do: not another organization's, nor another team's of
  the same one. How fast an answer starts therefore says nothing about what
  someone else asked.
- **No answer cache.** Answers from knowledge bases are never served from
  the answer cache, because two keys of one project can read different
  collections.

### How a passage is chosen

Each passage scores the following, and the highest scores win:

- its similarity in meaning to the question (cosine, 0 to 1), plus
- up to 0.3 for the share of the question's words it contains.

Words almost every passage has ("ne", "için", "the", "of") are not counted.
The meaning does most of the work: "how much leave do fathers get" finds the
paternity page, not the annual-leave page that shares more of its words.
The word match helps with what meaning cannot carry, such as a form number,
a product code or a name.

Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
`llmgw_kb_retrieval_seconds`.

## Training an adapter on a collection

An adapter (LoRA) changes how a model *writes*: the organisation's words,
format and tone. It does not reliably teach facts. On the PoC, a small
model trained on nine policy documents answered 4 of 9 rephrased questions
right and mixed facts up on the rest, while the same questions through the
knowledge base got 9 of 9. Use both: the knowledge base for what answers
say, an adapter for how they sound.

```sh
# 1. Questions and answers from the collection, written by a model of the
#    cluster. Read the file through and correct it.
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl

# 2. Train on a node. --gpu-memory is what the job needs of a card; it
#    runs beside an engine when the engine leaves that much free.
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
  --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
llmgw ctl train show JOB --follow

# 3. Make it the model's adapter, kept on the mirror.
llmgw ctl train publish JOB --restart
```

The node trains with its installed vLLM's torch and a small overlay (peft
and accelerate), as the engines' user. For a model llama.cpp serves, it also
converts the adapter to GGUF with the installed llama.cpp's own code. What
it makes is kept by the control plane (`llmgw ctl train download`).

An adapter is private like its collection:

- **It has an owner.** An adapter trained on a collection carries some of
  what the collection says in its weights, so it belongs to a part of the
  tree.
  - By default the owner is the collection's owner.
  - `--owner` names another, which must be able to read every collection
    the pairs came from.
  - Only callers in the owner's part of the tree see the adapter. To anyone
    else it is a model that does not exist, absent from `/v1/models` too.
- **It is public only by choice.** `--public` makes an adapter every
  caller's. It is refused for pairs from a knowledge base, so it is only for
  pairs written by hand.

## How it is kept apart

Passages live in PostgreSQL, partitioned by organization, with an index per
collection. Every search and every write names the organization and the
collections the caller may use. On top of that, row-level security on the
table checks the same thing again, so a query that forgot its filter
returns nothing rather than another tenant's passages.