llmgw/docs/knowledge.md

Knowledge bases

A knowledge base is a collection of a company's documents that the cluster can answer from. People put their documents into it from their own machines with llmgw-kb. The tool reads the files on that machine, cuts them into passages ("chunks"), has the gateway embed each passage with the person's own key, and sends the text and the vectors. The files themselves never leave the machine.

Why retrieval, and not training

Retrieval is the tool for knowledge: policies, procedures, contracts. The answer can name its source, a changed document is sent again in minutes, and a passage only ever reaches people allowed to read its collection.

Training a model on the documents (LoRA) changes how it writes, not what it reliably knows. It has no access control: whatever goes into the weights is available to everyone who may use them.

What it needs

• The PostgreSQL store with pgvector. On a SQLite store knowledge bases are off, and llmgw ctl kb list and the console say why.

  • llmgw server init and the high-availability database setup install pgvector and enable it.
  • A cluster set up before this build needs it by hand, once:
    1. Install the package on every database machine: sudo apt-get install postgresql-16-pgvector (use the server's version number).
    2. On the leader, enable it in llmgw's database: sudo -u postgres psql -d llmgw -c 'CREATE EXTENSION vector'.
    3. Restart the control planes: llmgw restart.

• An embedding model in the catalog, with capabilities.embeddings. Two good choices handle Turkish and run on CPUs: BAAI/bge-m3 (1024 dimensions) and Qwen/Qwen3-Embedding-0.6B.

  Add it as any model is added (Serve a model in the console, or the catalog), with its entry saying that it embeds:

  capabilities:
    embeddings: true
  

  A collection stays with the model it was made with: vectors from two models cannot be compared.

  Some embedding models are trained with an instruction before the question and none before the documents. Give it as the entry's query_prefix; llmgw puts it before questions only. For Qwen3-Embedding:

  query_prefix: "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "
  

  With pgvector 0.7 or later a collection's index is kept at half precision (halfvec), which halves its memory and takes models of up to 4000 dimensions; with an older pgvector the limit is 2000. A collection keeps the kind it was made with.

For administrators

Make a collection

A collection belongs to one part of the tenancy tree: an organization, a team, a project or a person. Whoever is in the owner, at any level below it, may read it and add to it.

llmgw ctl kb create team:Acme/hr policies bge-m3
llmgw ctl kb create user:USER-ID notes bge-m3 --org Acme

The model's dimension is asked of the model itself, so the model has to be able to serve when the collection is made. --dims 1024 gives it instead.

In the console: Knowledge, then A.

Who may read and add

The owner, and everything below it, may read and add. A grant gives the same to another part of the same organization, to a person, or to one key:

llmgw ctl kb grant Acme/policies project:Acme/sales/crm           # read
llmgw ctl kb grant Acme/policies team:Acme/legal --write          # read and add
llmgw ctl kb revoke Acme/policies project:Acme/sales/crm

How access works:

• Grants add up down the tree. A grant to a team reaches its projects and their keys.
• Nothing crosses organizations. A grant to another organization's team, project, key or person is refused.
• A person's own key (a key with no project) reads nothing by itself. The person names the project they act in (llmgw-kb login asks for it, or --project) and must be a member of it. Joining a second team therefore does not quietly widen what every key of theirs can read.
• No access looks like no collection. A collection a key may not read is answered exactly as one that does not exist, so even its name does not leak.

Answer from collections by default

Clients that cannot add a header still get answers from the documents when their part of the tree has a default:

llmgw ctl kb default team:Acme/hr Acme/policies Acme/forms
llmgw ctl kb default key:KEY-ID Acme/policies
llmgw ctl kb default team:Acme/hr --clear

• Which default applies. The narrowest one: a key's over its project's, a project's over its team's, a team's over its organization's.
• A request can override it. A request that names collections uses those instead, and X-Llmgw-Knowledge: none turns the default off for that request.
• A default never refuses a request. A collection the caller may not read is left out for them. If the search fails (the embedding model is down, say), the request is answered without passages and the failure is logged.
• Changes take effect within 30 seconds.

In the console: Knowledge, Enter on a collection, then Answer from it by default….

Look after it

llmgw ctl kb list                          # collections, owners, documents, grants
llmgw ctl kb docs Acme/policies            # its documents, who added each and when
llmgw ctl kb rm Acme/policies old/form.pdf # one document out
llmgw ctl kb search Acme/policies how many days of annual leave
llmgw ctl kb delete Acme/policies          # the collection, its documents and grants

Rules that apply:

• A project, team or organization that owns a collection is not deleted until its collections are. This is the same rule as for one with members.
• A person's own collections go with their account (users delete, users forget).
• Every document added or removed is in the audit log (llmgw ctl audit), by name. The text is never recorded.

A large import

A few thousand documents ask the embedding model for far more than it can answer at once. Give the key that sends them a band of its own, below the people's, that may wait for room for minutes rather than the 30 seconds the others wait:

llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, saturation_ceiling: 0.85},
  {name: bronze, priority: 100, saturation_ceiling: 0.3}, {name: bulk, priority: 50, saturation_ceiling: 0.3, queue_ttl: 5m}]'
llmgw keys create import --project Acme/docs --band bulk

The people's questions then go first, and the import waits behind them. A request that waits as long as its band allows is answered 504 with Retry-After, and llmgw-kb sends it again rather than counting the passage as failed. It gives up after 15 minutes.

For everyone: llmgw-kb

Get it

Download llmgw-kb for your system from the release page, next to llmgw:

System	File
Windows (Intel/AMD)	llmgw-kb-VERSION-windows-amd64.exe
Windows (ARM)	llmgw-kb-VERSION-windows-arm64.exe
macOS (Apple silicon)	llmgw-kb-VERSION-darwin-arm64
macOS (Intel)	llmgw-kb-VERSION-darwin-amd64
Linux	llmgw-kb-VERSION-linux-amd64 / -arm64

Rename it to llmgw-kb (llmgw-kb.exe on Windows) and put it somewhere on your PATH. You do not need to install anything else.

On macOS, a downloaded program that is not signed is stopped the first time. Either right-click it and choose Open, or run xattr -d com.apple.quarantine llmgw-kb.

Log in once

llmgw-kb login

It asks for three things, checks them, and saves them where only you can read them:

• The gateway's address.
• Your API key. It is typed without being shown.
• A project. Needed only for a personal key.

Instead of logging in you can set LLMGW_URL, LLMGW_API_KEY and LLMGW_PROJECT.

llmgw-kb collections        # what your key may read, and which you may add to

Send your documents

llmgw-kb sync policies ~/Documents/HR
llmgw-kb sync policies "C:\Users\ayse\Documents\İK" --include "*.pdf,*.docx"
llmgw-kb sync policies ./handbook ./forms --exclude "drafts/**"

Run it as often as you like. The first run sends everything. Later runs send only what changed, and do not even read files whose size and time have not changed.

Option	What it does
--dry-run	Reads and cuts everything, sends nothing, and lists what would leave the machine and how much.
--delete	Also removes documents whose files are gone. Only those this machine sent, from these folders; never anything someone else sent.
--retry	Reads again files skipped before, for example after installing LibreOffice.
--include, --exclude	Glob patterns. *.pdf matches a file name; reports/**/*.xlsx matches a path.
--hidden	Reads hidden files and folders too (left out by default).
-v	Lists unchanged files too.

Documents are named by their path under the folder, with forward slashes, on every system. With several folders, each folder's own name comes first.

llmgw-kb status policies    # what this machine sent and skipped, and why
llmgw-kb history            # the latest runs
llmgw-kb formats            # what can be read here

What it reads

Read here	Formats
Documents	PDF (text), Word (.docx), Excel (.xlsx), PowerPoint (.pptx, with speaker notes), OpenDocument (.odt, .ods, .odp), RTF, EPUB
Web and mail	HTML, MHTML, e-mail (.eml, with its attachments)
Text	Markdown, plain text, CSV, JSON, YAML, XML, subtitles, notebooks (.ipynb), source code
Archives	ZIP, with every file inside it read in turn

Read with a helper on your machine	Helper
Word, Excel and PowerPoint 97–2003 (.doc, .xls, .ppt), Pages, Numbers, Keynote and others	LibreOffice
Images and scanned PDFs	Tesseract (with the Turkish language data), plus Poppler's pdftoppm for PDFs
PDFs the built-in reader cannot parse	Poppler's pdftotext

llmgw-kb formats says which helpers it found. Text in UTF-8, UTF-16, or the Windows Turkish and Western code pages is read correctly.

A file that cannot be read is skipped with the reason, and the run goes on. Reasons include a locked Office file, a format with no helper, and a scanned page with no OCR.

What leaves your machine

For each new or changed document, two things leave the machine, both over the gateway's TLS connection:

• The passages, to be embedded. They go through /v1/embeddings like any request of yours. Your key's limits, budget and guardrails apply.
• The document itself. Its name, its file's SHA-256 digest and size, and the passages with their vectors.

The file itself is never sent. --dry-run shows exactly what would be.

The ledger

llmgw-kb keeps a small SQLite database of what it did with every document: the file it came from, its size, time and digest, and whether it was sent, skipped or failed, with the reason.

System	Where it is
Windows	%AppData%\llmgw-kb\ledger.db
macOS	~/Library/Application Support/llmgw-kb/ledger.db
Linux	~/.config/llmgw-kb/ledger.db

Deleting the ledger is safe: the next run hashes every file once, finds what the collection already has, and sends nothing that is unchanged. --ledger uses another file.

Asking questions of a collection

A chat request names the collections to answer from in a header. Any client that can add a header can do it: the OpenAI and Anthropic SDKs, Continue, or curl.

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer $LLMGW_API_KEY" \
  -H "X-Llmgw-Knowledge: politikalar" \
  -d '{"model": "qwen2.5-coder-14b-instruct",
       "messages": [{"role": "user", "content": "Kaç gün yıllık iznim var?"}]}'

client.chat.completions.create(
    model="qwen2.5-coder-14b-instruct",
    messages=[{"role": "user", "content": "Kaç gün yıllık iznim var?"}],
    extra_headers={"X-Llmgw-Knowledge": "politikalar"},
)

What the gateway does with such a request:

1. It finds the collections. It looks up the ones named, by name or ID, among those the key may read. A collection it may not read is answered 404, the same as one that does not exist.
2. It embeds the question. The last user message is embedded by the collection's model, on the caller's own account: their limits, budget and guardrails apply.
3. It searches. The collections are searched by meaning and by words (below).
4. It adds the passages. The best passages are numbered and added to the system message, with an instruction to answer from them, cite them as [1], [2], and say so when they do not hold the answer. The system message is created if the request has none. Passages that would not fit the model's context are left out, the weakest first.
5. It judges and sends the request. The request, passages included, then goes through the guardrails and to the model as usual. A document carrying instructions aimed at its reader is judged like anything else the model reads.

On the portal's chat page, Documents picks a collection the person may read in the chosen project, the default, or none. Under each answer, the sources it was given are listed by number.

How it is shaped:

X-Llmgw-Knowledge	Collections, comma separated; none for no passages, whatever the default.
X-Llmgw-Knowledge-K	How many passages, 1 to 20; 5 by default.
X-Llmgw-Project	For a personal key: the project it acts in.
x-llmgw-sources (answer)	The passages given, as document#chunk, in their numbered order; empty when nothing matched.

Behaviour to know about:

• No passages matched. The model is told that nothing in the documents answers the question, and asked to say so rather than answer from general knowledge.
• Other APIs. Requests through /v1/responses and /v1/messages (Codex, Claude Code) work the same way, since they are answered as chat completions.
• Separate prefix caches. On vLLM replicas, a request given passages carries a cache_salt derived from the collections they came from. Requests that could be given the same passages still share cached blocks, and no one else's do: not another organization's, nor another team's of the same one. How fast an answer starts therefore says nothing about what someone else asked.
• No answer cache. Answers from knowledge bases are never served from the answer cache, because two keys of one project can read different collections.

How a passage is chosen

Each passage scores the following, and the highest scores win:

• its similarity in meaning to the question (cosine, 0 to 1), plus
• up to 0.3 for the share of the question's words it contains.

Words almost every passage has ("ne", "için", "the", "of") are not counted. The meaning does most of the work: "how much leave do fathers get" finds the paternity page, not the annual-leave page that shares more of its words. The word match helps with what meaning cannot carry, such as a form number, a product code or a name.

Metrics: llmgw_kb_retrievals_total{result} (found, nothing, refused) and llmgw_kb_retrieval_seconds.

Training an adapter on a collection

An adapter (LoRA) changes how a model writes: the organisation's words, format and tone. It does not reliably teach facts. On the PoC, a small model trained on nine policy documents answered 4 of 9 rephrased questions right and mixed facts up on the rest, while the same questions through the knowledge base got 9 of 9. Use both: the knowledge base for what answers say, an adapter for how they sound.

# 1. Questions and answers from the collection, written by a model of the
#    cluster. Read the file through and correct it.
llmgw ctl kb dataset Acme/policies --model qwen2.5-coder-14b-instruct --out policies.jsonl

# 2. Train on a node. --gpu-memory is what the job needs of a card; it
#    runs beside an engine when the engine leaves that much free.
llmgw ctl train start policies-tone --model qwen2.5-0.5b-instruct \
  --base-repo Qwen/Qwen2.5-0.5B-Instruct --node cuda01 --gpu-memory 2300 --data policies.jsonl
llmgw ctl train show JOB --follow

# 3. Make it the model's adapter, kept on the mirror.
llmgw ctl train publish JOB --restart

The node trains with its installed vLLM's torch and a small overlay (peft and accelerate), as the engines' user. For a model llama.cpp serves, it also converts the adapter to GGUF with the installed llama.cpp's own code. What it makes is kept by the control plane (llmgw ctl train download).

An adapter is private like its collection:

• It has an owner. An adapter trained on a collection carries some of what the collection says in its weights, so it belongs to a part of the tree.
  • By default the owner is the collection's owner.
  • --owner names another, which must be able to read every collection the pairs came from.
  • Only callers in the owner's part of the tree see the adapter. To anyone else it is a model that does not exist, absent from /v1/models too.
• It is public only by choice. --public makes an adapter every caller's. It is refused for pairs from a knowledge base, so it is only for pairs written by hand.

How it is kept apart

Passages live in PostgreSQL, partitioned by organization, with an index per collection. Every search and every write names the organization and the collections the caller may use. On top of that, row-level security on the table checks the same thing again, so a query that forgot its filter returns nothing rather than another tenant's passages.