Chat requests answered from knowledge bases

A request naming collections in X-Llmgw-Knowledge has its last question
embedded on the caller's account, the best passages found and added to the
system message, numbered, with an instruction to cite them and to say when
they do not hold the answer; x-llmgw-sources says which were given. Done
before the guardrails judge the request, never served from the answer cache,
and a collection the key may not read is answered as one that does not
exist. Metrics llmgw_kb_retrievals_total and llmgw_kb_retrieval_seconds.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 15:27:50 +02:00
1 parent 65116064d2
commit 496625824b
10 files changed
+653 -28

No files matched your search

+79 -6
View File
@@ -7,12 +7,6 @@ passages ("chunks"), has the gateway embed each passage with the person's own
key, and sends the text and the vectors. The files themselves never leave
the machine.
> **Status.** Collections, access and filling them work now. Answering
> requests from them (the gateway finding the passages a question needs and
> putting them in front of the model) is the next step and is not in this
> build yet. Until then a collection can be searched by an administrator
> (`llmgw ctl kb search`) to check what it holds.
## Why retrieval, and not training
Retrieval is the tool for *knowledge*: policies, procedures, contracts. The
@@ -234,6 +228,85 @@ Deleting the ledger is safe: the next run hashes every file once, finds what
the collection already has, and sends nothing that is unchanged. `--ledger`
uses another file.
## Asking questions of a collection
A chat request names the collections to answer from in a header. Any client
that can add a header can do it: the OpenAI and Anthropic SDKs, Continue, or
curl.
```sh
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "X-Llmgw-Knowledge: politikalar" \
-d '{"model": "qwen2.5-coder-14b-instruct",
"messages": [{"role": "user", "content": "Kaç gün yıllık iznim var?"}]}'
```
```python
client.chat.completions.create(
model="qwen2.5-coder-14b-instruct",
messages=[{"role": "user", "content": "Kaç gün yıllık iznim var?"}],
extra_headers={"X-Llmgw-Knowledge": "politikalar"},
)
```
What the gateway does with such a request:
1. **It finds the collections.** It looks up the ones named, by name or ID,
among those the key may read. A collection it may not read is answered
404, the same as one that does not exist.
2. **It embeds the question.** The last user message is embedded by the
collection's model, on the caller's own account: their limits, budget and
guardrails apply.
3. **It searches.** The collections are searched by meaning and by words
(below).
4. **It adds the passages.** The best passages are numbered and added to the
system message, with an instruction to answer from them, cite them as
[1], [2], and say so when they do not hold the answer. The system message
is created if the request has none. Passages that would not fit the
model's context are left out, the weakest first.
5. **It judges and sends the request.** The request, passages included, then
goes through the guardrails and to the model as usual. A document
carrying instructions aimed at its reader is judged like anything else
the model reads.
How it is shaped:
| | |
|---|---|
| `X-Llmgw-Knowledge` | Collections, comma separated. |
| `X-Llmgw-Knowledge-K` | How many passages, 1 to 20; 5 by default. |
| `X-Llmgw-Project` | For a personal key: the project it acts in. |
| `x-llmgw-sources` (answer) | The passages given, as `document#chunk`, in their numbered order; empty when nothing matched. |
Behaviour to know about:
- **No passages matched.** The model is told that nothing in the documents
answers the question, and asked to say so rather than answer from general
knowledge.
- **Other APIs.** Requests through `/v1/responses` and `/v1/messages`
(Codex, Claude Code) work the same way, since they are answered as chat
completions.
- **No answer cache.** Answers from knowledge bases are never served from
the answer cache, because two keys of one project can read different
collections.
### How a passage is chosen
Each passage scores the following, and the highest scores win:
- its similarity in meaning to the question (cosine, 0 to 1), plus
- up to 0.3 for the share of the question's words it contains.
Words almost every passage has ("ne", "için", "the", "of") are not counted.
The meaning does most of the work: "how much leave do fathers get" finds the
paternity page, not the annual-leave page that shares more of its words.
The word match helps with what meaning cannot carry, such as a form number,
a product code or a name.
Metrics: `llmgw_kb_retrievals_total{result}` (found, nothing, refused) and
`llmgw_kb_retrieval_seconds`.
## How it is kept apart
Passages live in PostgreSQL, partitioned by organization, with an index per