Text from documents of many formats, and chunks of it, for knowledge bases

PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML, MHTML, mail with
its attachments, notebooks, subtitles, zip archives and text and source files
in any common encoding, in pure Go; legacy Office formats through LibreOffice
and scanned pages through Tesseract when they are installed. Chunks of about
500 tokens under their headings, overlapping by a paragraph.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 13:24:15 +02:00
1 parent d35630defe
commit a0cfafe772
16 files changed
+2844 -10

No files matched your search

+45 -10
View File
@@ -865,16 +865,51 @@ pgha on a running cluster — install by hand or add an upgrade step;
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).
### F2. Ingest (M)
- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add
<collection> <files…>`, admin API under `/api/v1/kb`, audit events.
- Text from PDF (text layer), DOCX, HTML, Markdown, plain text — in Go, no
Python. Split on headings/paragraphs, ~500 tokens with overlap. Embedded
through the router at the bronze band, accounted to the collection's
owner. One transaction per document; same digest skipped, changed digest
replaces. OCR out of scope.
- Console: a Knowledge screen (collections, documents, grants; add, grant,
revoke, delete); `docs/console.md`, `docs/knowledge.md`.
### F2. Ingest on the user's machine (L) — redesigned 2026-10-08
Uploading whole files to the control plane was dropped: a PDF is 20–100×
its text, and the work belongs where the files are. Instead a small client,
`llmgw-kb`, run by the person on their own machine with their own key:
- **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64
and arm64. `llmgw` itself does not build on Windows (agent syscalls), and
employees' laptops should not carry the server, agent and setup screens. A
test keeps server packages out of its imports.
- `llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs]
[--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256;
cache under os.UserCacheDir, digests not recomputed for unchanged files)
→ the collection's manifest from the gateway (name → digest only) → diff
→ for new/changed files only: extract, chunk, embed through the gateway's
`/v1/embeddings` with the person's key (its accounting, limits, guard and
band apply), and send name, digest, chunk texts and vectors; `--delete`
removes documents whose files are gone. `--dry-run` lists what would leave
the machine and how much. A file that cannot be read (locked, unsupported,
no text) is skipped with the reason; the run goes on.
- **Formats, as many as possible.** Built in, pure Go: PDF (text layer),
DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML
parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML,
Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives
recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF
and anything else LibreOffice opens (`soffice --headless --convert-to`),
images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8,
UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
- **Across systems:** document names relative to the synced root, `/`
separated, Unicode NFC (macOS hands out NFD), a warning for two names that
differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb
login` (os.UserConfigDir, 0600 where that means something); the system's
CA store, `--ca` for another.
- **Gateway endpoints** (client listener, person's key):
`GET /v1/kb/collections` (what this key may read/write),
`GET /v1/kb/collections/{id}/manifest`, `PUT
/v1/kb/collections/{id}/documents` (one document per request: name,
digest, bytes, chunks with text and embedding; dims checked, chunk count
and size bounded), `DELETE …/documents?name=`. Write needs a write grant
or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise
404.
- **Administration** stays with operators: `llmgw ctl kb
create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit,
console screen; `docs/knowledge.md`.
- Trade-off accepted: vectors come from the client, so a writer could store
vectors that do not match their text — only in collections they may
already write.
### F3. Retrieval on the request path (M)
- `llmgw.knowledge: [collections]` in the body (stripped before the