Text from documents of many formats, and chunks of it, for knowledge bases

PDF, Word, Excel, PowerPoint, OpenDocument, RTF, EPUB, HTML, MHTML, mail with
its attachments, notebooks, subtitles, zip archives and text and source files
in any common encoding, in pure Go; legacy Office formats through LibreOffice
and scanned pages through Tesseract when they are installed. Chunks of about
500 tokens under their headings, overlapping by a paragraph.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 13:24:15 +02:00
1 parent d35630defe
commit a0cfafe772
16 files changed
+2844 -10

No files matched your search

+45 -10
View File
@@ -865,16 +865,51 @@ pgha on a running cluster — install by hand or add an upgrade step;
and SQLite has none); tests need `LLMGW_TEST_POSTGRES` pointing at a
pgvector database as a non-superuser (a superuser bypasses RLS).
### F2. Ingest (M)
- `llmgw ctl kb create|list|delete|grant|revoke|docs`, `ctl kb add
<collection> <files…>`, admin API under `/api/v1/kb`, audit events.
- Text from PDF (text layer), DOCX, HTML, Markdown, plain text — in Go, no
Python. Split on headings/paragraphs, ~500 tokens with overlap. Embedded
through the router at the bronze band, accounted to the collection's
owner. One transaction per document; same digest skipped, changed digest
replaces. OCR out of scope.
- Console: a Knowledge screen (collections, documents, grants; add, grant,
revoke, delete); `docs/console.md`, `docs/knowledge.md`.
### F2. Ingest on the user's machine (L) — redesigned 2026-10-08
Uploading whole files to the control plane was dropped: a PDF is 20–100×
its text, and the work belongs where the files are. Instead a small client,
`llmgw-kb`, run by the person on their own machine with their own key:
- **Its own binary** (`cmd/llmgw-kb`), for Windows, macOS and Linux on amd64
and arm64. `llmgw` itself does not build on Windows (agent syscalls), and
employees' laptops should not carry the server, agent and setup screens. A
test keeps server packages out of its imports.
- `llmgw-kb sync <collection> <dir|file…> [--include/--exclude globs]
[--dry-run] [--delete]`: scan → local catalog (path, size, mtime, sha256;
cache under os.UserCacheDir, digests not recomputed for unchanged files)
→ the collection's manifest from the gateway (name → digest only) → diff
→ for new/changed files only: extract, chunk, embed through the gateway's
`/v1/embeddings` with the person's key (its accounting, limits, guard and
band apply), and send name, digest, chunk texts and vectors; `--delete`
removes documents whose files are gone. `--dry-run` lists what would leave
the machine and how much. A file that cannot be read (locked, unsupported,
no text) is skipped with the reason; the run goes on.
- **Formats, as many as possible.** Built in, pure Go: PDF (text layer),
DOCX/XLSX/PPTX, ODT/ODS/ODP, RTF, EPUB, HTML/XHTML, EML (text and HTML
parts), Jupyter notebooks, subtitles (SRT/VTT), CSV/TSV, JSON/YAML/XML,
Markdown/RST/AsciiDoc/Org, plain text and source code; ZIP archives
recursed into. With a helper found on the machine: legacy DOC/XLS/PPT, RTF
and anything else LibreOffice opens (`soffice --headless --convert-to`),
images and scanned PDFs through Tesseract (`tur+eng`). Text in UTF-8,
UTF-16 (BOM) or Windows-1254/1252 is converted to UTF-8.
- **Across systems:** document names relative to the synced root, `/`
separated, Unicode NFC (macOS hands out NFD), a warning for two names that
differ only in case; CRLF → LF; key from `LLMGW_API_KEY` or `llmgw-kb
login` (os.UserConfigDir, 0600 where that means something); the system's
CA store, `--ca` for another.
- **Gateway endpoints** (client listener, person's key):
`GET /v1/kb/collections` (what this key may read/write),
`GET /v1/kb/collections/{id}/manifest`, `PUT
/v1/kb/collections/{id}/documents` (one document per request: name,
digest, bytes, chunks with text and embedding; dims checked, chunk count
and size bounded), `DELETE …/documents?name=`. Write needs a write grant
or ownership on the key's chain (`CollectionsFor(…, write)`); otherwise
404.
- **Administration** stays with operators: `llmgw ctl kb
create|list|delete|grant|revoke|docs`, admin API `/api/v1/kb`, audit,
console screen; `docs/knowledge.md`.
- Trade-off accepted: vectors come from the client, so a writer could store
vectors that do not match their text — only in collections they may
already write.
### F3. Retrieval on the request path (M)
- `llmgw.knowledge: [collections]` in the body (stripped before the
+1
View File
@@ -18,6 +18,7 @@ require (
github.com/knadh/koanf/providers/env/v2 v2.0.1
github.com/knadh/koanf/providers/file v1.2.1
github.com/knadh/koanf/v2 v2.3.6
github.com/ledongthuc/pdf v0.0.0-20260907135840-6c8c28e0e8a0
github.com/mattn/go-isatty v0.0.24
github.com/prometheus/client_golang v1.24.1
github.com/prometheus/client_model v0.6.3
+2
View File
@@ -95,6 +95,8 @@ github.com/knadh/koanf/v2 v2.3.6 h1:JoQPSJmvS4aP0xNc8xMDr5tcrkSEInL23/Il7pITAKo=
github.com/knadh/koanf/v2 v2.3.6/go.mod h1:gRb40VRAbd4iJMYYD5IxZ6hfuopFcXBpc9bbQpZwo28=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/ledongthuc/pdf v0.0.0-20260907135840-6c8c28e0e8a0 h1:7Q+xNAZFmnfYOMweHN3c/PDFUKKfY1pVJ26K++QvVfU=
github.com/ledongthuc/pdf v0.0.0-20260907135840-6c8c28e0e8a0/go.mod h1:1fEHWurg7pvf5SG6XNE5Q8UZmOwex51Mkx3SLhrW5B4=
github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
+248
View File
@@ -0,0 +1,248 @@
// Package chunk splits extracted text into the pieces a knowledge base keeps
// and searches (BACKLOG F2).
//
// A chunk is a few paragraphs under the headings they sit beneath: about 500
// tokens, which an embedding model reads whole and a prompt can carry several
// of. Paragraphs are not cut while they fit; a section starts a new chunk
// when the current one is already a fair size, so a chunk rarely mixes two
// topics; and the last paragraph of a chunk is repeated at the start of the
// next, so an answer that spans the boundary is found whole in one of them.
package chunk
import (
"strings"
"unicode"
"unicode/utf8"
)
// Options size the chunks. Zero values take the defaults.
type Options struct {
// MaxChars is the most characters in a chunk's body: 1800, about 500
// tokens of Turkish or English.
MaxChars int
// Overlap is the most characters of the previous chunk repeated at the
// start of the next: 300.
Overlap int
}
func (o Options) max() int {
if o.MaxChars > 0 {
return o.MaxChars
}
return 1800
}
func (o Options) overlap() int {
if o.Overlap >= 0 && o.MaxChars > 0 {
return min(o.Overlap, o.max()/2)
}
return 300
}
// Chunk is one piece: the headings above it and its text.
type Chunk struct {
// Headings is the path of headings the chunk starts under, widest first.
Headings []string
Body string
}
// Text is what is stored and embedded: the heading path, then the body. The
// path is part of it so that "how many days?" under "Annual leave" finds the
// paragraph that says "twenty days" without naming leave.
func (c Chunk) Text() string {
if len(c.Headings) == 0 {
return c.Body
}
return strings.Join(c.Headings, " › ") + "\n\n" + c.Body
}
// Split cuts Markdown-shaped text into chunks.
func Split(text string, o Options) []Chunk {
max, overlap := o.max(), o.overlap()
var out []Chunk
type head struct {
level int
title string
}
var stack []head // the headings in force, widest first
titles := func() []string {
out := make([]string, len(stack))
for i, h := range stack {
out[i] = h.title
}
return out
}
var cur []string // blocks of the chunk being built
var curHeads []string
size := 0
// carried is the overlap at the start of cur: it does not count as the
// chunk having content of its own.
carried := 0
emit := func(keepOverlap bool) {
if size-carried <= 0 {
cur, size, carried = nil, 0, 0
return
}
out = append(out, Chunk{Headings: curHeads, Body: strings.Join(cur, "\n\n")})
last := cur[len(cur)-1]
cur, size, carried = nil, 0, 0
if keepOverlap && overlap > 0 && runes(last) <= overlap {
cur, size, carried = []string{last}, runes(last), runes(last)
}
}
add := func(b string) {
if len(cur) == 0 || size == carried {
curHeads = titles()
}
if size > 0 && size+2+runes(b) > max {
emit(true)
curHeads = titles()
}
cur = append(cur, b)
size += runes(b) + 2
}
for _, b := range blocks(text) {
if lvl, title := heading(b); lvl > 0 {
// A new section starts a new chunk once this one is worth
// keeping by itself; otherwise the heading goes into it.
if size-carried >= max/3 {
emit(false)
}
for len(stack) > 0 && stack[len(stack)-1].level >= lvl {
stack = stack[:len(stack)-1]
}
stack = append(stack, head{lvl, title})
if size-carried > 0 {
add(b)
} else {
// Nothing of its own yet: the chunk starts under this heading.
cur, size, carried = nil, 0, 0
curHeads = titles()
}
continue
}
for _, piece := range fit(b, max) {
add(piece)
}
}
emit(false)
return out
}
func runes(s string) int { return utf8.RuneCountInString(s) }
// heading reads a Markdown ATX heading line: its level and title.
func heading(b string) (int, string) {
if strings.Contains(b, "\n") {
return 0, ""
}
n := 0
for n < len(b) && b[n] == '#' {
n++
}
if n == 0 || n > 6 || n >= len(b) || b[n] != ' ' {
return 0, ""
}
return n, strings.TrimSpace(strings.TrimRight(strings.TrimSpace(b[n:]), "#"))
}
// blocks splits text on blank lines, keeping fenced code blocks whole.
func blocks(text string) []string {
var out []string
var b []string
fenced := false
flush := func() {
s := strings.TrimSpace(strings.Join(b, "\n"))
if s != "" {
out = append(out, s)
}
b = nil
}
for _, line := range strings.Split(text, "\n") {
t := strings.TrimSpace(line)
if strings.HasPrefix(t, "```") || strings.HasPrefix(t, "~~~") {
fenced = !fenced
}
if !fenced && t == "" {
flush()
continue
}
// A heading line is a block of its own even without blank lines
// around it.
if !fenced && len(b) > 0 {
if lvl, _ := heading(t); lvl > 0 {
flush()
}
}
b = append(b, strings.TrimRight(line, " \t"))
if !fenced {
if lvl, _ := heading(t); lvl > 0 {
flush()
}
}
}
flush()
return out
}
// fit cuts a block longer than max: at line ends, then sentence ends, then
// spaces, and only as a last resort mid-word.
func fit(b string, max int) []string {
if runes(b) <= max {
return []string{b}
}
for _, split := range []func(string) []string{lines, sentences, words} {
parts := split(b)
if len(parts) < 2 {
continue
}
var out []string
var cur strings.Builder
for _, p := range parts {
if cur.Len() > 0 && runes(cur.String())+runes(p) > max {
out = append(out, strings.TrimSpace(cur.String()))
cur.Reset()
}
if runes(p) > max {
out = append(out, fit(strings.TrimSpace(p), max)...)
continue
}
cur.WriteString(p)
}
if strings.TrimSpace(cur.String()) != "" {
out = append(out, strings.TrimSpace(cur.String()))
}
return out
}
// One word longer than a chunk: a hash, a line of base64.
r := []rune(b)
var out []string
for len(r) > max {
out, r = append(out, string(r[:max])), r[max:]
}
return append(out, string(r))
}
// lines splits keeping each line's newline.
func lines(s string) []string { return strings.SplitAfter(s, "\n") }
// sentences splits after ., ? or ! followed by a space, keeping both.
func sentences(s string) []string {
var out []string
start := 0
r := []rune(s)
for i := 0; i < len(r)-1; i++ {
if (r[i] == '.' || r[i] == '?' || r[i] == '!' || r[i] == '…') && unicode.IsSpace(r[i+1]) {
out = append(out, string(r[start:i+2]))
start = i + 2
}
}
if start < len(r) {
out = append(out, string(r[start:]))
}
return out
}
func words(s string) []string { return strings.SplitAfter(s, " ") }
+106
View File
@@ -0,0 +1,106 @@
package chunk
import (
"strings"
"testing"
"unicode/utf8"
)
func para(word string, n int) string { return strings.TrimSpace(strings.Repeat(word+" ", n)) }
func TestShortTextIsOneChunkUnderItsHeadings(t *testing.T) {
cs := Split("# İzin\n\n## Yıllık\n\nYirmi gün.\n\nBayramlar ayrı.\n", Options{})
if len(cs) != 1 {
t.Fatalf("chunks %+v", cs)
}
if got := cs[0].Text(); got != "İzin › Yıllık\n\nYirmi gün.\n\nBayramlar ayrı." {
t.Errorf("text %q", got)
}
}
func TestChunksStayUnderTheLimitAndOverlap(t *testing.T) {
var b strings.Builder
for i := range 20 {
b.WriteString(para("kelime"+string(rune('a'+i)), 30) + "\n\n")
}
cs := Split(b.String(), Options{MaxChars: 600, Overlap: 300})
if len(cs) < 5 {
t.Fatalf("only %d chunks", len(cs))
}
for i, c := range cs {
if n := utf8.RuneCountInString(c.Body); n > 600 {
t.Errorf("chunk %d has %d characters", i, n)
}
if i > 0 {
prev := strings.Split(cs[i-1].Body, "\n\n")
if !strings.HasPrefix(c.Body, prev[len(prev)-1]) {
t.Errorf("chunk %d does not start with the end of chunk %d", i, i-1)
}
}
}
}
func TestASectionStartsAChunkOnceTheLastIsWorthKeeping(t *testing.T) {
text := "# A\n\n" + para("a", 400) + "\n\n# B\n\nkısa\n\n# C\n\nde kısa\n"
cs := Split(text, Options{MaxChars: 1800})
if len(cs) != 2 {
t.Fatalf("chunks %+v", cs)
}
if cs[1].Headings[0] != "B" || !strings.Contains(cs[1].Body, "# C") || !strings.Contains(cs[1].Body, "de kısa") {
t.Errorf("small sections were not kept together: %+v", cs[1])
}
// The overlap does not run into a new section.
if strings.HasPrefix(cs[1].Body, "a a") {
t.Errorf("section B starts with section A's text")
}
}
func TestHeadingLevelsReplaceTheirSiblings(t *testing.T) {
text := "## A\n\n### A1\n\n" + para("x", 300) + "\n\n## B\n\n" + para("y", 300) + "\n"
cs := Split(text, Options{MaxChars: 700})
last := cs[len(cs)-1]
if strings.Join(last.Headings, "/") != "B" {
t.Errorf("headings after a sibling: %v", last.Headings)
}
if strings.Join(cs[0].Headings, "/") != "A/A1" {
t.Errorf("first headings %v", cs[0].Headings)
}
}
func TestLongParagraphsAreCutAtSentencesThenWords(t *testing.T) {
long := strings.Repeat("Bu bir cümledir. ", 200)
for _, c := range Split(long, Options{MaxChars: 500}) {
if utf8.RuneCountInString(c.Body) > 500 {
t.Errorf("chunk of %d", utf8.RuneCountInString(c.Body))
}
if !strings.HasSuffix(c.Body, ".") {
t.Errorf("cut mid-sentence: …%q", c.Body[len(c.Body)-20:])
}
}
word := strings.Repeat("ş", 1200)
cs := Split(word, Options{MaxChars: 500})
total := 0
for _, c := range cs {
total += utf8.RuneCountInString(c.Body)
if !utf8.ValidString(c.Body) {
t.Error("a multibyte character was cut")
}
}
if total != 1200 {
t.Errorf("a long word lost characters: %d", total)
}
}
func TestCodeFencesStayWhole(t *testing.T) {
text := "Örnek:\n\n```go\nfunc a() {\n\n\treturn\n}\n```\n\nSon.\n"
cs := Split(text, Options{})
if len(cs) != 1 || !strings.Contains(cs[0].Body, "func a() {\n\n\treturn\n}") {
t.Errorf("chunks %+v", cs)
}
}
func TestNothingToSayIsNoChunk(t *testing.T) {
if cs := Split("# Başlık\n\n## Alt\n", Options{}); len(cs) != 0 {
t.Errorf("headings alone made %+v", cs)
}
}
+215
View File
@@ -0,0 +1,215 @@
package extract
import (
"bytes"
"context"
"encoding/base64"
"errors"
"fmt"
"io"
"mime"
"mime/quotedprintable"
"net/mail"
"path"
"strings"
)
// maxArchiveMembers is the most members of one archive read; past it the
// rest are reported skipped.
const maxArchiveMembers = 10000
// maxArchiveBytes is the most an archive's members are inflated to in all.
const maxArchiveBytes = 2 << 30
// zipParts reads every member of a zip archive this can read, each a part
// named by its path inside.
func zipParts(ctx context.Context, name string, b []byte, o Options, depth int) ([]Part, error) {
zr, err := openZip(b)
if err != nil {
return nil, err
}
var parts []Part
var total int64
for i, f := range zr.File {
if err := ctx.Err(); err != nil {
return parts, err
}
if f.FileInfo().IsDir() || skipName(f.Name) {
continue
}
mname := memberName(f.Name)
if i >= maxArchiveMembers {
parts = append(parts, Part{Name: mname, Err: fmt.Errorf("%w: past the first %d members", ErrUnsupported, maxArchiveMembers)})
continue
}
if !Known(f.Name) {
continue
}
if int64(f.UncompressedSize64) > o.maxBytes() {
parts = append(parts, Part{Name: mname, Err: fmt.Errorf("%w: %d MiB is over the limit", ErrUnsupported, f.UncompressedSize64>>20)})
continue
}
total += int64(f.UncompressedSize64)
if total > maxArchiveBytes {
parts = append(parts, Part{Name: mname, Err: fmt.Errorf("%w: the archive inflates past %d GiB", ErrUnsupported, maxArchiveBytes>>30)})
break
}
data, err := readMember(f)
if err != nil {
parts = append(parts, Part{Name: mname, Err: err})
continue
}
inner, err := extractNested(ctx, mname, data, o, depth+1)
parts = append(parts, nest(mname, inner, err)...)
}
if len(parts) == 0 {
return nil, ErrNoText
}
return parts, nil
}
func extractNested(ctx context.Context, name string, data []byte, o Options, depth int) ([]Part, error) {
return extractDepth(ctx, name, data, o, depth)
}
// nest names the parts of a member after it: "a/b.pdf", "a/c.zip!/d.txt".
func nest(name string, parts []Part, err error) []Part {
if err != nil {
return []Part{{Name: name, Err: err}}
}
out := make([]Part, 0, len(parts))
for _, p := range parts {
if p.Name == "" {
p.Name = name
} else {
p.Name = name + "!/" + p.Name
}
out = append(out, p)
}
return out
}
// memberName is a member's path made safe to name a document: forward
// slashes, no leading slash or dot segments.
func memberName(n string) string {
n = strings.ReplaceAll(n, `\`, "/")
n = path.Clean("/" + n)
return strings.TrimPrefix(n, "/")
}
// skipName is what systems leave in archives that is not anybody's
// document: macOS resource forks, Office lock files, thumbnails.
func skipName(n string) bool {
base := path.Base(strings.ReplaceAll(n, `\`, "/"))
return strings.HasPrefix(n, "__MACOSX/") || strings.HasPrefix(base, "._") || strings.HasPrefix(base, "~$") ||
base == ".DS_Store" || base == "Thumbs.db" || base == "desktop.ini"
}
// mailParts reads an e-mail (.eml): the message — its headers and the plain
// text, or the HTML when there is no plain text — and each attachment this
// can read, named by its file name.
func mailParts(ctx context.Context, name string, b []byte, o Options, depth int) ([]Part, error) {
m, err := mail.ReadMessage(bytes.NewReader(b))
if err != nil {
return nil, err
}
dec := new(mime.WordDecoder)
dec.CharsetReader = func(cs string, in io.Reader) (io.Reader, error) {
all, err := io.ReadAll(in)
if err != nil {
return nil, err
}
return strings.NewReader(decodeCharset(all, cs)), nil
}
header := func(k string) string {
v := m.Header.Get(k)
if d, err := dec.DecodeHeader(v); err == nil {
return d
}
return v
}
var plain, htmlBody []string
var attachments []Part
err = walkMIME(m.Header.Get("Content-Type"), m.Header.Get("Content-Transfer-Encoding"), m.Body, 0,
func(ctype, filename string, params map[string]string, body []byte) {
if filename != "" {
if d, err := dec.DecodeHeader(filename); err == nil {
filename = d
}
if Known(filename) {
inner, err := extractNested(ctx, filename, body, o, depth+1)
attachments = append(attachments, nest(memberName(filename), inner, err)...)
}
return
}
switch ctype {
case "text/plain":
plain = append(plain, decodeCharset(body, params["charset"]))
case "text/html":
if t, err := htmlFrom(strings.NewReader(decodeCharset(body, params["charset"]))); err == nil {
htmlBody = append(htmlBody, t)
}
case "message/rfc822":
if inner, err := mailParts(ctx, "", body, o, depth+1); err == nil {
attachments = append(attachments, nest("message", inner, nil)...)
}
}
})
if err != nil && len(plain) == 0 && len(htmlBody) == 0 {
return nil, err
}
var text strings.Builder
if s := header("Subject"); s != "" {
text.WriteString("# " + s + "\n\n")
}
for _, k := range []string{"From", "To", "Cc", "Date"} {
if v := header(k); v != "" {
text.WriteString(k + ": " + v + "\n")
}
}
text.WriteString("\n")
body := plain
if len(body) == 0 {
body = htmlBody
}
for _, p := range body {
text.WriteString(p + "\n\n")
}
parts := []Part{{Text: Clean(text.String())}}
parts = append(parts, attachments...)
return parts, nil
}
// transferDecoder undoes a MIME part's Content-Transfer-Encoding.
func transferDecoder(cte string, r io.Reader) io.Reader {
switch strings.ToLower(strings.TrimSpace(cte)) {
case "base64":
return base64.NewDecoder(base64.StdEncoding, newlineStripper{r})
case "quoted-printable":
return quotedprintable.NewReader(r)
}
return r
}
// newlineStripper drops CR and LF, which base64 in mail is wrapped with and
// encoding/base64 does not skip in every position.
type newlineStripper struct{ r io.Reader }
func (n newlineStripper) Read(p []byte) (int, error) {
for {
k, err := n.r.Read(p)
j := 0
for _, c := range p[:k] {
if c != '\r' && c != '\n' {
p[j] = c
j++
}
}
if j > 0 || err != nil {
if errors.Is(err, io.EOF) && j > 0 {
return j, nil
}
return j, err
}
}
}
+296
View File
@@ -0,0 +1,296 @@
// Package extract turns documents into plain text for knowledge bases
// (BACKLOG F2), on the machine that holds them.
//
// The text it produces is Markdown-shaped: headings as "# ", "## ", list
// items as "- ", paragraphs separated by a blank line. The chunker splits on
// exactly those, so a format that knows its headings (DOCX styles, HTML h1–h6,
// ODT outline levels, slides, sheets) says so, and its chunks carry them.
//
// Built-in formats are pure Go, so the client builds for every system without
// a C toolchain. What Go cannot read — Word 97 files, scanned pages — goes to
// a helper when one is on the machine (LibreOffice, Tesseract, Poppler), and
// is otherwise skipped with a sentence saying which helper would read it.
package extract
import (
"bytes"
"context"
"errors"
"fmt"
"os"
"path"
"path/filepath"
"regexp"
"sort"
"strings"
"unicode/utf8"
)
// ErrUnsupported is a file this cannot read, with the reason wrapped around
// it ("a .doc needs LibreOffice").
var ErrUnsupported = errors.New("unsupported")
// ErrNoText is a file that was read and held no text: a scanned PDF without
// an OCR helper, an empty sheet.
var ErrNoText = errors.New("no text in it")
// Part is the text of one document. A file is one part; an archive or a mail
// with attachments is several, each named by its path inside, after the
// outer file's name and "!/".
type Part struct {
// Name is "" for the file itself, or the path inside it.
Name string
Text string
// Err is why this part was skipped, when it was; Text is then empty.
Err error
}
// Options are the limits and helpers extraction uses.
type Options struct {
Helpers Helpers
// MaxBytes is the largest file, or member of an archive, read; 0 is
// 256 MiB.
MaxBytes int64
// OCRLanguages are Tesseract's languages; "" is "tur+eng".
OCRLanguages string
}
func (o Options) maxBytes() int64 {
if o.MaxBytes > 0 {
return o.MaxBytes
}
return 256 << 20
}
// File extracts a file on disk.
func File(ctx context.Context, name string, o Options) ([]Part, error) {
st, err := os.Stat(name)
if err != nil {
return nil, err
}
if st.Size() > o.maxBytes() {
return nil, fmt.Errorf("%w: %d MiB is over the %d MiB limit", ErrUnsupported, st.Size()>>20, o.maxBytes()>>20)
}
data, err := os.ReadFile(name)
if err != nil {
return nil, err
}
return Bytes(ctx, filepath.Base(name), data, o)
}
// Bytes extracts a document held in memory; name gives its format.
func Bytes(ctx context.Context, name string, data []byte, o Options) ([]Part, error) {
return extractDepth(ctx, name, data, o, 0)
}
// maxDepth is how many archives deep members are followed: a zip in a mail
// in a zip, and no further.
const maxDepth = 3
func extractDepth(ctx context.Context, name string, data []byte, o Options, depth int) ([]Part, error) {
if err := ctx.Err(); err != nil {
return nil, err
}
ext := Ext(name)
if f, ok := containers[ext]; ok {
if depth >= maxDepth {
return nil, fmt.Errorf("%w: archives nested more than %d deep", ErrUnsupported, maxDepth)
}
return f(ctx, name, data, o, depth)
}
text, err := one(ctx, name, ext, data, o)
if err != nil {
return nil, err
}
text = Clean(text)
if strings.TrimSpace(text) == "" {
return nil, ErrNoText
}
return []Part{{Text: text}}, nil
}
// one extracts a single document.
func one(ctx context.Context, name, ext string, data []byte, o Options) (string, error) {
if f, ok := builtin[ext]; ok {
text, err := f(data)
if err == nil && strings.TrimSpace(text) != "" {
return text, nil
}
// A PDF the Go reader cannot parse, or one with no text layer, is
// what the helpers are for.
if alt, ok := fallback[ext]; ok {
if t, aerr := alt(ctx, name, data, o); aerr == nil {
return t, nil
} else if err == nil {
err = aerr
}
}
if err == nil {
err = ErrNoText
}
return "", err
}
if alt, ok := fallback[ext]; ok {
return alt(ctx, name, data, o)
}
if looksLikeText(data) {
return decodeText(data), nil
}
if o.Helpers.Office != "" {
return o.Helpers.officeToText(ctx, name, data, o)
}
return "", fmt.Errorf("%w: no reader for %q files (LibreOffice, if installed, may read it)", ErrUnsupported, ext)
}
// Ext is a name's extension, lower case and without the dot.
func Ext(name string) string {
return strings.TrimPrefix(strings.ToLower(path.Ext(strings.ReplaceAll(name, `\`, "/"))), ".")
}
// Supported lists the extensions read without a helper, and those read with
// one, for --help and the docs.
func Supported() (builtIn, withHelper []string) {
for e := range builtin {
builtIn = append(builtIn, e)
}
for e := range containers {
builtIn = append(builtIn, e)
}
for e := range textExts {
builtIn = append(builtIn, e)
}
for e := range fallback {
if _, ok := builtin[e]; !ok {
withHelper = append(withHelper, e)
}
}
sort.Strings(builtIn)
sort.Strings(withHelper)
return builtIn, withHelper
}
// Known reports whether a name's extension is one this reads, with or without
// a helper, or one of the text formats.
func Known(name string) bool {
e := Ext(name)
_, a := builtin[e]
_, b := containers[e]
_, c := fallback[e]
return a || b || c || textExts[e]
}
var (
builtin map[string]func([]byte) (string, error)
containers map[string]func(context.Context, string, []byte, Options, int) ([]Part, error)
fallback map[string]func(context.Context, string, []byte, Options) (string, error)
)
func init() {
builtin = map[string]func([]byte) (string, error){
"pdf": pdfText,
"html": htmlText, "htm": htmlText, "xhtml": htmlText, "mht": mhtmlText, "mhtml": mhtmlText,
"docx": docxText, "docm": docxText, "dotx": docxText, "dotm": docxText,
"xlsx": xlsxText, "xlsm": xlsxText, "xltx": xlsxText, "xltm": xlsxText,
"pptx": pptxText, "pptm": pptxText, "ppsx": pptxText, "ppsm": pptxText, "potx": pptxText,
"odt": odfText, "ott": odfText, "ods": odfText, "ots": odfText, "odp": odfText, "otp": odfText,
"fodt": flatODFText, "fods": flatODFText, "fodp": flatODFText,
"rtf": rtfText,
"epub": epubText,
"ipynb": notebookText,
"srt": subtitleText, "vtt": subtitleText,
}
for e := range textExts {
builtin[e] = func(b []byte) (string, error) { return decodeText(b), nil }
}
containers = map[string]func(context.Context, string, []byte, Options, int) ([]Part, error){
"zip": zipParts,
"eml": mailParts,
}
fallback = map[string]func(context.Context, string, []byte, Options) (string, error){
"pdf": pdfWithHelpers,
"rtf": viaOffice,
}
for _, e := range officeOnly {
fallback[e] = viaOffice
}
for _, e := range imageExts {
fallback[e] = viaOCR
}
}
// textExts are formats that are text already: read as they are, decoded.
var textExts = map[string]bool{
"txt": true, "text": true, "md": true, "markdown": true, "mdx": true, "rst": true, "adoc": true,
"asciidoc": true, "org": true, "tex": true, "log": true, "csv": true, "tsv": true, "json": true,
"jsonl": true, "ndjson": true, "yaml": true, "yml": true, "toml": true, "ini": true, "cfg": true,
"conf": true, "properties": true, "env": true, "xml": true, "svg": true, "sql": true,
"go": true, "py": true, "js": true, "mjs": true, "cjs": true, "ts": true, "tsx": true, "jsx": true,
"java": true, "kt": true, "kts": true, "scala": true, "c": true, "h": true, "cc": true, "cpp": true,
"cxx": true, "hpp": true, "hh": true, "cs": true, "fs": true, "vb": true, "rs": true, "swift": true,
"m": true, "mm": true, "rb": true, "php": true, "pl": true, "pm": true, "lua": true, "r": true,
"jl": true, "dart": true, "ex": true, "exs": true, "erl": true, "hs": true, "clj": true,
"sh": true, "bash": true, "zsh": true, "fish": true, "ps1": true, "psm1": true, "bat": true,
"cmd": true, "groovy": true, "gradle": true, "tf": true, "hcl": true, "proto": true,
"graphql": true, "gql": true, "css": true, "scss": true, "sass": true, "less": true,
"vue": true, "svelte": true, "dockerfile": true, "makefile": true, "mk": true, "cmake": true,
"bib": true, "nfo": true, "diff": true, "patch": true,
}
// officeOnly are formats only LibreOffice reads here: the binary Office
// formats before 2007, and others it opens.
var officeOnly = []string{
"doc", "dot", "xls", "xlt", "xlsb", "ppt", "pps", "pot", "wpd", "wps", "wri",
"pages", "numbers", "key", "sxw", "sxc", "sxi", "hwp", "pub", "vsd", "vsdx",
}
// imageExts are read by OCR.
var imageExts = []string{"png", "jpg", "jpeg", "tif", "tiff", "bmp", "gif", "webp", "jp2", "pnm"}
// looksLikeText reports whether bytes are text: no NUL in the first 8 KiB
// (UTF-16 has its BOM checked first), and mostly printable.
func looksLikeText(b []byte) bool {
if hasUTF16BOM(b) {
return true
}
head := b[:min(len(b), 8192)]
if bytes.IndexByte(head, 0) >= 0 {
return false
}
if utf8.Valid(head) {
return true
}
// A single-byte code page: allow it when control characters are rare.
ctl := 0
for _, c := range head {
if c < 0x20 && c != '\n' && c != '\r' && c != '\t' && c != '\f' {
ctl++
}
}
return ctl*100 < len(head)
}
var (
spaceRun = regexp.MustCompile(`[ \t\x{00A0}]+\n`)
blankRun = regexp.MustCompile(`\n{3,}`)
)
// Clean normalises extracted text: LF line ends, no trailing spaces, no more
// than one blank line in a row, no control characters but tab and newline.
func Clean(s string) string {
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.ReplaceAll(s, "\r", "\n")
s = strings.Map(func(r rune) rune {
switch {
case r == '\n' || r == '\t':
return r
case r == '\f' || r == '\v':
return '\n'
case r < 0x20 || r == 0x7f || r == utf8.RuneError || r == '\uFEFF':
return -1
}
return r
}, s)
s = spaceRun.ReplaceAllString(s, "\n")
s = blankRun.ReplaceAllString(s, "\n\n")
return strings.TrimSpace(s) + "\n"
}
+323
View File
@@ -0,0 +1,323 @@
package extract
import (
"archive/zip"
"bytes"
"context"
"encoding/base64"
"errors"
"os"
"strings"
"testing"
"golang.org/x/text/encoding/charmap"
"golang.org/x/text/encoding/unicode"
)
// zipOf builds a zip from name → content, in the order given.
func zipOf(t *testing.T, files ...string) []byte {
t.Helper()
var buf bytes.Buffer
w := zip.NewWriter(&buf)
for i := 0; i+1 < len(files); i += 2 {
f, err := w.Create(files[i])
if err != nil {
t.Fatal(err)
}
if _, err := f.Write([]byte(files[i+1])); err != nil {
t.Fatal(err)
}
}
if err := w.Close(); err != nil {
t.Fatal(err)
}
return buf.Bytes()
}
func extractOne(t *testing.T, name string, data []byte) string {
t.Helper()
parts, err := Bytes(context.Background(), name, data, Options{})
if err != nil {
t.Fatalf("%s: %v", name, err)
}
if len(parts) != 1 || parts[0].Err != nil {
t.Fatalf("%s: parts %+v", name, parts)
}
return parts[0].Text
}
func wants(t *testing.T, got string, want ...string) {
t.Helper()
for _, w := range want {
if !strings.Contains(got, w) {
t.Errorf("missing %q in:\n%s", w, got)
}
}
}
func TestTextInEveryEncodingBecomesUTF8(t *testing.T) {
const s = "Yıllık izin: Şubat'ta ğüşİöç"
u16, _ := unicode.UTF16(unicode.LittleEndian, unicode.UseBOM).NewEncoder().Bytes([]byte(s))
cp, _ := charmap.Windows1254.NewEncoder().Bytes([]byte(s))
for name, b := range map[string][]byte{
"utf8": []byte(s),
"utf8-bom": append([]byte{0xEF, 0xBB, 0xBF}, s...),
"utf16": u16,
"cp1254": cp,
"crlf": []byte(strings.ReplaceAll(s+"\nikinci", "\n", "\r\n")),
} {
got := extractOne(t, "a.txt", b)
if !strings.HasPrefix(got, s) {
t.Errorf("%s: got %q", name, got)
}
if strings.Contains(got, "\r") {
t.Errorf("%s: kept a CR", name)
}
}
// macOS writes names and sometimes text decomposed; the text comes out
// composed, so the same word is the same bytes everywhere.
if got := extractOne(t, "a.txt", []byte("Şirket")); !strings.HasPrefix(got, "Şirket") {
t.Errorf("NFD text came out as %q", got)
}
}
func TestHTMLKeepsHeadingsAndDropsScripts(t *testing.T) {
page := `<html><head><title>T</title><script>var x="gizli";</script></head><body>
<nav>menü</nav><h1>İzin Politikası</h1><p>Yıllık <b>izin</b> yirmi&nbsp;gün.</p>
<ul><li>Bir</li><li>İki</li></ul><table><tr><td>a</td><td>b</td></tr></table>
<p><b>Mer</b>haba</p></body></html>`
got := extractOne(t, "a.html", []byte(page))
wants(t, got, "# İzin Politikası", "Yıllık izin yirmi gün.", "- Bir", "- İki", "a\tb", "Merhaba")
for _, no := range []string{"gizli", "menü"} {
if strings.Contains(got, no) {
t.Errorf("%q leaked into:\n%s", no, got)
}
}
// Declared charset honoured.
l5, _ := charmap.ISO8859_9.NewEncoder().Bytes([]byte(`<meta charset="iso-8859-9"><p>Şirket</p>`))
wants(t, extractOne(t, "b.htm", l5), "Şirket")
}
func TestWordDocument(t *testing.T) {
doc := `<?xml version="1.0" encoding="UTF-8"?>
<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body>
<w:p><w:pPr><w:pStyle w:val="Balk1"/></w:pPr><w:r><w:t>Yıllık İzin</w:t></w:r></w:p>
<w:p><w:r><w:t xml:space="preserve">Çalışanlar </w:t></w:r><w:r><w:t>yirmi gün</w:t></w:r><w:del><w:r><w:delText>otuz</w:delText></w:r></w:del><w:r><w:tab/><w:t>izin alır.</w:t></w:r></w:p>
<w:p><w:pPr><w:numPr><w:ilvl w:val="0"/></w:numPr></w:pPr><w:r><w:t>Madde bir</w:t></w:r></w:p>
<w:tbl><w:tr><w:tc><w:p><w:r><w:t>Hücre1</w:t></w:r></w:p></w:tc><w:tc><w:p><w:r><w:t>Hücre2</w:t></w:r></w:p></w:tc></w:tr></w:tbl>
<w:p><w:pPr><w:pStyle w:val="Heading2"/></w:pPr><w:r><w:t>Masraflar</w:t></w:r></w:p>
</w:body></w:document>`
got := extractOne(t, "a.docx", zipOf(t, "word/document.xml", doc,
"word/footnotes.xml", `<w:footnotes xmlns:w="x"><w:footnote><w:p><w:r><w:t>Dipnot</w:t></w:r></w:p></w:footnote></w:footnotes>`))
wants(t, got, "# Yıllık İzin", "Çalışanlar yirmi gün\tizin alır.", "- Madde bir", "Hücre1\tHücre2", "## Masraflar", "Dipnot")
if strings.Contains(got, "otuz") {
t.Errorf("a tracked deletion is in the text:\n%s", got)
}
}
func TestWorkbook(t *testing.T) {
got := extractOne(t, "a.xlsx", zipOf(t,
"xl/workbook.xml", `<workbook xmlns:r="r"><sheets><sheet name="Bütçe" sheetId="1" r:id="rId1"/><sheet name="Boş" sheetId="2" r:id="rId2"/></sheets></workbook>`,
"xl/_rels/workbook.xml.rels", `<Relationships><Relationship Id="rId1" Target="worksheets/sheet1.xml"/><Relationship Id="rId2" Target="worksheets/sheet2.xml"/></Relationships>`,
"xl/sharedStrings.xml", `<sst><si><t>Kalem</t></si><si><r><t>Tu</t></r><r><t>tar</t></r></si><si><t>Kira</t></si></sst>`,
"xl/worksheets/sheet1.xml", `<worksheet><sheetData><row><c t="s"><v>0</v></c><c t="s"><v>1</v></c></row><row><c t="s"><v>2</v></c><c><v>1500.5</v></c><c t="inlineStr"><is><t>not</t></is></c></row></sheetData></worksheet>`,
"xl/worksheets/sheet2.xml", `<worksheet><sheetData/></worksheet>`,
))
wants(t, got, "## Bütçe", "Kalem\tTutar", "Kira\t1500.5\tnot")
}
func TestPresentationInItsOwnOrderWithNotes(t *testing.T) {
slide := func(text string) string {
return `<p:sld xmlns:a="a" xmlns:p="p"><p:cSld><p:spTree><p:sp><p:txBody><a:p><a:r><a:t>` + text +
`</a:t></a:r></a:p><a:p><a:fld type="slidenum"><a:t>9</a:t></a:fld></a:p></p:txBody></p:sp></p:spTree></p:cSld></p:sld>`
}
got := extractOne(t, "a.pptx", zipOf(t,
"ppt/presentation.xml", `<p:presentation xmlns:p="p" xmlns:r="r"><p:sldIdLst><p:sldId id="256" r:id="rId3"/><p:sldId id="257" r:id="rId2"/></p:sldIdLst></p:presentation>`,
"ppt/_rels/presentation.xml.rels", `<Relationships><Relationship Id="rId2" Target="slides/slide1.xml"/><Relationship Id="rId3" Target="slides/slide2.xml"/></Relationships>`,
"ppt/slides/slide1.xml", slide("İkinci slayt"),
"ppt/slides/slide2.xml", slide("Birinci slayt"),
"ppt/slides/_rels/slide2.xml.rels", `<Relationships><Relationship Id="rId1" Type="http://x/notesSlide" Target="../notesSlides/notesSlide7.xml"/></Relationships>`,
"ppt/notesSlides/notesSlide7.xml", slide("Konuşmacı notu"),
))
first, second := strings.Index(got, "Birinci slayt"), strings.Index(got, "İkinci slayt")
if first < 0 || second < 0 || first > second {
t.Errorf("slides out of the presentation's order:\n%s", got)
}
wants(t, got, "## Slide 1", "Notes:", "Konuşmacı notu")
if strings.Contains(got, "9") {
t.Errorf("a slide number field is in the text:\n%s", got)
}
}
func TestOpenDocument(t *testing.T) {
content := `<office:document-content xmlns:office="o" xmlns:text="t" xmlns:table="tb"><office:body><office:text>
<text:h text:outline-level="2">Bölüm</text:h><text:p>Bir<text:s text:c="2"/>iki<text:tab/>üç</text:p>
<text:list><text:list-item><text:p>madde</text:p></text:list-item></text:list>
<table:table table:name="Tablo1"><table:table-row><table:table-cell><text:p>x</text:p></table:table-cell><table:table-cell><text:p>y</text:p></table:table-cell></table:table-row></table:table>
</office:text></office:body></office:document-content>`
got := extractOne(t, "a.odt", zipOf(t, "mimetype", "application/vnd.oasis.opendocument.text", "content.xml", content))
wants(t, got, "## Bölüm", "Bir iki\tüç", "- madde", "## Tablo1", "x\ty")
wants(t, extractOne(t, "a.fodt", []byte(content)), "## Bölüm")
}
func TestRichText(t *testing.T) {
rtf := `{\rtf1\ansi\ansicpg1254\deff0{\fonttbl{\f0 Calibri;}}{\colortbl;\red0\green0\blue0;}` +
`{\*\generator Riched20;}\pard Y\'fdll\'fdk izin\par ` +
`\u350?irket \u-3913?\'3f ok\tab sonra\par{\header gizli}\cell b\row }`
got := extractOne(t, "a.rtf", []byte(rtf))
wants(t, got, "Yıllık izin", "Şirket", "ok\tsonra")
for _, no := range []string{"Calibri", "Riched", "gizli"} {
if strings.Contains(got, no) {
t.Errorf("%q leaked into:\n%s", no, got)
}
}
}
func TestEbookInSpineOrder(t *testing.T) {
got := extractOne(t, "a.epub", zipOf(t,
"mimetype", "application/epub+zip",
"META-INF/container.xml", `<container><rootfiles><rootfile full-path="OEBPS/content.opf"/></rootfiles></container>`,
"OEBPS/content.opf", `<package><manifest><item id="c1" href="b.xhtml"/><item id="c2" href="a.xhtml"/></manifest><spine><itemref idref="c2"/><itemref idref="c1"/></spine></package>`,
"OEBPS/a.xhtml", `<html><body><h1>Önce</h1></body></html>`,
"OEBPS/b.xhtml", `<html><body><h1>Sonra</h1></body></html>`,
))
if a, b := strings.Index(got, "Önce"), strings.Index(got, "Sonra"); a < 0 || b < a {
t.Errorf("chapters out of spine order:\n%s", got)
}
}
func TestNotebookAndSubtitles(t *testing.T) {
nb := `{"metadata":{"kernelspec":{"language":"python"}},"cells":[{"cell_type":"markdown","source":["# Analiz\n","Veri"]},
{"cell_type":"code","source":"print(1)","outputs":[{"text":"çıktı"}]}]}`
got := extractOne(t, "a.ipynb", []byte(nb))
wants(t, got, "# Analiz", "```python\nprint(1)\n```")
if strings.Contains(got, "çıktı") {
t.Errorf("an output is in the text:\n%s", got)
}
srt := "1\n00:00:01,000 --> 00:00:02,000\n<i>Merhaba</i>\n\n2\n00:00:02,000 --> 00:00:03,000\nMerhaba\nDünya\n"
if got := extractOne(t, "a.srt", []byte(srt)); got != "Merhaba\nDünya\n" {
t.Errorf("subtitles came out as %q", got)
}
}
func TestPDFTextLayer(t *testing.T) {
b, err := os.ReadFile("testdata/turkish.pdf")
if err != nil {
t.Fatal(err)
}
wants(t, extractOne(t, "a.pdf", b), "Yıllık İzin Politikası", "Masraflar finans ekibine iletilir.")
// Garbage is a failure, not a panic.
if _, err := Bytes(context.Background(), "bad.pdf", []byte("%PDF-1.4 nonsense"), Options{}); err == nil {
t.Error("a broken PDF was read")
}
}
func TestArchivesAreReadMemberByMember(t *testing.T) {
inner := zipOf(t, "deep/c.md", "# Derin")
archive := zipOf(t,
"docs/a.txt", "Birinci",
`docs\b.html`, "<p>İkinci</p>",
"__MACOSX/docs/._a.txt", "çöp",
"docs/~$kilit.docx", "çöp",
"inner.zip", string(inner),
"image.bin", "\x00\x01",
)
parts, err := Bytes(context.Background(), "arsiv.zip", archive, Options{})
if err != nil {
t.Fatal(err)
}
got := map[string]string{}
for _, p := range parts {
if p.Err != nil {
t.Errorf("%s: %v", p.Name, p.Err)
}
got[p.Name] = p.Text
}
if len(got) != 3 || !strings.Contains(got["docs/a.txt"], "Birinci") || !strings.Contains(got["docs/b.html"], "İkinci") ||
!strings.Contains(got["inner.zip!/deep/c.md"], "# Derin") {
t.Errorf("archive parts %v", got)
}
}
func TestMailWithAttachments(t *testing.T) {
att := zipOf(t, "x.txt", "ekteki metin")
enc := func(b []byte) string {
s := base64.StdEncoding.EncodeToString(b)
var lines []string
for len(s) > 76 {
lines, s = append(lines, s[:76]), s[76:]
}
return strings.Join(append(lines, s), "\r\n")
}
msg := "From: =?UTF-8?B?QXnFn2U=?= <a@example.com>\r\n" +
"Subject: =?iso-8859-9?Q?=DEubat_raporu?=\r\n" +
"Content-Type: multipart/mixed; boundary=B\r\n\r\n" +
"--B\r\nContent-Type: multipart/alternative; boundary=C\r\n\r\n" +
"--C\r\nContent-Type: text/plain; charset=utf-8\r\nContent-Transfer-Encoding: quoted-printable\r\n\r\n" +
"Rapor ekte, te=C5=9Fekk=C3=BCrler.\r\n" +
"--C\r\nContent-Type: text/html; charset=utf-8\r\n\r\n<p>HTML hali</p>\r\n--C--\r\n" +
"--B\r\nContent-Type: application/zip; name=\"ek.zip\"\r\nContent-Transfer-Encoding: base64\r\n" +
"Content-Disposition: attachment; filename=\"ek.zip\"\r\n\r\n" + enc(att) + "\r\n--B--\r\n"
parts, err := Bytes(context.Background(), "a.eml", []byte(msg), Options{})
if err != nil {
t.Fatal(err)
}
if len(parts) != 2 {
t.Fatalf("parts %+v", parts)
}
wants(t, parts[0].Text, "# Şubat raporu", "From: Ayşe", "Rapor ekte, teşekkürler.")
if strings.Contains(parts[0].Text, "HTML hali") {
t.Errorf("the HTML alternative was read beside the plain text:\n%s", parts[0].Text)
}
if parts[1].Name != "ek.zip!/x.txt" || !strings.Contains(parts[1].Text, "ekteki metin") {
t.Errorf("attachment %+v", parts[1])
}
}
func TestUnreadableFilesSayWhy(t *testing.T) {
_, err := Bytes(context.Background(), "a.doc", []byte{0xD0, 0xCF, 0x11, 0xE0, 0}, Options{})
if !errors.Is(err, ErrUnsupported) || !strings.Contains(err.Error(), "LibreOffice") {
t.Errorf("a .doc without LibreOffice gave %v", err)
}
_, err = Bytes(context.Background(), "a.png", []byte{0x89, 'P', 'N', 'G', 0}, Options{})
if !errors.Is(err, ErrUnsupported) || !strings.Contains(err.Error(), "Tesseract") {
t.Errorf("an image without Tesseract gave %v", err)
}
_, err = Bytes(context.Background(), "a.bin", []byte{0, 1, 2, 3}, Options{})
if !errors.Is(err, ErrUnsupported) {
t.Errorf("binary gave %v", err)
}
_, err = Bytes(context.Background(), "a.txt", []byte(" \n"), Options{})
if !errors.Is(err, ErrNoText) {
t.Errorf("an empty file gave %v", err)
}
}
func TestHelpersWhenInstalled(t *testing.T) {
h := FindHelpers()
ctx := context.Background()
if h.Tesseract != "" && h.PdfToPPM != "" {
// The text-layer PDF read by OCR instead: what a scanned one gets.
b, _ := os.ReadFile("testdata/turkish.pdf")
got, err := pdfWithHelpers(ctx, "a.pdf", b, Options{Helpers: Helpers{Tesseract: h.Tesseract, PdfToPPM: h.PdfToPPM}})
if err != nil {
t.Fatal(err)
}
wants(t, got, "Masraflar")
} else {
t.Log("no Tesseract with pdftoppm here; OCR not tried")
}
if h.Office == "" {
t.Log("no LibreOffice here; legacy Office formats not tried")
return
}
// A Word document converted to the legacy format and back would need a
// fixture made by Word; an ODT through the Office path proves the
// conversion runs and its result is read.
content := `<office:document-content xmlns:office="urn:oasis:names:tc:opendocument:xmlns:office:1.0" xmlns:text="urn:oasis:names:tc:opendocument:xmlns:text:1.0"><office:body><office:text><text:p>LibreOffice yolu</text:p></office:text></office:body></office:document-content>`
got, err := h.officeToText(ctx, "a.fodt", []byte(`<?xml version="1.0"?>`+content), Options{})
if err != nil {
t.Fatal(err)
}
wants(t, got, "LibreOffice yolu")
}
+277
View File
@@ -0,0 +1,277 @@
package extract
import (
"context"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"runtime"
"sort"
"strings"
"time"
)
// Helpers are programs on this machine that read what Go here cannot. Any
// may be "", and the formats that need it are then skipped with a reason.
type Helpers struct {
// Office is LibreOffice's soffice: the binary Office formats and others.
Office string
// Tesseract reads images and scanned pages.
Tesseract string
// PdfToText and PdfToPPM are Poppler's: a second PDF reader, and pages
// rendered for Tesseract.
PdfToText string
PdfToPPM string
}
// helperTimeout bounds one helper run; a conversion that hangs is a file
// skipped, not a sync stopped.
const helperTimeout = 3 * time.Minute
// FindHelpers looks for the helpers on PATH and where their installers put
// them on each system.
func FindHelpers() Helpers {
return Helpers{
Office: find("soffice", officePlaces()...),
Tesseract: find("tesseract", tesseractPlaces()...),
PdfToText: find("pdftotext"),
PdfToPPM: find("pdftoppm"),
}
}
// Describe says which helpers were found, and what each missing one would
// read.
func (h Helpers) Describe() []string {
line := func(have, name, reads string) string {
if have != "" {
return fmt.Sprintf("%-11s %s", name, have)
}
return fmt.Sprintf("%-11s not found — %s are skipped", name, reads)
}
return []string{
line(h.Office, "LibreOffice", "Word 97–2003, Excel 97–2003, PowerPoint 97–2003, Pages, Numbers, Keynote and similar files"),
line(h.Tesseract, "Tesseract", "images and scanned PDFs"),
line(h.PdfToText, "pdftotext", "PDFs the built-in reader cannot parse"),
line(h.PdfToPPM, "pdftoppm", "scanned PDFs (Tesseract needs their pages as images)"),
}
}
func find(name string, places ...string) string {
exe := name
if runtime.GOOS == "windows" {
exe += ".exe"
}
if p, err := exec.LookPath(exe); err == nil {
return p
}
if runtime.GOOS != "windows" {
if p, err := exec.LookPath(name); err == nil {
return p
}
}
for _, p := range places {
if matches, _ := filepath.Glob(p); len(matches) > 0 {
sort.Strings(matches)
return matches[len(matches)-1]
}
}
return ""
}
func officePlaces() []string {
switch runtime.GOOS {
case "windows":
return []string{
`C:\Program Files\LibreOffice*\program\soffice.exe`,
`C:\Program Files (x86)\LibreOffice*\program\soffice.exe`,
}
case "darwin":
return []string{"/Applications/LibreOffice.app/Contents/MacOS/soffice"}
}
return []string{"/usr/bin/libreoffice", "/usr/lib/libreoffice/program/soffice", "/opt/libreoffice*/program/soffice",
"/snap/bin/libreoffice"}
}
func tesseractPlaces() []string {
switch runtime.GOOS {
case "windows":
return []string{`C:\Program Files\Tesseract-OCR\tesseract.exe`, `C:\Program Files (x86)\Tesseract-OCR\tesseract.exe`}
case "darwin":
return []string{"/opt/homebrew/bin/tesseract", "/usr/local/bin/tesseract"}
}
return nil
}
// workDir is a temporary directory for one helper run, removed after.
func workDir() (string, func(), error) {
d, err := os.MkdirTemp("", "llmgw-kb-")
if err != nil {
return "", nil, err
}
return d, func() { _ = os.RemoveAll(d) }, nil
}
// writeInput puts a document where a helper can read it, under a safe name
// that keeps its extension.
func writeInput(dir, name string, data []byte) (string, error) {
p := filepath.Join(dir, "input."+Ext(name))
return p, os.WriteFile(p, data, 0o600)
}
func run(ctx context.Context, dir string, bin string, args ...string) ([]byte, error) {
ctx, cancel := context.WithTimeout(ctx, helperTimeout)
defer cancel()
cmd := exec.CommandContext(ctx, bin, args...)
cmd.Dir = dir
out, err := cmd.Output()
if ctx.Err() != nil {
return nil, fmt.Errorf("%s took over %s", filepath.Base(bin), helperTimeout)
}
if err != nil {
var ee *exec.ExitError
if errors.As(err, &ee) && len(ee.Stderr) > 0 {
return nil, fmt.Errorf("%s: %v: %s", filepath.Base(bin), err, lastLine(ee.Stderr))
}
return nil, fmt.Errorf("%s: %w", filepath.Base(bin), err)
}
return out, nil
}
func lastLine(b []byte) string {
lines := strings.Split(strings.TrimSpace(string(b)), "\n")
return strings.TrimSpace(lines[len(lines)-1])
}
// fileURL is a path as LibreOffice takes its profile directory: a file URL,
// with a drive letter on Windows.
func fileURL(p string) string {
p = filepath.ToSlash(p)
if !strings.HasPrefix(p, "/") {
p = "/" + p
}
return "file://" + p
}
var spreadsheets = map[string]bool{"xls": true, "xlt": true, "xlsb": true, "numbers": true, "sxc": true}
var presentations = map[string]bool{"ppt": true, "pps": true, "pot": true, "key": true, "sxi": true}
func viaOffice(ctx context.Context, name string, data []byte, o Options) (string, error) {
if o.Helpers.Office == "" {
return "", fmt.Errorf("%w: a .%s file needs LibreOffice, which is not installed here", ErrUnsupported, Ext(name))
}
return o.Helpers.officeToText(ctx, name, data, o)
}
// officeToText has LibreOffice convert a document to the matching Office
// Open XML format, then reads that: headings, sheets and slides survive,
// which a conversion to plain text would lose.
func (h Helpers) officeToText(ctx context.Context, name string, data []byte, _ Options) (string, error) {
dir, done, err := workDir()
if err != nil {
return "", err
}
defer done()
in, err := writeInput(dir, name, data)
if err != nil {
return "", err
}
ext := Ext(name)
to, read := "docx", docxText
switch {
case spreadsheets[ext]:
to, read = "xlsx", xlsxText
case presentations[ext]:
to, read = "pptx", pptxText
}
// A profile of its own: two conversions at once with the user's
// profile wait on each other's lock, or fail.
profile := fileURL(filepath.Join(dir, "profile"))
if _, err := run(ctx, dir, h.Office, "-env:UserInstallation="+profile, "--headless", "--norestore",
"--convert-to", to, "--outdir", dir, in); err != nil {
return "", err
}
converted, err := os.ReadFile(filepath.Join(dir, "input."+to))
if err != nil {
return "", fmt.Errorf("LibreOffice did not convert it: %w", err)
}
return read(converted)
}
func viaOCR(ctx context.Context, name string, data []byte, o Options) (string, error) {
if o.Helpers.Tesseract == "" {
return "", fmt.Errorf("%w: an image needs Tesseract to be read, which is not installed here", ErrUnsupported)
}
dir, done, err := workDir()
if err != nil {
return "", err
}
defer done()
in, err := writeInput(dir, name, data)
if err != nil {
return "", err
}
return ocr(ctx, dir, o, in)
}
func ocr(ctx context.Context, dir string, o Options, file string) (string, error) {
langs := o.OCRLanguages
if langs == "" {
langs = "tur+eng"
}
out, err := run(ctx, dir, o.Helpers.Tesseract, file, "stdout", "-l", langs)
if err != nil && langs != "eng" {
// A Tesseract without the Turkish data still reads English.
out, err = run(ctx, dir, o.Helpers.Tesseract, file, "stdout", "-l", "eng")
}
return string(out), err
}
// maxOCRPages bounds how many pages of one scanned PDF are read.
const maxOCRPages = 500
// pdfWithHelpers reads a PDF the built-in reader could not: with pdftotext,
// and when that finds no text either, page by page through Tesseract.
func pdfWithHelpers(ctx context.Context, name string, data []byte, o Options) (string, error) {
h := o.Helpers
if h.PdfToText == "" && (h.Tesseract == "" || h.PdfToPPM == "") {
return "", fmt.Errorf("%w: no text layer the built-in reader can use; Poppler (pdftotext) or "+
"Tesseract with pdftoppm would read it", ErrNoText)
}
dir, done, err := workDir()
if err != nil {
return "", err
}
defer done()
in, err := writeInput(dir, name, data)
if err != nil {
return "", err
}
if h.PdfToText != "" {
if out, err := run(ctx, dir, h.PdfToText, "-enc", "UTF-8", in, "-"); err == nil && strings.TrimSpace(string(out)) != "" {
return string(out), nil
}
}
if h.Tesseract == "" || h.PdfToPPM == "" {
return "", fmt.Errorf("%w: a scanned PDF needs Tesseract and pdftoppm", ErrNoText)
}
if _, err := run(ctx, dir, h.PdfToPPM, "-r", "300", "-gray", "-png", "-l", fmt.Sprint(maxOCRPages), in,
filepath.Join(dir, "page")); err != nil {
return "", err
}
pages, _ := filepath.Glob(filepath.Join(dir, "page-*.png"))
// pdftoppm pads page numbers to the width of the last one, so the
// names sort in page order.
sort.Strings(pages)
var all strings.Builder
for _, p := range pages {
t, err := ocr(ctx, dir, o, p)
if err != nil {
return "", err
}
all.WriteString(t)
all.WriteString("\n\n")
}
return all.String(), nil
}
+277
View File
@@ -0,0 +1,277 @@
package extract
import (
"bytes"
"io"
"mime"
"mime/multipart"
"net/mail"
"strings"
"golang.org/x/net/html"
"golang.org/x/net/html/charset"
)
// out builds Markdown-shaped text: paragraphs, headings, list items, table
// rows, with whitespace collapsed inside them.
type out struct {
b strings.Builder
// line is the text of the line being built, whitespace collapsed.
line strings.Builder
prefix string
}
// text adds running text; runs of whitespace become one space, and none is
// added where the source had none ("<b>Mer</b>haba" is one word).
func (o *out) text(s string) {
if s == "" {
return
}
r := []rune(s)
lead, trail := isSpace(r[0]), isSpace(r[len(r)-1])
for i, f := range strings.FieldsFunc(s, isSpace) {
if i > 0 || lead {
o.space()
}
o.line.WriteString(f)
}
if trail {
o.space()
}
}
// space adds one space unless the line is empty or already ends in one.
func (o *out) space() {
l := o.line.String()
if l != "" && !strings.HasSuffix(l, " ") && !strings.HasSuffix(l, "\t") {
o.line.WriteByte(' ')
}
}
// raw adds text as it is, for preformatted blocks.
func (o *out) raw(s string) { o.line.WriteString(s) }
// tab separates cells.
func (o *out) tab() {
o.trimLine()
o.line.WriteString("\t")
}
func (o *out) trimLine() {
s := strings.TrimRight(o.line.String(), " ")
o.line.Reset()
o.line.WriteString(s)
}
// newline ends the line being built, if it has anything.
func (o *out) newline() {
s := strings.TrimSpace(o.line.String())
o.line.Reset()
if s == "" {
return
}
o.b.WriteString(o.prefix)
o.b.WriteString(s)
o.b.WriteByte('\n')
o.prefix = ""
}
// para ends a paragraph: the line, and a blank line after it.
func (o *out) para() {
o.newline()
if o.b.Len() > 0 && !strings.HasSuffix(o.b.String(), "\n\n") {
o.b.WriteByte('\n')
}
}
// heading starts a heading of a level, 1 to 6.
func (o *out) heading(level int) {
o.para()
level = max(1, min(level, 6))
o.prefix = strings.Repeat("#", level) + " "
}
// item starts a list item.
func (o *out) item() {
o.newline()
o.prefix = "- "
}
func (o *out) String() string {
o.newline()
return o.b.String()
}
func isSpace(r rune) bool {
return r == ' ' || r == '\t' || r == '\n' || r == '\r' || r == '\f' || r == ' '
}
// htmlSkip are elements whose content is not text a reader sees.
var htmlSkip = map[string]bool{
"script": true, "style": true, "noscript": true, "template": true, "svg": true,
"head": true, "nav": true, "iframe": true, "object": true, "canvas": true, "button": true, "select": true,
}
// htmlBlock are elements that end a paragraph.
var htmlBlock = map[string]bool{
"p": true, "div": true, "section": true, "article": true, "main": true, "header": true,
"footer": true, "aside": true, "blockquote": true, "pre": true, "table": true, "ul": true,
"ol": true, "dl": true, "figure": true, "figcaption": true, "address": true, "hr": true,
"form": true, "fieldset": true, "details": true, "summary": true, "body": true, "dt": true, "dd": true,
"caption": true,
}
// htmlText reads an HTML page, in the charset it declares.
func htmlText(b []byte) (string, error) {
r, err := charset.NewReader(bytes.NewReader(b), "text/html")
if err != nil {
r = bytes.NewReader(b)
}
return htmlFrom(r)
}
// htmlFrom reads HTML already decoded to UTF-8.
func htmlFrom(r io.Reader) (string, error) {
z := html.NewTokenizer(r)
var o out
skip, pre := 0, 0
for {
tt := z.Next()
switch tt {
case html.ErrorToken:
if z.Err() == io.EOF {
return o.String(), nil
}
return o.String(), z.Err()
case html.TextToken:
if skip > 0 {
continue
}
if pre > 0 {
o.raw(string(z.Text()))
} else {
o.text(string(z.Text()))
}
case html.StartTagToken, html.SelfClosingTagToken, html.EndTagToken:
name, _ := z.TagName()
tag := string(name)
end := tt == html.EndTagToken
if htmlSkip[tag] {
if tt == html.StartTagToken {
skip++
} else if end && skip > 0 {
skip--
}
continue
}
if skip > 0 {
continue
}
switch {
case len(tag) == 2 && tag[0] == 'h' && tag[1] >= '1' && tag[1] <= '6':
if end {
o.para()
} else {
o.heading(int(tag[1] - '0'))
}
case tag == "title" && !end:
// Inside head, which is skipped; a title outside it is a
// heading.
o.heading(1)
case tag == "li":
if !end {
o.item()
} else {
o.newline()
}
case tag == "br":
o.newline()
case tag == "tr":
o.newline()
case tag == "td" || tag == "th":
if end {
o.tab()
}
case tag == "pre":
if end {
pre--
} else {
pre++
}
o.para()
case htmlBlock[tag]:
o.para()
}
}
}
}
// mhtmlText reads a web page saved as one file (MHTML): a MIME message whose
// HTML part is the page.
func mhtmlText(b []byte) (string, error) {
m, err := mail.ReadMessage(bytes.NewReader(b))
if err != nil {
return "", err
}
var best string
err = walkMIME(m.Header.Get("Content-Type"), m.Header.Get("Content-Transfer-Encoding"), m.Body, 0,
func(ctype, _ string, params map[string]string, body []byte) {
if best != "" {
return
}
switch ctype {
case "text/html":
best, _ = htmlFrom(strings.NewReader(decodeCharset(body, params["charset"])))
}
})
return best, err
}
// walkMIME calls leaf for every non-multipart part of a MIME body, decoded
// from its transfer encoding.
func walkMIME(contentType, cte string, body io.Reader, depth int,
leaf func(ctype, filename string, params map[string]string, body []byte),
) error {
if depth > 8 {
return nil
}
ctype, params, err := mime.ParseMediaType(contentType)
if err != nil {
ctype, params = "text/plain", map[string]string{}
}
if strings.HasPrefix(ctype, "multipart/") {
mr := multipart.NewReader(body, params["boundary"])
for {
p, err := mr.NextRawPart()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
if err := walkMIME(p.Header.Get("Content-Type"), p.Header.Get("Content-Transfer-Encoding"), p, depth+1,
func(ct, fn string, pr map[string]string, b []byte) {
if fn == "" {
fn = filenameOf(p.Header.Get("Content-Disposition"))
}
leaf(ct, fn, pr, b)
}); err != nil {
return err
}
}
}
data, err := io.ReadAll(transferDecoder(cte, body))
if err != nil {
return err
}
leaf(ctype, params["name"], params, data)
return nil
}
func filenameOf(disposition string) string {
_, p, err := mime.ParseMediaType(disposition)
if err != nil {
return ""
}
return p["filename"]
}
+172
View File
@@ -0,0 +1,172 @@
package extract
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"path"
"regexp"
"strings"
"github.com/ledongthuc/pdf"
)
// epubText reads an e-book: its chapters in reading order (the spine).
func epubText(b []byte) (string, error) {
zr, err := openZip(b)
if err != nil {
return "", err
}
container, _ := member(zr, "META-INF/container.xml")
if container == nil {
return "", errors.New("not an EPUB: no META-INF/container.xml")
}
var opf string
_ = walkXML(container, func(e xmlEvent) {
if e.start && e.name == "rootfile" && opf == "" {
opf = e.get("full-path")
}
})
pkg, _ := member(zr, opf)
if pkg == nil {
return "", fmt.Errorf("not an EPUB: no package document %q", opf)
}
dir := path.Dir(opf)
items := map[string]string{}
var spine []string
_ = walkXML(pkg, func(e xmlEvent) {
switch {
case e.start && e.name == "item":
items[e.get("id")] = path.Clean(path.Join(dir, e.get("href")))
case e.start && e.name == "itemref":
spine = append(spine, e.get("idref"))
}
})
var all strings.Builder
for _, id := range spine {
page, _ := member(zr, items[id])
if page == nil {
continue
}
t, err := htmlText(page)
if err != nil {
continue
}
all.WriteString(t)
all.WriteString("\n\n")
}
return all.String(), nil
}
// notebookText reads a Jupyter notebook: Markdown cells as they are, code
// cells fenced; outputs left out, since they are mostly numbers and images.
func notebookText(b []byte) (string, error) {
var nb struct {
Cells []struct {
CellType string `json:"cell_type"`
Source json.RawMessage `json:"source"`
} `json:"cells"`
Metadata struct {
Kernel struct {
Language string `json:"language"`
} `json:"kernelspec"`
} `json:"metadata"`
}
if err := json.Unmarshal(b, &nb); err != nil {
return "", err
}
var out strings.Builder
for _, c := range nb.Cells {
src := jsonText(c.Source)
if strings.TrimSpace(src) == "" {
continue
}
switch c.CellType {
case "code":
out.WriteString("```" + nb.Metadata.Kernel.Language + "\n" + strings.TrimRight(src, "\n") + "\n```\n\n")
default:
out.WriteString(strings.TrimRight(src, "\n") + "\n\n")
}
}
return out.String(), nil
}
// jsonText is a notebook source: a string, or a list of lines.
func jsonText(raw json.RawMessage) string {
var s string
if json.Unmarshal(raw, &s) == nil {
return s
}
var lines []string
if json.Unmarshal(raw, &lines) == nil {
return strings.Join(lines, "")
}
return ""
}
var (
subtitleTiming = regexp.MustCompile(`^\d{1,2}:\d{2}(:\d{2})?[.,]\d{3}\s+-->`)
subtitleIndex = regexp.MustCompile(`^\d+$`)
subtitleTag = regexp.MustCompile(`<[^>]+>`)
)
// subtitleText reads SRT and WebVTT: the spoken lines, without numbers,
// timings and styling.
func subtitleText(b []byte) (string, error) {
var out strings.Builder
last := ""
for _, line := range strings.Split(decodeText(b), "\n") {
line = strings.TrimSpace(line)
switch {
case line == "", line == "WEBVTT", strings.HasPrefix(line, "NOTE"),
subtitleIndex.MatchString(line), subtitleTiming.MatchString(line):
continue
}
line = strings.TrimSpace(subtitleTag.ReplaceAllString(line, ""))
// Captions repeat a line across cues as it scrolls.
if line == "" || line == last {
continue
}
last = line
out.WriteString(line + "\n")
}
return out.String(), nil
}
// pdfText reads a PDF's text layer, page by page, rows in reading order.
// A scanned PDF has none; pdfWithHelpers is then tried.
func pdfText(b []byte) (text string, err error) {
// The reader panics on some malformed files rather than failing; a bad
// PDF is one skipped file, not a stopped sync.
defer func() {
if r := recover(); r != nil {
text, err = "", fmt.Errorf("the PDF could not be parsed: %v", r)
}
}()
r, err := pdf.NewReader(bytes.NewReader(b), int64(len(b)))
if err != nil {
return "", err
}
var o out
for i := 1; i <= r.NumPage(); i++ {
p := r.Page(i)
if p.V.IsNull() {
continue
}
rows, err := p.GetTextByRow()
if err != nil {
continue
}
for _, row := range rows {
var line strings.Builder
for _, w := range row.Content {
line.WriteString(w.S)
}
o.text(line.String())
o.newline()
}
o.para()
}
return o.String(), nil
}
+117
View File
@@ -0,0 +1,117 @@
package extract
import (
"errors"
"fmt"
"strconv"
)
// odfText reads an OpenDocument file (LibreOffice's ODT, ODS, ODP): its
// content.xml.
func odfText(b []byte) (string, error) {
zr, err := openZip(b)
if err != nil {
return "", err
}
content, err := member(zr, "content.xml")
if err != nil {
return "", err
}
if content == nil {
return "", errors.New("not an OpenDocument file: no content.xml")
}
return odfXML(content)
}
// flatODFText reads the single-file XML form (.fodt, .fods, .fodp).
func flatODFText(b []byte) (string, error) { return odfXML(b) }
// odfXML reads OpenDocument's text model, which all three kinds share:
// text:h with an outline level is a heading, text:p a paragraph, list items,
// table rows and cells, slides (draw:page) and sheets (table:table) named as
// headings.
func odfXML(b []byte) (string, error) {
var o out
skip, slide := 0, 0
// Paragraphs inside a cell or a list item do not end the row or the list.
inCell, inItem := 0, 0
err := walkXML(b, func(e xmlEvent) {
switch {
case e.start:
switch e.name {
case "tracked-changes", "notes-configuration", "annotation", "frame-name":
skip++
case "h":
lvl, _ := strconv.Atoi(e.get("outline-level"))
o.heading(max(lvl, 1))
case "list-item":
inItem++
o.item()
case "table-cell":
inCell++
case "tab":
o.raw("\t")
case "s":
n, _ := strconv.Atoi(e.get("c"))
for range max(n, 1) {
o.raw(" ")
}
case "line-break":
o.newline()
case "page":
slide++
o.heading(2)
name := e.get("name")
if name == "" || isDefaultPageName(name) {
name = fmt.Sprintf("Slide %d", slide)
}
o.text(name)
o.para()
case "table":
if n := e.get("name"); n != "" {
o.heading(2)
o.text(n)
o.para()
}
}
case e.end:
switch e.name {
case "tracked-changes", "notes-configuration", "annotation", "frame-name":
skip--
case "h", "p":
switch {
case inCell > 0:
o.text(" ")
case inItem > 0:
o.newline()
default:
o.para()
}
case "list-item":
inItem--
o.newline()
case "table-cell":
inCell--
o.tab()
case "table-row":
o.newline()
case "table", "page":
o.para()
}
default:
if skip == 0 {
o.raw(e.text)
}
}
})
return o.String(), err
}
// isDefaultPageName is LibreOffice's own name for an unnamed slide, "page1".
func isDefaultPageName(s string) bool {
if len(s) < 5 || s[:4] != "page" {
return false
}
_, err := strconv.Atoi(s[4:])
return err == nil
}
+463
View File
@@ -0,0 +1,463 @@
package extract
import (
"archive/zip"
"bytes"
"encoding/xml"
"errors"
"fmt"
"io"
"path"
"regexp"
"sort"
"strconv"
"strings"
)
// maxMember is the most one member of an Office file or EPUB is inflated
// to: a zip bomb stops here rather than at the machine's memory.
const maxMember = 512 << 20
func openZip(b []byte) (*zip.Reader, error) {
return zip.NewReader(bytes.NewReader(b), int64(len(b)))
}
// member reads one file inside a zip, by its exact name; nil when absent.
func member(zr *zip.Reader, name string) ([]byte, error) {
for _, f := range zr.File {
if f.Name == name {
return readMember(f)
}
}
return nil, nil
}
func readMember(f *zip.File) ([]byte, error) {
rc, err := f.Open()
if err != nil {
return nil, err
}
defer rc.Close()
b, err := io.ReadAll(io.LimitReader(rc, maxMember+1))
if err != nil {
return nil, err
}
if len(b) > maxMember {
return nil, fmt.Errorf("%s inflates past %d MiB", f.Name, maxMember>>20)
}
return b, nil
}
// relID is an element's r:id, the namespaced id that names a relationship —
// not its own plain id, which sldId has as well.
func (e xmlEvent) relID() string {
for _, a := range e.attr {
if a.Name.Local == "id" && a.Name.Space != "" {
return a.Value
}
}
return ""
}
// xmlEvent is one step of walking an XML document: a start, an end, or
// character data, with the element's local name and attributes.
type xmlEvent struct {
start, end bool
name string
attr []xml.Attr
text string
}
func (e xmlEvent) get(local string) string {
for _, a := range e.attr {
if a.Name.Local == local {
return a.Value
}
}
return ""
}
// walkXML calls f for each element start and end and each run of character
// data. Namespaces are dropped: OOXML and ODF are read by local names.
func walkXML(b []byte, f func(xmlEvent)) error {
d := xml.NewDecoder(bytes.NewReader(b))
d.Strict = false
d.CharsetReader = func(cs string, in io.Reader) (io.Reader, error) {
all, err := io.ReadAll(in)
if err != nil {
return nil, err
}
return strings.NewReader(decodeCharset(all, cs)), nil
}
for {
tok, err := d.Token()
if errors.Is(err, io.EOF) {
return nil
}
if err != nil {
return err
}
switch t := tok.(type) {
case xml.StartElement:
f(xmlEvent{start: true, name: t.Name.Local, attr: t.Attr})
case xml.EndElement:
f(xmlEvent{end: true, name: t.Name.Local})
case xml.CharData:
f(xmlEvent{text: string(t)})
}
}
}
// headingLevel reads a heading level out of a paragraph style: Word's
// "Heading1", "Heading 2", "heading3", its Turkish "Balk1" (the id Word gives
// "Başlık 1"), "Title" and "Subtitle".
func headingLevel(style string) int {
s := strings.ToLower(strings.ReplaceAll(style, " ", ""))
switch s {
case "title", "konubaşlığı", "konubal":
return 1
case "subtitle", "altkonubaşlığı", "altkonubal":
return 2
}
for _, p := range []string{"heading", "balk", "başlık", "berschrift", "titre", "título", "titolo", "kop"} {
if rest, ok := strings.CutPrefix(s, p); ok {
if n, err := strconv.Atoi(rest); err == nil && n >= 1 && n <= 9 {
return min(n, 6)
}
}
}
return 0
}
// docxText reads a Word document: the body, then footnotes and endnotes.
// Paragraph styles give headings; tables are rows of tab-separated cells.
func docxText(b []byte) (string, error) {
zr, err := openZip(b)
if err != nil {
return "", err
}
body, err := member(zr, "word/document.xml")
if err != nil {
return "", err
}
if body == nil {
return "", errors.New("not a Word document: no word/document.xml")
}
var o out
if err := wordML(body, &o); err != nil {
return "", err
}
for _, extra := range []string{"word/footnotes.xml", "word/endnotes.xml"} {
if x, _ := member(zr, extra); x != nil {
_ = wordML(x, &o)
}
}
return o.String(), nil
}
func wordML(b []byte, o *out) error {
inText, inDel, inCell := false, 0, 0
level, list := 0, false
return walkXML(b, func(e xmlEvent) {
switch {
case e.start:
switch e.name {
case "p":
level, list = 0, false
case "pStyle":
level = headingLevel(e.get("val"))
if strings.Contains(strings.ToLower(e.get("val")), "list") {
list = true
}
case "numPr":
list = true
case "t":
inText = true
if level > 0 && inCell == 0 && o.line.Len() == 0 && o.prefix == "" {
o.heading(level)
} else if list && inCell == 0 && o.line.Len() == 0 && o.prefix == "" {
o.item()
}
case "delText":
// Tracked deletions are not the document's text.
inDel++
case "tc":
inCell++
case "tab":
o.raw("\t")
case "br", "cr":
o.newline()
}
case e.end:
switch e.name {
case "t":
inText = false
case "delText":
inDel--
case "p":
switch {
case inCell > 0:
// A cell's paragraphs stay in its row.
o.text(" ")
case list:
o.newline()
default:
o.para()
}
case "tc":
inCell--
o.tab()
case "tr":
o.newline()
case "tbl":
o.para()
}
default:
if inText && inDel == 0 {
o.raw(e.text)
}
}
})
}
// xlsxText reads a workbook: each sheet under its name as a heading, each
// row a line of tab-separated cells, shared strings resolved.
func xlsxText(b []byte) (string, error) {
zr, err := openZip(b)
if err != nil {
return "", err
}
var shared []string
if ss, _ := member(zr, "xl/sharedStrings.xml"); ss != nil {
var cur strings.Builder
inT, inRPh := false, 0
_ = walkXML(ss, func(e xmlEvent) {
switch {
case e.start && e.name == "si":
cur.Reset()
case e.end && e.name == "si":
shared = append(shared, cur.String())
case e.start && e.name == "rPh":
inRPh++ // phonetic guides, not the cell's text
case e.end && e.name == "rPh":
inRPh--
case e.start && e.name == "t":
inT = true
case e.end && e.name == "t":
inT = false
case !e.start && !e.end && inT && inRPh == 0:
cur.WriteString(e.text)
}
})
}
sheets, err := xlsxSheets(zr)
if err != nil {
return "", err
}
var o out
for _, sh := range sheets {
data, err := member(zr, sh.path)
if err != nil || data == nil {
continue
}
o.heading(2)
o.text(sh.name)
o.para()
var cell strings.Builder
cellType, inV, inIs := "", false, false
first := true
_ = walkXML(data, func(e xmlEvent) {
switch {
case e.start && e.name == "row":
first = true
case e.end && e.name == "row":
o.newline()
case e.start && e.name == "c":
cellType = e.get("t")
cell.Reset()
case e.end && e.name == "c":
v := cell.String()
if cellType == "s" {
if i, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && i >= 0 && i < len(shared) {
v = shared[i]
}
}
v = strings.TrimSpace(strings.ReplaceAll(v, "\n", " "))
if v != "" {
if !first {
o.tab()
}
o.text(v)
first = false
}
case e.start && e.name == "v":
inV = true
case e.end && e.name == "v":
inV = false
case e.start && e.name == "is":
inIs = true
case e.end && e.name == "is":
inIs = false
case !e.start && !e.end && (inV || inIs):
cell.WriteString(e.text)
}
})
o.para()
}
return o.String(), nil
}
type sheetRef struct{ name, path string }
// xlsxSheets lists a workbook's sheets in their order, with the part each is
// kept in.
func xlsxSheets(zr *zip.Reader) ([]sheetRef, error) {
wb, err := member(zr, "xl/workbook.xml")
if err != nil || wb == nil {
return nil, errors.New("not a workbook: no xl/workbook.xml")
}
rels := map[string]string{}
if r, _ := member(zr, "xl/_rels/workbook.xml.rels"); r != nil {
_ = walkXML(r, func(e xmlEvent) {
if e.start && e.name == "Relationship" {
rels[e.get("Id")] = e.get("Target")
}
})
}
var out []sheetRef
_ = walkXML(wb, func(e xmlEvent) {
if !e.start || e.name != "sheet" {
return
}
target := rels[e.relID()]
if target == "" {
return
}
p := path.Clean(path.Join("xl", target))
if strings.HasPrefix(target, "/") {
p = strings.TrimPrefix(target, "/")
}
out = append(out, sheetRef{name: e.get("name"), path: p})
})
return out, nil
}
var slideNum = regexp.MustCompile(`(\d+)\.xml$`)
func numbered(zr *zip.Reader, prefix string) []*zip.File {
var fs []*zip.File
for _, f := range zr.File {
if strings.HasPrefix(f.Name, prefix) && slideNum.MatchString(f.Name) && !strings.Contains(f.Name[len(prefix):], "/") {
fs = append(fs, f)
}
}
n := func(f *zip.File) int {
v, _ := strconv.Atoi(slideNum.FindStringSubmatch(f.Name)[1])
return v
}
sort.Slice(fs, func(i, j int) bool { return n(fs[i]) < n(fs[j]) })
return fs
}
// rels reads a part's relationships: id → target, and type → target, with
// targets resolved against the part's directory.
func rels(zr *zip.Reader, part string) (byID, byType map[string]string) {
byID, byType = map[string]string{}, map[string]string{}
dir, base := path.Split(part)
r, _ := member(zr, dir+"_rels/"+base+".rels")
if r == nil {
return byID, byType
}
_ = walkXML(r, func(e xmlEvent) {
if !e.start || e.name != "Relationship" || e.get("TargetMode") == "External" {
return
}
t := e.get("Target")
if strings.HasPrefix(t, "/") {
t = strings.TrimPrefix(t, "/")
} else {
t = path.Clean(path.Join(dir, t))
}
byID[e.get("Id")] = t
ty := e.get("Type")
byType[ty[strings.LastIndex(ty, "/")+1:]] = t
})
return byID, byType
}
// pptxText reads a presentation: each slide, in the presentation's order, as
// a heading, its text, and its speaker notes.
func pptxText(b []byte) (string, error) {
zr, err := openZip(b)
if err != nil {
return "", err
}
var order []string
if pres, _ := member(zr, "ppt/presentation.xml"); pres != nil {
ids, _ := rels(zr, "ppt/presentation.xml")
_ = walkXML(pres, func(e xmlEvent) {
if e.start && e.name == "sldId" {
if t := ids[e.relID()]; t != "" {
order = append(order, t)
}
}
})
}
if len(order) == 0 {
for _, f := range numbered(zr, "ppt/slides/slide") {
order = append(order, f.Name)
}
}
if len(order) == 0 {
return "", errors.New("not a presentation: no slides")
}
var o out
drawing := func(d []byte) {
inT, inField := false, 0
_ = walkXML(d, func(e xmlEvent) {
switch {
case e.start && e.name == "fld":
// Slide numbers and dates, not the slide's text.
inField++
case e.end && e.name == "fld":
inField--
case e.start && e.name == "t":
inT = true
case e.end && e.name == "t":
inT = false
case e.end && e.name == "p":
o.newline()
case e.start && e.name == "br":
o.newline()
case e.end && e.name == "sp":
o.para()
case e.end && e.name == "tc":
o.tab()
case e.end && e.name == "tr":
o.newline()
case !e.start && !e.end && inT && inField == 0:
o.raw(e.text)
}
})
}
for i, part := range order {
d, err := member(zr, part)
if err != nil || d == nil {
continue
}
o.heading(2)
o.text(fmt.Sprintf("Slide %d", i+1))
o.para()
drawing(d)
if _, byType := rels(zr, part); byType["notesSlide"] != "" {
if nd, _ := member(zr, byType["notesSlide"]); nd != nil {
o.para()
o.text("Notes:")
o.newline()
drawing(nd)
}
}
o.para()
}
return o.String(), nil
}
+207
View File
@@ -0,0 +1,207 @@
package extract
import (
"errors"
"strconv"
"strings"
"unicode/utf16"
"golang.org/x/text/encoding/charmap"
"golang.org/x/text/unicode/norm"
)
// rtfSkip are destinations whose content is not the document's text.
var rtfSkip = map[string]bool{
"fonttbl": true, "colortbl": true, "stylesheet": true, "info": true, "pict": true,
"header": true, "headerl": true, "headerr": true, "headerf": true, "footer": true,
"footerl": true, "footerr": true, "footerf": true, "object": true, "xmlnstbl": true,
"listtable": true, "listoverridetable": true, "rsidtbl": true, "generator": true,
"themedata": true, "colorschememapping": true, "latentstyles": true, "datastore": true,
"filetbl": true, "revtbl": true, "pgdsctbl": true, "fldinst": true, "bkmkstart": true,
"bkmkend": true, "shppict": true, "nonshppict": true, "datafield": true, "blipuid": true,
}
// rtfText reads Rich Text Format: text, paragraphs, tabs and table cells,
// characters in the document's code page (\ansicpg) or as \u escapes;
// font tables, pictures, headers and other destinations skipped.
func rtfText(b []byte) (string, error) {
s := string(b)
if !strings.HasPrefix(strings.TrimSpace(s), "{\\rtf") {
return "", errors.New("not RTF: no {\\rtf header")
}
type state struct {
skip bool
uc int // how many fallback characters follow a \u
}
cp := charmap.Windows1252
stack := []state{{uc: 1}}
var o out
var pending []byte // \'hh bytes, decoded in runs so a code page's pairs stay together
var surrogate rune
flush := func() {
if len(pending) == 0 {
return
}
d, _ := cp.NewDecoder().Bytes(pending)
o.raw(string(d))
pending = pending[:0]
}
skipChars := 0
cur := func() *state { return &stack[len(stack)-1] }
for i := 0; i < len(s); i++ {
c := s[i]
switch c {
case '{':
flush()
stack = append(stack, *cur())
case '}':
flush()
if len(stack) > 1 {
stack = stack[:len(stack)-1]
}
case '\\':
if i+1 >= len(s) {
break
}
n := s[i+1]
switch {
case n == '\'' && i+3 < len(s):
v, err := strconv.ParseUint(s[i+2:i+4], 16, 8)
i += 3
if err != nil || cur().skip {
continue
}
if skipChars > 0 {
skipChars--
continue
}
pending = append(pending, byte(v))
case n == '*':
i++
cur().skip = true
case n == '\\' || n == '{' || n == '}':
i++
if !cur().skip {
flush()
o.raw(string(n))
}
case n == '~':
i++
if !cur().skip {
o.raw(" ")
}
case n == '-' || n == '_':
i++
case n == '\n' || n == '\r':
i++
if !cur().skip {
flush()
o.para()
}
case isLetter(n):
j := i + 1
for j < len(s) && isLetter(s[j]) {
j++
}
word := s[i+1 : j]
k := j
if k < len(s) && (s[k] == '-' || (s[k] >= '0' && s[k] <= '9')) {
k++
for k < len(s) && s[k] >= '0' && s[k] <= '9' {
k++
}
}
arg, hasArg := 0, k > j
if hasArg {
arg, _ = strconv.Atoi(s[j:k])
}
if k < len(s) && s[k] == ' ' {
k++
}
i = k - 1
flush()
if rtfSkip[word] {
cur().skip = true
continue
}
switch word {
case "ansicpg":
cp = codePage(arg)
case "uc":
cur().uc = arg
case "u":
if cur().skip {
continue
}
r := rune(arg)
if r < 0 {
r += 65536
}
switch {
case utf16.IsSurrogate(r) && r < 0xDC00:
surrogate = r
case utf16.IsSurrogate(r):
o.raw(string(utf16.DecodeRune(surrogate, r)))
surrogate = 0
default:
o.raw(string(r))
}
skipChars = cur().uc
case "par", "sect", "page":
if !cur().skip {
o.para()
}
case "line":
if !cur().skip {
o.newline()
}
case "tab":
if !cur().skip {
o.raw("\t")
}
case "cell":
if !cur().skip {
o.tab()
}
case "row":
if !cur().skip {
o.newline()
}
case "emdash":
o.raw("—")
case "endash":
o.raw("–")
case "bullet":
o.raw("•")
case "lquote", "rquote":
o.raw("'")
case "ldblquote", "rdblquote":
o.raw("\"")
}
default:
i++
}
case '\r', '\n':
// Line breaks in RTF source are not text.
default:
if cur().skip {
continue
}
if skipChars > 0 {
skipChars--
continue
}
if c >= 0x80 {
// Unescaped 8-bit text, as some writers leave it.
pending = append(pending, c)
continue
}
flush()
o.raw(string(c))
}
}
flush()
return norm.NFC.String(o.String()), nil
}
func isLetter(c byte) bool { return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') }
Binary file not shown.
+95
View File
@@ -0,0 +1,95 @@
package extract
import (
"bytes"
"strings"
"unicode/utf8"
"golang.org/x/net/html/charset"
"golang.org/x/text/encoding/charmap"
"golang.org/x/text/encoding/unicode"
"golang.org/x/text/unicode/norm"
)
func hasUTF16BOM(b []byte) bool {
return len(b) >= 2 && ((b[0] == 0xFF && b[1] == 0xFE) || (b[0] == 0xFE && b[1] == 0xFF))
}
// decodeText turns a text file into UTF-8, NFC.
//
// UTF-8 is taken as it is (its BOM dropped); UTF-16 by its BOM, which is how
// Windows' Notepad writes "Unicode". Anything else is a single-byte code
// page, and which one cannot be known from the bytes; Windows-1254 is
// assumed, since it is what a Turkish Windows writes and it agrees with
// Windows-1252 on every letter but six (ğĞıİşŞ in place of ðÐýÝþÞ) — which
// are rare in the western languages 1252 serves.
func decodeText(b []byte) string {
var s string
switch {
case bytes.HasPrefix(b, []byte{0xEF, 0xBB, 0xBF}):
s = string(b[3:])
case hasUTF16BOM(b):
out, err := unicode.UTF16(unicode.LittleEndian, unicode.ExpectBOM).NewDecoder().Bytes(b)
if err != nil {
return ""
}
s = string(out)
case utf8.Valid(b):
s = string(b)
default:
out, err := charmap.Windows1254.NewDecoder().Bytes(b)
if err != nil {
return ""
}
s = string(out)
}
return norm.NFC.String(strings.ReplaceAll(s, "\r\n", "\n"))
}
// decodeCharset decodes bytes declared to be in a charset, as mail and web
// pages declare theirs, by any of the WHATWG labels; an unknown or missing
// one is treated as decodeText would.
func decodeCharset(b []byte, cs string) string {
cs = strings.ToLower(strings.TrimSpace(cs))
if cs == "" || cs == "utf-8" || cs == "utf8" || cs == "us-ascii" {
return decodeText(b)
}
enc, _ := charset.Lookup(cs)
if enc == nil {
return decodeText(b)
}
out, err := enc.NewDecoder().Bytes(b)
if err != nil {
return decodeText(b)
}
return norm.NFC.String(strings.ReplaceAll(string(out), "\r\n", "\n"))
}
// codePage is the decoder for an RTF \ansicpgN or a Windows code page number.
func codePage(n int) *charmap.Charmap {
switch n {
case 1250:
return charmap.Windows1250
case 1251:
return charmap.Windows1251
case 1252:
return charmap.Windows1252
case 1253:
return charmap.Windows1253
case 1254:
return charmap.Windows1254
case 1255:
return charmap.Windows1255
case 1256:
return charmap.Windows1256
case 1257:
return charmap.Windows1257
case 1258:
return charmap.Windows1258
case 850:
return charmap.CodePage850
case 437:
return charmap.CodePage437
}
return charmap.Windows1252
}