Backlog F5: where a large knowledge-base import spends its time

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-09 15:19:35 +02:00
1 parent 92733317a9
commit 8932502768
1 file changed
+21
+21
View File
@@ -1065,6 +1065,27 @@ training yet; quality gating of a published adapter is by hand (ctl eval).
---
### F5. Large imports (M) — measured 2026-10-09, open
The metallurgy corpus (2,314 files, 108,779 chunks, bge-m3 on the one
GPU) went in at about 25 chunks/s, the same over a phone hotspot and the
LAN. The engine's slots were the limit: llama.cpp `--parallel 4` embeds
four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
at a third of its CPUs.
- An embedding model's profile with many slots by default (`--parallel`
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
longest chunk), or `ctl model serve` choosing that for a model with
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
- Embedding on the gateway's side of a document PUT, so the client sends
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
the text and vector go up again (~11 KB).
- The store next: HNSW inserts took half of one of pg01's two cores; at a
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
0.7) and a larger store machine.
- Done: a batch of embeddings checked text by text against the context
(branch `embeddings-batch`); it was refused as one over-long prompt.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, no auth, `ctlClient` in `internal/cli/ctl.go`.