From 3894e8e5fdf581e249660677fa16ca4eeb73f30d Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Fri, 9 Oct 2026 21:36:23 +0200 Subject: [PATCH] Backlog F5: Qwen3-Embedding-4B measured, on llama.cpp and vLLM, and against bge-m3 Co-Authored-By: Claude Opus 5.5 (1M context) --- BACKLOG.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/BACKLOG.md b/BACKLOG.md index 5b7c9f8..65c8fe1 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1085,6 +1085,21 @@ at a third of its CPUs. 0.7) and a larger store machine. - Done: a batch of embeddings checked text by text against the context (branch `embeddings-batch`); it was refused as one over-long prompt. +- Done: `query_prefix` on an embedding model, before questions only. +- Qwen3-Embedding-4B on the RTX 6000 (2026-10-09): llama.cpp Q8 1,575 + tok/s as first set up (32k cache, 16 slots), 2,700 with a 16k cache, + FlashAttention and ubatch 1–2k; f16 no better. vLLM fp16 (`--runner + pooling`) 6,200 direct, 4,700–4,900 through the gateway in the import, + and its vectors match llama.cpp's (cosine 0.9995). The 103,579 chunks + took 3 h. Wanted: an embedding model's GPU profile on vLLM and CPU + profile on llama.cpp, which needs a source per profile (the catalog has + one per model; the import swapped the entry by hand), and an admission + queue limit that suits bulk imports (30 s turned some away). +- Retrieval on 40 questions written from the corpus, document in the top + 5: bge-m3 31/40 (Turkish questions 13/21), Qwen3-Embedding-4B 37/40 + (18/21), with its query prefix 36/40 and first place 31/40 instead of + 28 (MRR 0.738 → 0.789 → 0.838). Next: translating the question with the + chat model, and a reranker. ## D. Facts worth knowing before starting