Training data from a collection: llmgw ctl kb dataset

A chat model of the cluster reads each chunk and writes questions a person
might ask of it with the answers it holds, streamed as NDJSON a chunk at a
time, for people to correct before an adapter is trained on it. A chunk that
gave no pairs says so on its own line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 20:04:37 +02:00
1 parent 0a6a586dfe
commit ae95ac8261
8 files changed
+455

No files matched your search

+5
View File
@@ -116,6 +116,11 @@ func (c *ctlClient) do(ctx context.Context, method, path string, body, out any)
if out == nil {
return nil
}
// A stream, as it comes: NDJSON a long run writes line by line.
if w, ok := out.(io.Writer); ok {
_, err := io.Copy(w, resp.Body)
return err
}
// An older control plane answers a path it does not know with its
// dashboard: a page, which read as JSON said only "invalid character '<'".
if strings.HasPrefix(resp.Header.Get("Content-Type"), "text/html") {