Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
73091d7c97 | ||
|
|
b8ed368e31 | ||
|
|
29d7f54cfd | ||
|
|
cacaa35c3f |
No files matched your search
+22
-7
@@ -35,14 +35,16 @@ What is left, most needed first. The detail is in the item each names.
|
||||
146's address names was given 152's; what was added meanwhile reached 146 when it came back; a removal was
|
||||
carried both ways. Found and fixed there: add-peer on a mirror machine of its own, files without checksums never
|
||||
copied, and `mirror remove` neither recorded nor seen by the running mirror.
|
||||
6. **F5**: large imports limited by the embedding engine's slots. (M)
|
||||
6. **F5**: large imports. Done 2026-10-29: the gateway embeds what `llmgw-kb` sends as text; the slots and
|
||||
halfvec were done before. Left: a larger store machine before a million chunks (the operator's), and the
|
||||
embedding model on a GPU for a real import.
|
||||
|
||||
**From Kong** (section L): L1–L5 and L9 done (v0.15.0); left of L4 is routing by cost between outside
|
||||
providers, of L5 cache-write and audio prices. L6 and L7 are llmgw-80's; L8 (A2A) when asked.
|
||||
|
||||
**Models we can trust** (section M, 2026-10-25): M1 done (pickles refused; the explorer lists only what this
|
||||
cluster's hardware holds, with CPU and GPU tok/s). Next M2 (every download pinned and checked against Hugging
|
||||
Face), then M5 (engines without the internet), M3, M4, M6.
|
||||
Face, done 2026-10-30), then M5 (engines without the internet), M3, M4, M6.
|
||||
|
||||
**Governance.**
|
||||
7. **C1**: a tenant-admin role, and forms hidden from viewers.
|
||||
@@ -1128,13 +1130,19 @@ four texts per pass, while llmgw-kb sends 32 a request from 6 workers. The
|
||||
usage rows show 4.6 s a request on average, 1.8 s of it in the gateway's
|
||||
queue and 0.04 s in the guard. The GPU sat at 18–50% and the store machine
|
||||
at a third of its CPUs.
|
||||
- An embedding model's profile with many slots by default (`--parallel`
|
||||
- Done for `ctl model serve` (32 slots of 1024 tokens; `ctl model context
|
||||
MODEL 1024 --slots 32` resizes one served before): an embedding model's profile with many slots by default (`--parallel`
|
||||
32–64, `--ctx-size` to match, `--batch-size`/`--ubatch-size` at least the
|
||||
longest chunk), or `ctl model serve` choosing that for a model with
|
||||
`capabilities.embeddings`. With the default 512, one 516-token chunk failed.
|
||||
- Embedding on the gateway's side of a document PUT, so the client sends
|
||||
text only: today an embeddings answer is ~21 KB of JSON a chunk, and
|
||||
the text and vector go up again (~11 KB).
|
||||
- Done (2026-10-29): embedding on the gateway's side of a document PUT.
|
||||
`llmgw-kb` sends text only to a gateway that says `server_embeds`; the
|
||||
gateway embeds through its own `/v1/embeddings` as the caller, 32 a batch,
|
||||
four batches at once, waiting out a busy engine. On 40 DTIC files the
|
||||
upload went from 25.7 to 3.5 MiB, and the ~21 KB a chunk that came down
|
||||
is gone; on 6 files both ways gave the same 32 chunks and the same search
|
||||
answer. Speed is the engine's either way: the PoC's bge-m3 on node02's two
|
||||
CPU cores embeds ~0.2 chunks/s, so a real import still wants it on a GPU.
|
||||
- The store next: HNSW inserts took half of one of pg01's two cores; at a
|
||||
million chunks the vectors alone outgrow its 4 GB. halfvec (pgvector ≥
|
||||
0.7) and a larger store machine.
|
||||
@@ -1815,7 +1823,14 @@ pickle files. `allow_pickle: true` on a model lets it through, on purpose. On th
|
||||
PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that
|
||||
needs it.
|
||||
|
||||
### M2. Every download pinned and checked against Hugging Face (M)
|
||||
### M2. Every download pinned and checked against Hugging Face (M) — done 2026-10-30
|
||||
Done: what is added from Hugging Face is pinned (the file's commit URL, or
|
||||
`--revision` for vLLM/SGLang) with each file's sha256 in the origin; the node
|
||||
refuses and deletes a copy that is not it, naming both, and keeps the
|
||||
checksum beside a cached file; the mirror checks what `ctl model serve`
|
||||
fetches, `mirror add` a Hugging Face URL. Left: models declared in the
|
||||
settings are not pinned, the admin page's and the console's mirror fetches
|
||||
send no sha256, and adapters by hub id are not pinned.
|
||||
- `hf:org/repo/file` follows `main` and is checked by nothing; a bare
|
||||
`org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is.
|
||||
Both: resolve to the commit when the model is added, keep it in the
|
||||
|
||||
@@ -25,6 +25,28 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **The gateway embeds the documents `llmgw-kb` sends.** A document may be sent as its passages' text alone; the
|
||||
gateway embeds them through its own `/v1/embeddings` as the caller, 32 at a time and four batches at once,
|
||||
waiting out a busy engine rather than failing the document. About a kilobyte a passage leaves the machine instead
|
||||
of a vector down and up again, some thirty times more. `llmgw-kb` does so with a gateway that says it can
|
||||
(`server_embeds` in a collection's answer); `--embed-here` keeps the old way.
|
||||
- **Weights pinned to a commit, and checked against what Hugging Face published.** A model added from Hugging Face
|
||||
(`hf:`, a resolve URL on a branch, or a repository vLLM or SGLang downloads) is pinned to the commit it is at
|
||||
then — the file's URL at that commit, or `--revision` — and each file's published sha256 goes in its origin. A node
|
||||
refuses and deletes a copy that is not it, naming both checksums; the mirror does the same for what `ctl model
|
||||
serve` fetches, and `llmgw mirror add` for a Hugging Face URL (`--sha256` for others). Nodes need this version to
|
||||
check; older ones fetch as before.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **A name with a slash in it, asked through a standby control plane.** `llmgw ctl kb delete Demo/hr` on a standby
|
||||
was answered 405: the standby passed the request on to the leader with `Demo%2Fhr` unescaped, two segments instead
|
||||
of one. The path now reaches the leader as it was sent.
|
||||
|
||||
## v0.15.3 — 2026-10-29
|
||||
|
||||
A model stopped and kept, or removed with its files; the second mirror fixed where it was tried.
|
||||
|
||||
@@ -3849,7 +3849,11 @@ type EngineProfile struct {
|
||||
KvEvents bool `protobuf:"varint,13,opt,name=kv_events,json=kvEvents,proto3" json:"kv_events,omitempty"`
|
||||
// allow_pickle lets the node fetch weights in a pickle format, which run
|
||||
// code when loaded: the catalog's allow_pickle, set on purpose.
|
||||
AllowPickle bool `protobuf:"varint,14,opt,name=allow_pickle,json=allowPickle,proto3" json:"allow_pickle,omitempty"`
|
||||
AllowPickle bool `protobuf:"varint,14,opt,name=allow_pickle,json=allowPickle,proto3" json:"allow_pickle,omitempty"`
|
||||
// model_sha256 is the sha256 the registry published for the file
|
||||
// model_path names, from the catalog's origin: the node refuses a copy
|
||||
// that is not it. Empty when nothing is known to check against.
|
||||
ModelSha256 string `protobuf:"bytes,15,opt,name=model_sha256,json=modelSha256,proto3" json:"model_sha256,omitempty"`
|
||||
unknownFields protoimpl.UnknownFields
|
||||
sizeCache protoimpl.SizeCache
|
||||
}
|
||||
@@ -3982,6 +3986,13 @@ func (x *EngineProfile) GetAllowPickle() bool {
|
||||
return false
|
||||
}
|
||||
|
||||
func (x *EngineProfile) GetModelSha256() string {
|
||||
if x != nil {
|
||||
return x.ModelSha256
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// RemoveWeights deletes cached model files a node no longer needs.
|
||||
//
|
||||
// The paths are computed by the control plane from the catalog, but the agent
|
||||
@@ -5536,7 +5547,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" +
|
||||
"\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" +
|
||||
"\vcpu_threads\x18\x11 \x01(\x05R\n" +
|
||||
"cpuThreads\"\xf0\x03\n" +
|
||||
"cpuThreads\"\x93\x04\n" +
|
||||
"\rEngineProfile\x12\x12\n" +
|
||||
"\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" +
|
||||
"\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" +
|
||||
@@ -5553,7 +5564,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" +
|
||||
"\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" +
|
||||
"\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x12\x1b\n" +
|
||||
"\tkv_events\x18\r \x01(\bR\bkvEvents\x12!\n" +
|
||||
"\fallow_pickle\x18\x0e \x01(\bR\vallowPickle\x1a6\n" +
|
||||
"\fallow_pickle\x18\x0e \x01(\bR\vallowPickle\x12!\n" +
|
||||
"\fmodel_sha256\x18\x0f \x01(\tR\vmodelSha256\x1a6\n" +
|
||||
"\bEnvEntry\x12\x10\n" +
|
||||
"\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" +
|
||||
"\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" +
|
||||
|
||||
@@ -767,6 +767,10 @@ message EngineProfile {
|
||||
// allow_pickle lets the node fetch weights in a pickle format, which run
|
||||
// code when loaded: the catalog's allow_pickle, set on purpose.
|
||||
bool allow_pickle = 14;
|
||||
// model_sha256 is the sha256 the registry published for the file
|
||||
// model_path names, from the catalog's origin: the node refuses a copy
|
||||
// that is not it. Empty when nothing is known to check against.
|
||||
string model_sha256 = 15;
|
||||
}
|
||||
|
||||
// RemoveWeights deletes cached model files a node no longer needs.
|
||||
|
||||
@@ -331,6 +331,8 @@ hidden files, version control folders and Office lock files are not.`,
|
||||
f.BoolVar(&o.Delete, "delete", false, "remove documents this machine sent whose files are gone")
|
||||
f.BoolVar(&o.Retry, "retry", false, "read again files skipped before")
|
||||
f.IntVar(&o.Workers, "workers", 4, "files read and sent at once")
|
||||
f.BoolVar(&o.EmbedHere, "embed-here", false,
|
||||
"embed on this machine and send the vectors, even to a gateway that embeds text it is sent")
|
||||
f.StringVar(&ocr, "ocr-languages", "tur+eng", "Tesseract's languages for images and scanned pages")
|
||||
f.Int64Var(&maxMiB, "max-size", 256, "skip files larger than this many MiB")
|
||||
f.BoolVarP(&verbose, "verbose", "v", false, "list unchanged files too")
|
||||
|
||||
@@ -2890,6 +2890,10 @@
|
||||
"writable": {
|
||||
"type": "boolean",
|
||||
"description": "Writable is whether this caller may add and remove documents."
|
||||
},
|
||||
"server_embeds": {
|
||||
"type": "boolean",
|
||||
"description": "ServerEmbeds says a document may be sent as text alone: the gateway embeds its chunks, as the caller, with the collection's model."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -897,6 +897,24 @@ the node — an adapter given by its id, a TensorRT-LLM model's — is refused
|
||||
when the model is added, unless it says `allow_pickle`: mirror its
|
||||
safetensors and give that instead.
|
||||
|
||||
**Weights are pinned to a commit and checked.** A model added with
|
||||
`ctl model add`, the API or the console, whose weights come from Hugging Face,
|
||||
is pinned to the commit the repository is at when it is added:
|
||||
|
||||
- `hf:org/repo/file` and a `…/resolve/main/…` URL become that commit's URL;
|
||||
- a repository vLLM or SGLang downloads itself is given `--revision`;
|
||||
- the sha256 Hugging Face published for each file goes in the model's origin
|
||||
(its definition, `GET /api/v1/models/NAME`, shows it).
|
||||
|
||||
A node then refuses a copy of the file that is not it, naming both checksums,
|
||||
and deletes it; a cached copy is checked once and its checksum kept beside it.
|
||||
The mirror does the same for a model `ctl model serve` fetches, and
|
||||
`llmgw mirror add` for a Hugging Face URL (`--sha256` for any other). A new
|
||||
upload to the repository reaches the cluster only through `ctl model update`,
|
||||
which says what changed. Not pinned: models declared in the cluster's settings,
|
||||
which are not added this way, and a vLLM model whose arguments already say
|
||||
`--revision` keeps that one.
|
||||
|
||||
Put them where the node can read them. `scp`, a shared filesystem, or the
|
||||
engine's own downloader are all valid, and which you want depends on how many
|
||||
nodes you have.
|
||||
|
||||
+16
-14
@@ -3,9 +3,9 @@
|
||||
A knowledge base is a collection of a company's documents that the cluster
|
||||
can answer from. People put their documents into it from their own machines
|
||||
with `llmgw-kb`. The tool reads the files on that machine, cuts them into
|
||||
passages ("chunks"), has the gateway embed each passage with the person's own
|
||||
key, and sends the text and the vectors. The files themselves never leave
|
||||
the machine.
|
||||
passages ("chunks") and sends their text; the gateway embeds each passage
|
||||
with the person's own key and keeps the text and the vectors. The files
|
||||
themselves never leave the machine.
|
||||
|
||||
## Why retrieval, and not training
|
||||
|
||||
@@ -159,10 +159,12 @@ llmgw ctl config set router.admission.bands '[{name: gold, priority: 300, satura
|
||||
llmgw keys create import --project Acme/docs --band bulk
|
||||
```
|
||||
|
||||
The people's questions then go first, and the import waits behind them. A
|
||||
request that waits as long as its band allows is answered 504 with
|
||||
`Retry-After`, and `llmgw-kb` sends it again rather than counting the
|
||||
passage as failed. It gives up after 15 minutes.
|
||||
The people's questions then go first, and the import waits behind them. The
|
||||
gateway embeds a document's passages 32 at a time, four batches at once,
|
||||
enough to fill an embedding replica's slots; a batch told to wait (`Retry-After`)
|
||||
waits up to five minutes before the document is answered the same, and
|
||||
`llmgw-kb` sends it again rather than counting it as failed. It gives up
|
||||
after 15 minutes.
|
||||
|
||||
## For everyone: `llmgw-kb`
|
||||
|
||||
@@ -225,6 +227,7 @@ not changed.
|
||||
| `--include`, `--exclude` | Glob patterns. `*.pdf` matches a file name; `reports/**/*.xlsx` matches a path. |
|
||||
| `--hidden` | Reads hidden files and folders too (left out by default). |
|
||||
| `-v` | Lists unchanged files too. |
|
||||
| `--embed-here` | Embeds the passages from this machine and sends the vectors. With a gateway that does not embed them itself it does so anyway. |
|
||||
|
||||
Documents are named by their path under the folder, with forward slashes,
|
||||
on every system. With several folders, each folder's own name comes first.
|
||||
@@ -259,13 +262,12 @@ scanned page with no OCR.
|
||||
|
||||
### What leaves your machine
|
||||
|
||||
For each new or changed document, two things leave the machine, both over
|
||||
the gateway's TLS connection:
|
||||
|
||||
- **The passages, to be embedded.** They go through `/v1/embeddings` like
|
||||
any request of yours. Your key's limits, budget and guardrails apply.
|
||||
- **The document itself.** Its name, its file's SHA-256 digest and size,
|
||||
and the passages with their vectors.
|
||||
For each new or changed document, one request leaves the machine, over the
|
||||
gateway's TLS connection: its name, its file's SHA-256 digest and size, and
|
||||
its passages' text. The gateway embeds the passages through `/v1/embeddings`
|
||||
as you, so your key's limits, budget and guardrails apply as if you had
|
||||
asked. That is about a kilobyte a passage; embedding here (`--embed-here`)
|
||||
brings each vector down and sends it up again, some thirty times more.
|
||||
|
||||
The file itself is never sent. `--dry-run` shows exactly what would be.
|
||||
|
||||
|
||||
@@ -878,6 +878,7 @@ func (a *Agent) removeWeights(paths []string) {
|
||||
a.log.Warn("could not remove cached weights", "path", p, "err", err)
|
||||
continue
|
||||
}
|
||||
_ = os.Remove(weights.ChecksumPath(abs))
|
||||
a.log.Info("removed cached weights no model asks for", "path", p)
|
||||
}
|
||||
}
|
||||
@@ -1133,7 +1134,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan<
|
||||
},
|
||||
})
|
||||
})
|
||||
if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle); err != nil {
|
||||
if path, err := a.weights().EnsureChecked(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle, spec.Profile.ModelSHA256); err != nil {
|
||||
stop()
|
||||
report(supervisor.Failed, err.Error(), nil)
|
||||
return
|
||||
|
||||
@@ -50,6 +50,10 @@ type Profile struct {
|
||||
// AllowPickle lets the node fetch weights in a pickle format, which run
|
||||
// code when loaded; the catalog's allow_pickle.
|
||||
AllowPickle bool `json:"allow_pickle,omitempty"`
|
||||
// ModelSHA256 is the sha256 the registry published for the file
|
||||
// ModelPath names; the node refuses a copy that is not it. Empty when
|
||||
// nothing is known to check against.
|
||||
ModelSHA256 string `json:"model_sha256,omitempty"`
|
||||
}
|
||||
|
||||
// ShardSpec is one shard of one replica on this node.
|
||||
|
||||
@@ -94,6 +94,13 @@ var ErrNotLocal = errors.New("the weights are not on this node and the source sa
|
||||
// A source in a pickle format, or a set with such a file in it, is refused
|
||||
// before anything is fetched unless allowPickle; see Pickled.
|
||||
func (f *Fetcher) Ensure(ctx context.Context, source string, allowPickle bool) (path string, err error) {
|
||||
return f.EnsureChecked(ctx, source, allowPickle, "")
|
||||
}
|
||||
|
||||
// EnsureChecked is Ensure for a file whose sha256 the registry published: a
|
||||
// copy that is not it, downloaded or cached, is refused rather than served.
|
||||
// An empty sha256 checks nothing.
|
||||
func (f *Fetcher) EnsureChecked(ctx context.Context, source string, allowPickle bool, sha256 string) (path string, err error) {
|
||||
ctx, span := obs.Tracer(tracerName).Start(ctx, "fetch weights",
|
||||
trace.WithAttributes(attribute.String("llmgw.weights.source", redact(source))))
|
||||
defer func() {
|
||||
@@ -103,7 +110,7 @@ func (f *Fetcher) Ensure(ctx context.Context, source string, allowPickle bool) (
|
||||
if !allowPickle && !IsSet(source) && Pickled(source) {
|
||||
return "", refusePickle(source)
|
||||
}
|
||||
return f.ensure(ctx, source, allowPickle)
|
||||
return f.ensure(ctx, source, allowPickle, strings.ToLower(sha256))
|
||||
}
|
||||
|
||||
// tracerName names the spans this package opens.
|
||||
@@ -125,7 +132,7 @@ func redact(source string) string {
|
||||
return u.String()
|
||||
}
|
||||
|
||||
func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (string, error) {
|
||||
func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool, want string) (string, error) {
|
||||
if source == "" {
|
||||
return "", fmt.Errorf("no source given")
|
||||
}
|
||||
@@ -156,8 +163,13 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (
|
||||
return "", err
|
||||
}
|
||||
if _, err := os.Stat(target); err == nil {
|
||||
outcome(ctx, "cached")
|
||||
return target, nil
|
||||
if ok, err := f.cachedIs(target, want); err != nil {
|
||||
return "", err
|
||||
} else if ok {
|
||||
outcome(ctx, "cached")
|
||||
return target, nil
|
||||
}
|
||||
// Not the file the catalog names: fetched again below, and checked.
|
||||
}
|
||||
|
||||
f.mu.Lock()
|
||||
@@ -179,7 +191,7 @@ func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (
|
||||
f.mu.Unlock()
|
||||
|
||||
outcome(ctx, "downloaded")
|
||||
call.path, call.err = f.fetch(ctx, source, target)
|
||||
call.path, call.err = f.fetchChecked(ctx, source, target, want)
|
||||
close(call.done)
|
||||
|
||||
f.mu.Lock()
|
||||
@@ -193,6 +205,80 @@ func (f *Fetcher) fetch(ctx context.Context, source, target string) (string, err
|
||||
return f.fetchAs(ctx, source, source, target)
|
||||
}
|
||||
|
||||
// fetchChecked is fetch for a file whose sha256 is known: a download that
|
||||
// is not it is deleted and refused, naming both, and one that is has its
|
||||
// checksum kept beside it so a cached copy is not read again at each start.
|
||||
func (f *Fetcher) fetchChecked(ctx context.Context, source, target, want string) (string, error) {
|
||||
got, err := f.fetch(ctx, source, target)
|
||||
if err != nil || want == "" {
|
||||
return got, err
|
||||
}
|
||||
sum, err := fileSHA256(target)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if sum != want {
|
||||
_ = os.Remove(target)
|
||||
return "", fmt.Errorf("%s arrived with sha256 %s, and the registry published %s for it: refused and deleted",
|
||||
redact(source), sum, want)
|
||||
}
|
||||
_ = writeChecksum(target, sum)
|
||||
return target, nil
|
||||
}
|
||||
|
||||
// cachedIs says whether a cached file is the one whose sha256 is want: by
|
||||
// the checksum kept beside it, or worked out once and then kept. One that
|
||||
// is not is deleted, to be fetched again. Any file is, when want is empty.
|
||||
func (f *Fetcher) cachedIs(target, want string) (bool, error) {
|
||||
kept := readChecksum(target)
|
||||
if want == "" || kept == want {
|
||||
return true, nil
|
||||
}
|
||||
if kept != "" {
|
||||
// Checked when it was fetched, as another file: the same address
|
||||
// gives the same bytes again, and another model may be serving
|
||||
// these. Refused, and left.
|
||||
return false, fmt.Errorf("the copy of this file here has sha256 %s, as fetched, and the catalog names %s: refused",
|
||||
kept, want)
|
||||
}
|
||||
sum, err := fileSHA256(target)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if sum == want {
|
||||
_ = writeChecksum(target, sum)
|
||||
return true, nil
|
||||
}
|
||||
if f.Log != nil {
|
||||
f.Log.Warn("a cached file is not the one the catalog names; fetching it again",
|
||||
"path", target, "sha256", sum, "want", want)
|
||||
}
|
||||
_ = os.Remove(checksumPath(target))
|
||||
return false, os.Remove(target)
|
||||
}
|
||||
|
||||
// checksumPath is where a cached file's sha256 is kept: beside it, hidden,
|
||||
// so no scan takes it for weights.
|
||||
func checksumPath(file string) string {
|
||||
return filepath.Join(filepath.Dir(file), "."+filepath.Base(file)+".sha256")
|
||||
}
|
||||
|
||||
func writeChecksum(file, sum string) error {
|
||||
return os.WriteFile(checksumPath(file), []byte(sum+"\n"), 0o644) //nolint:gosec // a checksum, for anyone to read
|
||||
}
|
||||
|
||||
func readChecksum(file string) string {
|
||||
b, err := os.ReadFile(checksumPath(file)) //nolint:gosec // beside a file this fetcher wrote
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(b))
|
||||
}
|
||||
|
||||
// ChecksumPath is where the sha256 of a cached file is kept, for whoever
|
||||
// deletes the file to delete with it.
|
||||
func ChecksumPath(file string) string { return checksumPath(file) }
|
||||
|
||||
// fetchAs downloads one URL but reports progress under another name.
|
||||
//
|
||||
// The two differ for a set: the shards waiting on it asked for the version, not
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
package weights
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
@@ -287,3 +289,75 @@ func TestAnAbsolutePathThatIsMissingIsStillAnError(t *testing.T) {
|
||||
t.Errorf("err = %v, want ErrNotLocal", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A file whose sha256 the registry published is served only as that file:
|
||||
// changed upstream after the model was added, it is refused with both
|
||||
// checksums named, and a cached copy that is not it is fetched again.
|
||||
func TestAFileThatIsNotWhatTheRegistryPublishedIsRefused(t *testing.T) {
|
||||
body := "the weights as published"
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte(body))
|
||||
}))
|
||||
defer srv.Close()
|
||||
sum := sha256.Sum256([]byte(body))
|
||||
want := hex.EncodeToString(sum[:])
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
src := srv.URL + "/model.gguf"
|
||||
|
||||
path, err := f.EnsureChecked(t.Context(), src, false, strings.ToUpper(want))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if readChecksum(path) != want {
|
||||
t.Errorf("the checksum was not kept beside it: %q", readChecksum(path))
|
||||
}
|
||||
|
||||
// The cached copy changed under it: fetched again, and the right one kept.
|
||||
if err := os.WriteFile(path, []byte("tampered"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = os.Remove(checksumPath(path))
|
||||
if _, err := f.EnsureChecked(t.Context(), src, false, want); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if b, _ := os.ReadFile(path); string(b) != body {
|
||||
t.Errorf("the cached copy that was not it was served: %q", b)
|
||||
}
|
||||
|
||||
// Upstream changed after the model was added.
|
||||
body = "weights someone replaced"
|
||||
f2 := &Fetcher{CacheDir: t.TempDir()}
|
||||
_, err = f2.EnsureChecked(t.Context(), src, false, want)
|
||||
if err == nil || !strings.Contains(err.Error(), want) || !strings.Contains(err.Error(), "sha256") {
|
||||
t.Fatalf("not refused, or without both checksums: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(CachePathFor(f2.CacheDir, src)); !os.IsNotExist(err) {
|
||||
t.Error("the file that was not it was kept")
|
||||
}
|
||||
// With nothing to check against, it is fetched as before.
|
||||
if _, err := f2.Ensure(t.Context(), src, false); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}
|
||||
|
||||
// A cached file checked when it was fetched, that another model names with
|
||||
// another checksum, is refused and left for the model it is.
|
||||
func TestACheckedFileAnotherChecksumNamesIsRefusedAndLeft(t *testing.T) {
|
||||
body := "the weights"
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) }))
|
||||
defer srv.Close()
|
||||
sum := sha256.Sum256([]byte(body))
|
||||
f := &Fetcher{CacheDir: t.TempDir()}
|
||||
src := srv.URL + "/m.gguf"
|
||||
path, err := f.EnsureChecked(t.Context(), src, false, hex.EncodeToString(sum[:]))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wrong := strings.Repeat("0", 64)
|
||||
if _, err := f.EnsureChecked(t.Context(), src, false, wrong); err == nil || !strings.Contains(err.Error(), wrong) {
|
||||
t.Fatalf("not refused: %v", err)
|
||||
}
|
||||
if b, err := os.ReadFile(path); err != nil || string(b) != body {
|
||||
t.Errorf("the checked file was not left: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -73,8 +73,10 @@ func (f *leaderForward) standby(next http.Handler) http.Handler {
|
||||
p := &httputil.ReverseProxy{
|
||||
Rewrite: func(pr *httputil.ProxyRequest) {
|
||||
pr.Out.URL.Scheme, pr.Out.URL.Host = "https", target.Host
|
||||
// Escaped as it came: a name with a slash in it, Demo/hr
|
||||
// as Demo%2Fhr, is one segment, not two.
|
||||
pr.Out.URL.Path = forwardPrefix + pr.In.URL.Path
|
||||
pr.Out.URL.RawPath = ""
|
||||
pr.Out.URL.RawPath = forwardPrefix + pr.In.URL.EscapedPath()
|
||||
pr.Out.Host = target.Host
|
||||
// The leader's own token is not this one's, and the
|
||||
// certificate is what it believes.
|
||||
@@ -221,7 +223,7 @@ func (f *leaderForward) leaderSide(admin http.Handler) http.Handler {
|
||||
return
|
||||
}
|
||||
r2 := r.Clone(r.Context())
|
||||
r2.URL.Path, r2.URL.RawPath = path, ""
|
||||
r2.URL.Path, r2.URL.RawPath = path, strings.TrimPrefix(r.URL.EscapedPath(), forwardPrefix)
|
||||
w.Header().Set(hdrAnsweredBy, f.self)
|
||||
admin.ServeHTTP(w, r2)
|
||||
})
|
||||
|
||||
@@ -77,7 +77,7 @@ func newForwardCluster(t *testing.T) *forwardCluster {
|
||||
|
||||
// The leader's admin API: says whose it is, and who it was asked by.
|
||||
admin := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"path": r.URL.Path, "by": r.Header.Get(hdrForwardedBy),
|
||||
_ = json.NewEncoder(w).Encode(map[string]string{"path": r.URL.Path, "raw": r.URL.EscapedPath(), "by": r.Header.Get(hdrForwardedBy),
|
||||
"auth": r.Header.Get("Authorization"), "actor": r.Header.Get("X-Llmgw-Actor")})
|
||||
})
|
||||
mux := http.NewServeMux()
|
||||
@@ -242,3 +242,13 @@ func TestAStandbysPortalIsTheLeaders(t *testing.T) {
|
||||
t.Errorf("a forwarded request was forwarded again: %q", body)
|
||||
}
|
||||
}
|
||||
|
||||
// A name with a slash in it, escaped, stays one segment on the leader: a
|
||||
// collection Demo/hr removed through a standby was a 405 there.
|
||||
func TestAnEscapedSlashReachesTheLeaderEscaped(t *testing.T) {
|
||||
c := newForwardCluster(t)
|
||||
resp, got := c.ask(t, "/api/v1/kb/collections/Demo%2Fhr")
|
||||
if resp.StatusCode != http.StatusOK || got["raw"] != "/api/v1/kb/collections/Demo%2Fhr" {
|
||||
t.Fatalf("%d %v", resp.StatusCode, got)
|
||||
}
|
||||
}
|
||||
+28
-2
@@ -16,6 +16,7 @@ import (
|
||||
"github.com/spf13/cobra"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/tlsconf"
|
||||
)
|
||||
@@ -57,7 +58,7 @@ func newMirrorCmd() *cobra.Command {
|
||||
}
|
||||
|
||||
func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
var model, version, as string
|
||||
var model, version, as, sha, registry string
|
||||
|
||||
c := &cobra.Command{
|
||||
Use: "add <url>...",
|
||||
@@ -79,7 +80,7 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
if model == "" || version == "" {
|
||||
return fmt.Errorf("--model and --version are required: they are the layout")
|
||||
}
|
||||
if as != "" && len(args) > 1 {
|
||||
if (as != "" || sha != "") && len(args) > 1 {
|
||||
return fmt.Errorf("--as names one file; with %d URLs each keeps its own name", len(args))
|
||||
}
|
||||
out := cmd.OutOrStdout()
|
||||
@@ -90,6 +91,21 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
file = mirror.LastPathElement(source)
|
||||
}
|
||||
e := mirror.Entry{Model: model, Version: version, File: file}
|
||||
// From the registry: what it published for the file, to
|
||||
// compare with what arrives.
|
||||
want := strings.ToLower(sha)
|
||||
if id, rev, path, ok := hf.FileSource(source, registry); ok && want == "" {
|
||||
repo, err := hf.New(registry, 30*time.Second).RepoAt(cmd.Context(), id, rev)
|
||||
if err != nil {
|
||||
return fmt.Errorf("looking up %s in the registry, for the sha256 it published: %w "+
|
||||
"(--sha256 gives it instead)", id, err)
|
||||
}
|
||||
for _, f := range repo.Files {
|
||||
if f.Path == path {
|
||||
want = f.SHA256
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Fprintf(out, "fetching %s\n", source)
|
||||
added, err := mirror.Add(cmd.Context(), *dir, source, e, func(done, total int64) {
|
||||
if total > 0 {
|
||||
@@ -102,6 +118,14 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if want != "" && added.SHA256 != want {
|
||||
_ = mirror.Remove(*dir, e)
|
||||
return fmt.Errorf("%s arrived with sha256 %s, and %s was published for it: deleted",
|
||||
source, added.SHA256, want)
|
||||
}
|
||||
if want != "" {
|
||||
fmt.Fprintln(out, " sha256 matches the one published")
|
||||
}
|
||||
fmt.Fprintf(out, "%s/%s/%s %s\n sha256 %s\n",
|
||||
added.Entry.Model, added.Entry.Version, added.Entry.File,
|
||||
fileSize(added.Entry.Bytes), added.SHA256)
|
||||
@@ -123,6 +147,8 @@ func newMirrorAddCmd(dir *string) *cobra.Command {
|
||||
c.Flags().StringVar(&model, "model", "", "the model's name, one directory level")
|
||||
c.Flags().StringVar(&version, "version", "", "the version, one directory level; a date works well")
|
||||
c.Flags().StringVar(&as, "as", "", "the filename to save as, for one URL; taken from the URL when omitted")
|
||||
c.Flags().StringVar(&sha, "sha256", "", "the sha256 the file must have, for one URL; looked up for a Hugging Face URL")
|
||||
c.Flags().StringVar(®istry, "registry", hf.DefaultEndpoint, "the registry a resolve URL is looked up in")
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
@@ -453,6 +453,9 @@ func newServerCmd() *cobra.Command {
|
||||
if st := rt.Store(); st != nil {
|
||||
svc := modelsvc.New(cfg.Models, st, admin, log)
|
||||
svc.SetCheck(servableCheck(cfg.WantedEngines()["llamacpp"]))
|
||||
// What is added from the registry is pinned to the
|
||||
// commit it is at, with each file's sha256 for the nodes.
|
||||
svc.SetRegistry(hf.New(cfg.Mirror.Registry, 30*time.Second))
|
||||
// Other llmgw clusters' models, offered as PEER/MODEL:
|
||||
// read before the catalog is built, so a model that falls
|
||||
// back to one is whole at startup.
|
||||
|
||||
@@ -236,6 +236,7 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard {
|
||||
Adapters: adaptersToWire(p.Adapters),
|
||||
KvEvents: p.KVEvents,
|
||||
AllowPickle: p.AllowPickle,
|
||||
ModelSha256: p.ModelSHA256,
|
||||
},
|
||||
StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(),
|
||||
Pool: cmd.Pool,
|
||||
@@ -274,6 +275,7 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration
|
||||
Adapters: adaptersFromWire(p.GetAdapters()),
|
||||
KVEvents: p.GetKvEvents(),
|
||||
AllowPickle: p.GetAllowPickle(),
|
||||
ModelSHA256: p.GetModelSha256(),
|
||||
},
|
||||
}
|
||||
return spec, time.Duration(m.GetStartupTimeoutMs()) * time.Millisecond
|
||||
|
||||
@@ -1327,6 +1327,26 @@ func (m Model) SourceOf(p Profile) string {
|
||||
return m.Source
|
||||
}
|
||||
|
||||
// OriginSHA256 is the sha256 the registry published for the one file a
|
||||
// source names, from the model's origin: "" for a set, a repository, or a
|
||||
// file the origin does not list.
|
||||
func (m Model) OriginSHA256(source string) string {
|
||||
if m.Origin == nil || strings.HasSuffix(source, "/") {
|
||||
return ""
|
||||
}
|
||||
p := source
|
||||
if u, err := url.Parse(source); err == nil && u.Path != "" {
|
||||
p = u.Path
|
||||
}
|
||||
name := path.Base(p)
|
||||
for _, f := range m.Origin.Files {
|
||||
if f.SHA256 != "" && path.Base(f.Path) == name {
|
||||
return f.SHA256
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// WeightSources are every source a model's replicas may load: the model's
|
||||
// and each profile's of its own, once each. What a node must not have
|
||||
// collected from under it.
|
||||
@@ -1402,6 +1422,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) {
|
||||
Adapters: m.engineAdapters(),
|
||||
KVEvents: p.KVEvents,
|
||||
AllowPickle: m.AllowPickle,
|
||||
ModelSHA256: m.OriginSHA256(m.SourceOf(p)),
|
||||
}.WithRuntimeLoRA(m.MaxLoadedAdapters), true
|
||||
}
|
||||
return engines.Profile{}, false
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
package catalog
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestTheFileASourceNamesIsGivenItsPublishedChecksum(t *testing.T) {
|
||||
m := Model{Name: "q", Source: "http://10.0.0.146:8081/q/abc/q4.gguf", Origin: &Origin{Repo: "o/r", Revision: "abc",
|
||||
Files: []OriginFile{{Path: "sub/q4.gguf", SHA256: "aaaa"}, {Path: "config.json"}}}}
|
||||
for src, want := range map[string]string{
|
||||
m.Source: "aaaa",
|
||||
"https://huggingface.co/o/r/resolve/abc/sub/q4.gguf": "aaaa",
|
||||
"http://10.0.0.146:8081/q/abc/": "", // a set
|
||||
"o/r": "",
|
||||
"http://10.0.0.146:8081/q/abc/config.json": "", // no checksum published
|
||||
} {
|
||||
if got := m.OriginSHA256(src); got != want {
|
||||
t.Errorf("%s: %q, want %q", src, got, want)
|
||||
}
|
||||
}
|
||||
m.Profiles = []Profile{{Name: "cpu", Engine: "llamacpp"}}
|
||||
if ep, ok := m.EngineProfile("cpu"); !ok || ep.ModelSHA256 != "aaaa" {
|
||||
t.Errorf("the node is not told it: %+v", ep)
|
||||
}
|
||||
m.Origin = nil
|
||||
if m.OriginSHA256(m.Source) != "" {
|
||||
t.Error("a checksum with no origin")
|
||||
}
|
||||
}
|
||||
@@ -848,6 +848,9 @@ func (s *Server) mirror(ctx context.Context, j *Job, repo hf.Repo, set hf.FileSe
|
||||
}
|
||||
resp, err := s.deps.Mirror.Fetch(ctx, mirror.FetchRequest{
|
||||
Source: repo.DownloadURL("", f.Path), Model: name, Version: version, File: file,
|
||||
// The mirror deletes a copy that is not what the registry
|
||||
// published, rather than holding it for the nodes.
|
||||
SHA256: f.SHA256,
|
||||
})
|
||||
// Already there is not a failure: the same commit fetched before.
|
||||
if err != nil && !strings.Contains(err.Error(), "already") {
|
||||
|
||||
@@ -51,6 +51,8 @@ type Service struct {
|
||||
// check refuses a model that cannot run before it is added; see
|
||||
// SetCheck.
|
||||
check func(context.Context, catalog.Model) error
|
||||
// registry pins what is added from it to a commit; nil pins nothing.
|
||||
registry Registry
|
||||
|
||||
// federated are the models federated clusters offer; see SetFederated.
|
||||
federated func() []catalog.Model
|
||||
@@ -133,6 +135,11 @@ func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew b
|
||||
"placement could not tell when a node is full", m.Name, p.Name)
|
||||
}
|
||||
}
|
||||
if s.registry != nil {
|
||||
if err := pin(ctx, s.registry, &m); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if s.check != nil {
|
||||
if err := s.check(ctx, m); err != nil {
|
||||
return err
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
package modelsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
)
|
||||
|
||||
// A model added with weights from the registry is pinned to the commit they
|
||||
// were at: a file named by hf: or by a resolve URL on a branch becomes that
|
||||
// commit's URL, and a repository an engine downloads itself is given
|
||||
// --revision. The registry's sha256 of each file goes in the model's origin,
|
||||
// for the nodes to check what they fetch against. Without it, a new upload
|
||||
// to the repository reached the nodes that fetched after it, unseen, beside
|
||||
// nodes that still had the old one.
|
||||
|
||||
// Registry is where a repository's commit and files are looked up.
|
||||
type Registry interface {
|
||||
RepoAt(ctx context.Context, id, rev string) (hf.Repo, error)
|
||||
Base() string
|
||||
}
|
||||
|
||||
// SetRegistry has every model added from now on pinned against it.
|
||||
func (s *Service) SetRegistry(r Registry) { s.registry = r }
|
||||
|
||||
// engineDownloads are the engines that download a repository named by its id
|
||||
// themselves, and take --revision.
|
||||
var engineDownloads = map[string]bool{"vllm": true, "sglang": true}
|
||||
|
||||
var repoID = regexp.MustCompile(`^[A-Za-z0-9][\w.-]*/[\w.-]+$`)
|
||||
|
||||
// pin pins a model's registry sources, in place. One already given an origin
|
||||
// — served through the mirror, which records it — is left as it is.
|
||||
func pin(ctx context.Context, reg Registry, m *catalog.Model) error {
|
||||
if m.Origin != nil {
|
||||
return nil
|
||||
}
|
||||
repos := map[string]hf.Repo{}
|
||||
lookup := func(id, rev string) (hf.Repo, error) {
|
||||
key := id + "@" + rev
|
||||
if r, ok := repos[key]; ok {
|
||||
return r, nil
|
||||
}
|
||||
r, err := reg.RepoAt(ctx, id, rev)
|
||||
if err != nil {
|
||||
return r, fmt.Errorf("model %q: looking up %s in the registry, to pin it to a commit: %w", m.Name, id, err)
|
||||
}
|
||||
repos[key] = r
|
||||
return r, nil
|
||||
}
|
||||
record := func(r hf.Repo, files ...hf.File) {
|
||||
if m.Origin == nil {
|
||||
m.Origin = &catalog.Origin{Repo: r.ID, Revision: r.Revision}
|
||||
}
|
||||
if m.Origin.Repo != r.ID || m.Origin.Revision != r.Revision {
|
||||
return // one origin a model; a second repository is pinned, not recorded
|
||||
}
|
||||
for _, f := range files {
|
||||
m.Origin.Files = append(m.Origin.Files, catalog.OriginFile{Path: f.Path, SHA256: f.SHA256, Bytes: f.Bytes})
|
||||
}
|
||||
}
|
||||
|
||||
// One file: its commit's URL.
|
||||
pinned := map[string]string{}
|
||||
for _, src := range m.WeightSources() {
|
||||
id, rev, file, ok := hf.FileSource(src, reg.Base())
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
r, err := lookup(id, rev)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
i := slices.IndexFunc(r.Files, func(f hf.File) bool { return f.Path == file })
|
||||
if i < 0 {
|
||||
return fmt.Errorf("model %q: %s has no file %s at %s", m.Name, id, file, r.ShortRevision())
|
||||
}
|
||||
pinned[src] = r.DownloadURL(reg.Base(), file)
|
||||
record(r, r.Files[i])
|
||||
}
|
||||
if to, ok := pinned[m.Source]; ok {
|
||||
m.Source = to
|
||||
}
|
||||
for i := range m.Profiles {
|
||||
if to, ok := pinned[m.Profiles[i].Source]; ok {
|
||||
m.Profiles[i].Source = to
|
||||
}
|
||||
}
|
||||
|
||||
// A repository an engine downloads: --revision.
|
||||
for i := range m.Profiles {
|
||||
p := &m.Profiles[i]
|
||||
src := m.SourceOf(*p)
|
||||
if !engineDownloads[p.Engine] || !repoID.MatchString(src) {
|
||||
continue
|
||||
}
|
||||
r, err := lookup(src, argValue(p.Args, "--revision"))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if argValue(p.Args, "--revision") == "" {
|
||||
p.Args = append(slices.Clone(p.Args), "--revision", r.Revision)
|
||||
}
|
||||
var weights []hf.File
|
||||
for _, f := range r.Files {
|
||||
if f.IsWeights() {
|
||||
weights = append(weights, f)
|
||||
}
|
||||
}
|
||||
if m.Origin == nil || m.Origin.Repo != r.ID || len(m.Origin.Files) == 0 {
|
||||
record(r, weights...)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// argValue is a flag's value in arguments, "" when absent.
|
||||
func argValue(args []string, flag string) string {
|
||||
for i, a := range args {
|
||||
if a == flag && i+1 < len(args) {
|
||||
return args[i+1]
|
||||
}
|
||||
if v, ok := strings.CutPrefix(a, flag+"="); ok {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
package modelsvc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
)
|
||||
|
||||
const commit = "0123456789abcdef0123456789abcdef01234567"
|
||||
|
||||
type fakeRegistry struct{ asked []string }
|
||||
|
||||
func (f *fakeRegistry) Base() string { return "https://huggingface.co" }
|
||||
|
||||
func (f *fakeRegistry) RepoAt(_ context.Context, id, rev string) (hf.Repo, error) {
|
||||
f.asked = append(f.asked, id+"@"+rev)
|
||||
if id == "gone/repo" {
|
||||
return hf.Repo{}, errors.New("404")
|
||||
}
|
||||
return hf.Repo{ID: id, Revision: commit, Files: []hf.File{
|
||||
{Path: "q4.gguf", Bytes: 10, SHA256: "aaaa"},
|
||||
{Path: "model-00001-of-00002.safetensors", Bytes: 20, SHA256: "bbbb"},
|
||||
{Path: "config.json", Bytes: 1},
|
||||
}}, nil
|
||||
}
|
||||
|
||||
func TestAFileFromTheRegistryIsPinnedToItsCommit(t *testing.T) {
|
||||
for _, src := range []string{"hf:o/r/q4.gguf", "https://huggingface.co/o/r/resolve/main/q4.gguf"} {
|
||||
m := catalog.Model{Name: "q", Source: src, Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}}
|
||||
if err := pin(t.Context(), &fakeRegistry{}, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.Source != "https://huggingface.co/o/r/resolve/"+commit+"/q4.gguf" {
|
||||
t.Errorf("%s became %s", src, m.Source)
|
||||
}
|
||||
if m.Origin == nil || m.Origin.Revision != commit || m.OriginSHA256(m.Source) != "aaaa" {
|
||||
t.Errorf("%s: origin %+v", src, m.Origin)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestARepositoryAnEngineDownloadsIsGivenItsRevision(t *testing.T) {
|
||||
m := catalog.Model{Name: "q", Source: "o/r", Profiles: []catalog.Profile{
|
||||
{Name: "gpu", Engine: "vllm", Args: []string{"--max-model-len", "4096"}},
|
||||
{Name: "cpu", Engine: "llamacpp", Source: "hf:o/r/q4.gguf"}}}
|
||||
if err := pin(t.Context(), &fakeRegistry{}, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if a := m.Profiles[0].Args; !slices.Equal(a, []string{"--max-model-len", "4096", "--revision", commit}) {
|
||||
t.Errorf("vllm args %v", a)
|
||||
}
|
||||
if !strings.Contains(m.Profiles[1].Source, commit) {
|
||||
t.Errorf("the llama.cpp profile's file: %s", m.Profiles[1].Source)
|
||||
}
|
||||
// A revision given is the one looked up, and kept.
|
||||
reg := &fakeRegistry{}
|
||||
m = catalog.Model{Name: "q", Source: "o/r", Profiles: []catalog.Profile{{Name: "gpu", Engine: "vllm", Args: []string{"--revision", "v2"}}}}
|
||||
if err := pin(t.Context(), reg, &m); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !slices.Contains(reg.asked, "o/r@v2") || argValue(m.Profiles[0].Args, "--revision") != "v2" {
|
||||
t.Errorf("asked %v, args %v", reg.asked, m.Profiles[0].Args)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhatIsNotTheRegistrysIsLeftAndWhatCannotBePinnedIsRefused(t *testing.T) {
|
||||
m := catalog.Model{Name: "q", Source: "http://10.0.0.146:8081/q/abc/q4.gguf", Profiles: []catalog.Profile{{Name: "cpu", Engine: "llamacpp"}}}
|
||||
reg := &fakeRegistry{}
|
||||
if err := pin(t.Context(), reg, &m); err != nil || len(reg.asked) != 0 || m.Origin != nil {
|
||||
t.Errorf("a mirror's file: %v %v %+v", err, reg.asked, m.Origin)
|
||||
}
|
||||
// Served through the mirror, with its origin: left alone.
|
||||
m = catalog.Model{Name: "q", Source: "hf:o/r/q4.gguf", Origin: &catalog.Origin{Repo: "o/r"}}
|
||||
if err := pin(t.Context(), reg, &m); err != nil || m.Source != "hf:o/r/q4.gguf" {
|
||||
t.Errorf("an origin given was not kept: %v %s", err, m.Source)
|
||||
}
|
||||
for _, src := range []string{"hf:gone/repo/q4.gguf", "hf:o/r/missing.gguf"} {
|
||||
m = catalog.Model{Name: "q", Source: src}
|
||||
if err := pin(t.Context(), reg, &m); err == nil {
|
||||
t.Errorf("%s was not refused", src)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -715,3 +715,35 @@ func ModelName(repoID string) string {
|
||||
name = strings.TrimSuffix(name, "-gguf")
|
||||
return strings.ToLower(name)
|
||||
}
|
||||
|
||||
// FileSource reads a source naming one file of a registry repository: hf:
|
||||
// with the file, or a resolve URL on the registry. A resolve URL already at
|
||||
// a commit is still looked up, for the file's sha256.
|
||||
func FileSource(src, base string) (id, rev, file string, ok bool) {
|
||||
if rest, found := strings.CutPrefix(src, "hf:"); found {
|
||||
parts := strings.Split(rest, "/")
|
||||
if len(parts) < 3 {
|
||||
return "", "", "", false
|
||||
}
|
||||
return parts[0] + "/" + parts[1], "", strings.Join(parts[2:], "/"), true
|
||||
}
|
||||
u, err := url.Parse(src)
|
||||
if err != nil || (u.Scheme != "https" && u.Scheme != "http") || !isRegistryHost(u.Host, base) {
|
||||
return "", "", "", false
|
||||
}
|
||||
parts := strings.Split(strings.Trim(u.Path, "/"), "/")
|
||||
if len(parts) < 5 || parts[2] != "resolve" {
|
||||
return "", "", "", false
|
||||
}
|
||||
return parts[0] + "/" + parts[1], parts[3], strings.Join(parts[4:], "/"), true
|
||||
}
|
||||
|
||||
// isRegistryHost says whether a host is the registry's: Hugging Face's own, or
|
||||
// the endpoint the cluster names instead.
|
||||
func isRegistryHost(host, base string) bool {
|
||||
if host == "huggingface.co" {
|
||||
return true
|
||||
}
|
||||
b, err := url.Parse(base)
|
||||
return err == nil && b.Host != "" && b.Host == host
|
||||
}
|
||||
@@ -63,6 +63,8 @@ type Collection struct {
|
||||
EmbedModel string `json:"embed_model"`
|
||||
Dims int `json:"dims"`
|
||||
Writable bool `json:"writable"`
|
||||
// ServerEmbeds is a gateway that embeds a document sent as text alone.
|
||||
ServerEmbeds bool `json:"server_embeds"`
|
||||
}
|
||||
|
||||
// RemoteDoc is a document the collection holds.
|
||||
@@ -82,8 +84,9 @@ type Manifest struct {
|
||||
|
||||
// ChunkOut is one chunk as sent.
|
||||
type ChunkOut struct {
|
||||
Text string `json:"text"`
|
||||
Embedding []float32 `json:"embedding"`
|
||||
Text string `json:"text"`
|
||||
// Embedding is empty when the gateway embeds the chunk.
|
||||
Embedding []float32 `json:"embedding,omitempty"`
|
||||
}
|
||||
|
||||
// DocOut is one document as sent.
|
||||
|
||||
@@ -46,8 +46,11 @@ type Options struct {
|
||||
Retry bool
|
||||
// Workers is how many files are read and sent at once; 0 is 4.
|
||||
Workers int
|
||||
Extract extract.Options
|
||||
Chunk chunk.Options
|
||||
// EmbedHere embeds the chunks on this machine even when the gateway
|
||||
// would: a vector up and down for each, rather than the text alone.
|
||||
EmbedHere bool
|
||||
Extract extract.Options
|
||||
Chunk chunk.Options
|
||||
}
|
||||
|
||||
// Event is something a sync did to one document, for showing progress.
|
||||
@@ -359,8 +362,12 @@ func (s *syncer) send(ctx context.Context, f file, name, digest, text string) {
|
||||
texts[i] = c.Text()
|
||||
size += int64(len(texts[i]))
|
||||
}
|
||||
// About ten bytes of JSON a dimension.
|
||||
size += int64(len(cs) * s.coll.Dims * 10)
|
||||
// Sent as text alone when the gateway embeds it; else about ten bytes
|
||||
// of JSON a dimension with each chunk.
|
||||
serverEmbeds := s.coll.ServerEmbeds && !s.o.EmbedHere
|
||||
if !serverEmbeds {
|
||||
size += int64(len(cs) * s.coll.Dims * 10)
|
||||
}
|
||||
if size > maxDocJSON {
|
||||
s.fail(ctx, f, name, digest, fmt.Errorf("%d MiB to send is over the gateway's limit for one document; split the file", size>>20))
|
||||
return
|
||||
@@ -369,12 +376,19 @@ func (s *syncer) send(ctx context.Context, f file, name, digest, text string) {
|
||||
s.emit(Event{Kind: "would-send", Name: name, Chunks: len(cs), Bytes: size})
|
||||
return
|
||||
}
|
||||
out := DocOut{Name: name, Digest: digest, Bytes: f.size, Chunks: make([]ChunkOut, len(cs))}
|
||||
if serverEmbeds {
|
||||
for i := range cs {
|
||||
out.Chunks[i] = ChunkOut{Text: texts[i]}
|
||||
}
|
||||
s.put(ctx, f, name, digest, out, len(cs), size)
|
||||
return
|
||||
}
|
||||
vecs, err := s.c.Embed(ctx, s.coll.EmbedModel, texts)
|
||||
if err != nil {
|
||||
s.fail(ctx, f, name, digest, fmt.Errorf("embedding with %s: %w", s.coll.EmbedModel, err))
|
||||
return
|
||||
}
|
||||
out := DocOut{Name: name, Digest: digest, Bytes: f.size, Chunks: make([]ChunkOut, len(cs))}
|
||||
for i := range cs {
|
||||
if len(vecs[i]) != s.coll.Dims {
|
||||
s.fail(ctx, f, name, digest, fmt.Errorf("%s gave %d dimensions and the collection takes %d",
|
||||
@@ -383,12 +397,17 @@ func (s *syncer) send(ctx context.Context, f file, name, digest, text string) {
|
||||
}
|
||||
out.Chunks[i] = ChunkOut{Text: texts[i], Embedding: vecs[i]}
|
||||
}
|
||||
s.put(ctx, f, name, digest, out, len(cs), size)
|
||||
}
|
||||
|
||||
// put sends a document and records it.
|
||||
func (s *syncer) put(ctx context.Context, f file, name, digest string, out DocOut, chunks int, size int64) {
|
||||
if _, err := s.c.Put(ctx, s.coll.ID, out); err != nil {
|
||||
s.fail(ctx, f, name, digest, err)
|
||||
return
|
||||
}
|
||||
s.record(ctx, s.entry(f, name, digest, Sent, "", len(cs)))
|
||||
s.emit(Event{Kind: "sent", Name: name, Chunks: len(cs), Bytes: size})
|
||||
s.record(ctx, s.entry(f, name, digest, Sent, "", chunks))
|
||||
s.emit(Event{Kind: "sent", Name: name, Chunks: chunks, Bytes: size})
|
||||
}
|
||||
|
||||
func (s *syncer) fail(ctx context.Context, f file, name, digest string, err error) {
|
||||
|
||||
@@ -22,6 +22,10 @@ type fakeGateway struct {
|
||||
puts int
|
||||
embedded int
|
||||
busyFirst bool
|
||||
// serverEmbeds is a gateway that embeds text it is sent; vectors
|
||||
// counts chunks that came with one.
|
||||
serverEmbeds bool
|
||||
vectors int
|
||||
}
|
||||
|
||||
func (g *fakeGateway) serve(t *testing.T) *Client {
|
||||
@@ -37,6 +41,7 @@ func (g *fakeGateway) serve(t *testing.T) *Client {
|
||||
}
|
||||
m := Manifest{Collection: coll}
|
||||
m.Collection.Writable = g.writable
|
||||
m.Collection.ServerEmbeds = g.serverEmbeds
|
||||
for _, d := range g.docs {
|
||||
m.Documents = append(m.Documents, d)
|
||||
}
|
||||
@@ -48,6 +53,11 @@ func (g *fakeGateway) serve(t *testing.T) *Client {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
g.puts++
|
||||
for _, c := range d.Chunks {
|
||||
if len(c.Embedding) > 0 {
|
||||
g.vectors++
|
||||
}
|
||||
}
|
||||
g.docs[d.Name] = RemoteDoc{Name: d.Name, Digest: d.Digest, Chunks: len(d.Chunks)}
|
||||
_, _ = w.Write([]byte(`{"changed":true}`))
|
||||
})
|
||||
@@ -270,3 +280,22 @@ func TestFiltersAndNames(t *testing.T) {
|
||||
t.Errorf("an empty folder gave %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAGatewayThatEmbedsIsSentTextAlone(t *testing.T) {
|
||||
g, c, l, dir := setup(t)
|
||||
g.serverEmbeds = true
|
||||
rep, k := run(t, c, l, Options{Collection: "ik", Roots: []string{dir}})
|
||||
if len(k["sent"]) != 2 || rep.Failed != 0 {
|
||||
t.Fatalf("%+v %v", rep, k)
|
||||
}
|
||||
if g.embedded != 0 || g.vectors != 0 {
|
||||
t.Errorf("embedded here: %d asked, %d vectors sent", g.embedded, g.vectors)
|
||||
}
|
||||
|
||||
// --embed-here sends the vectors all the same.
|
||||
write(t, dir, "politika/yeni.md", "Yeni kural.\n")
|
||||
run(t, c, l, Options{Collection: "ik", Roots: []string{dir}, EmbedHere: true})
|
||||
if g.embedded == 0 || g.vectors == 0 {
|
||||
t.Errorf("--embed-here: %d asked, %d vectors sent", g.embedded, g.vectors)
|
||||
}
|
||||
}
|
||||
@@ -75,6 +75,9 @@ type FetchRequest struct {
|
||||
// empty, which is right almost always and wrong visibly rather than
|
||||
// quietly when it is not.
|
||||
File string `json:"file"`
|
||||
// SHA256 is what the registry published for the file: a download that
|
||||
// is not it is deleted and the fetch fails. Empty checks nothing.
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
}
|
||||
|
||||
// FetchResponse reports where a download stands, and where it will be.
|
||||
@@ -102,7 +105,7 @@ func (a *API) fetch(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
e := Entry{Model: req.Model, Version: req.Version, File: req.File}
|
||||
st, err := a.m.Start(req.Source, e)
|
||||
st, err := a.m.StartChecked(req.Source, e, strings.ToLower(req.SHA256))
|
||||
switch {
|
||||
case errors.Is(err, ErrAlreadyFetching):
|
||||
// Not an error the caller needs to handle: a control plane that asks
|
||||
|
||||
@@ -112,6 +112,12 @@ func (m *Manager) Start(source string, e Entry) (Status, error) {
|
||||
return m.start(source, e, "", m.client)
|
||||
}
|
||||
|
||||
// StartChecked is Start for a file whose sha256 the registry published: one
|
||||
// that arrives otherwise is deleted, and the fetch fails saying both.
|
||||
func (m *Manager) StartChecked(source string, e Entry, sha string) (Status, error) {
|
||||
return m.start(source, e, sha, m.client)
|
||||
}
|
||||
|
||||
// startCopy fetches a file a peer mirror holds, from a private address,
|
||||
// and keeps it only if its checksum is the peer's.
|
||||
func (m *Manager) startCopy(source string, e Entry, sha string) (Status, error) {
|
||||
@@ -163,9 +169,10 @@ func (m *Manager) run(source string, e Entry, key, expect string, client *http.C
|
||||
})
|
||||
|
||||
if err == nil && expect != "" && added.SHA256 != expect {
|
||||
// Not what the peer holds: a file changed or cut short on the way.
|
||||
// Not what the registry published, or what the peer holds: a file
|
||||
// changed upstream, or on the way.
|
||||
_ = Remove(m.root, e)
|
||||
err = fmt.Errorf("the copy's sha256 %s is not the peer's %s", added.SHA256, expect)
|
||||
err = fmt.Errorf("%s arrived with sha256 %s, not the %s expected of it: deleted", e.File, added.SHA256, expect)
|
||||
}
|
||||
if err == nil {
|
||||
// Added again after it was removed: the new one stands.
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -243,3 +244,30 @@ func TestAFileRemovedByHandIsNotReportedAndItsRemovalIsCarried(t *testing.T) {
|
||||
t.Error("the peer kept it")
|
||||
}
|
||||
}
|
||||
|
||||
// A fetch given the registry's sha256 keeps only that file.
|
||||
func TestAFetchThatIsNotWhatTheRegistryPublishedIsDeleted(t *testing.T) {
|
||||
served := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
_, _ = w.Write([]byte("replaced upstream"))
|
||||
}))
|
||||
defer served.Close()
|
||||
tm := newTestMirror(t)
|
||||
e := Entry{Model: "qwen", Version: "v1", File: "m.gguf"}
|
||||
if _, err := tm.m.StartChecked(served.URL+"/m.gguf", e, "aaaa"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
var st Status
|
||||
for time.Now().Before(deadline) {
|
||||
if st, _ = tm.m.Status("qwen", "v1"); st.State == Failed {
|
||||
break
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if st.State != Failed || !strings.Contains(st.Err, "aaaa") {
|
||||
t.Fatalf("%+v", st)
|
||||
}
|
||||
if _, err := os.Stat(e.Path(tm.root)); !os.IsNotExist(err) {
|
||||
t.Error("kept")
|
||||
}
|
||||
}
|
||||
+44
-4
@@ -53,6 +53,9 @@ type KBCollectionView struct {
|
||||
Dims int `json:"dims"`
|
||||
// Writable is whether this caller may add and remove documents.
|
||||
Writable bool `json:"writable"`
|
||||
// ServerEmbeds says a document may be sent as text alone: the gateway
|
||||
// embeds its chunks, as the caller, with the collection's model.
|
||||
ServerEmbeds bool `json:"server_embeds"`
|
||||
}
|
||||
|
||||
// KBDocumentView is a document as the manifest lists it.
|
||||
@@ -77,8 +80,10 @@ type KBPutDocument struct {
|
||||
Digest string `json:"digest"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
Chunks []struct {
|
||||
Text string `json:"text"`
|
||||
Embedding []float32 `json:"embedding"`
|
||||
Text string `json:"text"`
|
||||
// Embedding is the chunk's vector; with none on any chunk the
|
||||
// gateway embeds them.
|
||||
Embedding []float32 `json:"embedding,omitempty"`
|
||||
} `json:"chunks"`
|
||||
}
|
||||
|
||||
@@ -124,7 +129,7 @@ func (r *Router) kbCaller(w http.ResponseWriter, req *http.Request) (auth.Princi
|
||||
|
||||
func (r *Router) kbView(ctx context.Context, c storepkg.KBCollection, writable bool) KBCollectionView {
|
||||
return KBCollectionView{ID: c.ID, Name: c.Name, Owner: string(c.OwnerScope) + ":" + c.OwnerID,
|
||||
OwnerName: r.store.OwnerName(ctx, c.OwnerScope, c.OwnerID), EmbedModel: c.EmbedModel, Dims: c.Dims, Writable: writable}
|
||||
OwnerName: r.store.OwnerName(ctx, c.OwnerScope, c.OwnerID), EmbedModel: c.EmbedModel, Dims: c.Dims, Writable: writable, ServerEmbeds: true}
|
||||
}
|
||||
|
||||
// kbCollection finds a collection by id or name among those the caller may
|
||||
@@ -281,6 +286,19 @@ func (r *Router) handleKBPut(w http.ResponseWriter, req *http.Request) {
|
||||
fmt.Sprintf("%d chunks: a document has at most %d", len(in.Chunks), kbMaxChunks))
|
||||
return
|
||||
}
|
||||
// Text alone: the gateway embeds it, as the caller, so their limits and
|
||||
// usage count it as if they had asked.
|
||||
embedded := 0
|
||||
for _, ch := range in.Chunks {
|
||||
if len(ch.Embedding) > 0 {
|
||||
embedded++
|
||||
}
|
||||
}
|
||||
if embedded > 0 && embedded < len(in.Chunks) {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error",
|
||||
"some chunks have an embedding and some not: send every chunk's, or none for the gateway to embed")
|
||||
return
|
||||
}
|
||||
chunks := make([]storepkg.KBChunk, len(in.Chunks))
|
||||
for i, ch := range in.Chunks {
|
||||
if !utf8.ValidString(ch.Text) || strings.TrimSpace(ch.Text) == "" {
|
||||
@@ -292,7 +310,7 @@ func (r *Router) handleKBPut(w http.ResponseWriter, req *http.Request) {
|
||||
fmt.Sprintf("chunk %d is over %d characters", i, kbMaxChunkRunes))
|
||||
return
|
||||
}
|
||||
if len(ch.Embedding) != c.Dims {
|
||||
if embedded > 0 && len(ch.Embedding) != c.Dims {
|
||||
writeError(w, http.StatusBadRequest, "invalid_request_error",
|
||||
fmt.Sprintf("chunk %d has %d dimensions; collection %s is embedded by %s in %d", i,
|
||||
len(ch.Embedding), c.Name, c.EmbedModel, c.Dims))
|
||||
@@ -300,6 +318,28 @@ func (r *Router) handleKBPut(w http.ResponseWriter, req *http.Request) {
|
||||
}
|
||||
chunks[i] = storepkg.KBChunk{Body: strings.ToValidUTF8(ch.Text, ""), Embedding: ch.Embedding}
|
||||
}
|
||||
if embedded == 0 {
|
||||
texts := make([]string, len(chunks))
|
||||
for i := range chunks {
|
||||
texts[i] = chunks[i].Body
|
||||
}
|
||||
vecs, err := r.kbEmbed(req, c, texts)
|
||||
if err != nil {
|
||||
var ee *kbEmbedError
|
||||
if errors.As(err, &ee) {
|
||||
if ee.retryAfter != "" {
|
||||
w.Header().Set("Retry-After", ee.retryAfter)
|
||||
}
|
||||
writeError(w, ee.status, "invalid_request_error", err.Error())
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusBadGateway, "server_error", err.Error())
|
||||
return
|
||||
}
|
||||
for i := range chunks {
|
||||
chunks[i].Embedding = vecs[i]
|
||||
}
|
||||
}
|
||||
d, changed, err := r.store.PutDocument(req.Context(), c, storepkg.KBDocument{Name: name, Digest: in.Digest,
|
||||
Bytes: in.Bytes, AddedBy: kbActor(p)}, chunks)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
storepkg "gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// A document sent as text alone is embedded here, through the gateway's own
|
||||
// /v1/embeddings as the caller: their key, limits and usage, as if they had
|
||||
// asked. The client then sends a kilobyte a chunk instead of the vector's
|
||||
// twenty in each direction.
|
||||
|
||||
const (
|
||||
// kbEmbedBatch is how many chunks go in one embeddings request, and
|
||||
// kbEmbedAtOnce how many such requests are in flight for one document:
|
||||
// enough to fill an embedding replica's slots.
|
||||
kbEmbedBatch = 32
|
||||
kbEmbedAtOnce = 4
|
||||
// kbEmbedWait is how long one batch waits out answers that say to try
|
||||
// again, before the document is refused with the same advice.
|
||||
kbEmbedWait = 5 * time.Minute
|
||||
)
|
||||
|
||||
// kbEmbedError is an embeddings answer that was not a success, passed on as
|
||||
// the API gave it.
|
||||
type kbEmbedError struct {
|
||||
status int
|
||||
retryAfter string
|
||||
msg string
|
||||
}
|
||||
|
||||
func (e *kbEmbedError) Error() string { return e.msg }
|
||||
|
||||
// kbEmbed embeds texts with the collection's model, as the request's caller.
|
||||
func (r *Router) kbEmbed(req *http.Request, c storepkg.KBCollection, texts []string) ([][]float32, error) {
|
||||
ctx, cancel := context.WithCancel(req.Context())
|
||||
defer cancel()
|
||||
out := make([][]float32, len(texts))
|
||||
var (
|
||||
mu sync.Mutex
|
||||
first error
|
||||
wg sync.WaitGroup
|
||||
)
|
||||
slots := make(chan struct{}, kbEmbedAtOnce)
|
||||
for start := 0; start < len(texts); start += kbEmbedBatch {
|
||||
end := min(start+kbEmbedBatch, len(texts))
|
||||
wg.Add(1)
|
||||
slots <- struct{}{}
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
defer func() { <-slots }()
|
||||
vecs, err := r.kbEmbedBatch(ctx, req, c, texts[start:end])
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if err != nil {
|
||||
if first == nil {
|
||||
first = err
|
||||
cancel()
|
||||
}
|
||||
return
|
||||
}
|
||||
copy(out[start:end], vecs)
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if first != nil {
|
||||
return nil, first
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// kbEmbedBatch embeds one batch, waiting out answers that say when to try
|
||||
// again.
|
||||
func (r *Router) kbEmbedBatch(ctx context.Context, req *http.Request, c storepkg.KBCollection, texts []string) ([][]float32, error) {
|
||||
call := mediaCall{req: req.WithContext(ctx)}
|
||||
deadline := time.Now().Add(kbEmbedWait)
|
||||
for {
|
||||
status, h, b, err := r.self(call, http.MethodPost, "/v1/embeddings",
|
||||
map[string]any{"model": c.EmbedModel, "input": texts, "encoding_format": "float"})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if status < 300 {
|
||||
return kbVectors(b, len(texts), c)
|
||||
}
|
||||
retry := h.Get("Retry-After")
|
||||
if retry == "" || (status != http.StatusTooManyRequests && status < 500) || time.Now().After(deadline) {
|
||||
return nil, &kbEmbedError{status: status, retryAfter: retry,
|
||||
msg: fmt.Sprintf("embedding with %s: %s", c.EmbedModel, errorMessage(b))}
|
||||
}
|
||||
secs, _ := strconv.Atoi(retry)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-time.After(time.Duration(max(secs, 1)) * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// kbVectors reads an embeddings answer: one vector a text, of the
|
||||
// collection's dimensions.
|
||||
func kbVectors(b []byte, n int, c storepkg.KBCollection) ([][]float32, error) {
|
||||
var res struct {
|
||||
Data []struct {
|
||||
Index int `json:"index"`
|
||||
Embedding []float32 `json:"embedding"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if err := json.Unmarshal(b, &res); err != nil {
|
||||
return nil, fmt.Errorf("embedding with %s: the answer is not JSON: %w", c.EmbedModel, err)
|
||||
}
|
||||
if len(res.Data) != n {
|
||||
return nil, fmt.Errorf("embedding with %s: %d of %d texts came back", c.EmbedModel, len(res.Data), n)
|
||||
}
|
||||
out := make([][]float32, n)
|
||||
for _, d := range res.Data {
|
||||
if d.Index < 0 || d.Index >= n || out[d.Index] != nil {
|
||||
return nil, fmt.Errorf("embedding with %s: an embedding came back with index %d", c.EmbedModel, d.Index)
|
||||
}
|
||||
if len(d.Embedding) != c.Dims {
|
||||
return nil, &kbEmbedError{status: http.StatusBadRequest,
|
||||
msg: fmt.Sprintf("%s gave %d dimensions and collection %s takes %d", c.EmbedModel, len(d.Embedding), c.Name, c.Dims)}
|
||||
}
|
||||
out[d.Index] = d.Embedding
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package router
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/store"
|
||||
)
|
||||
|
||||
// A document's text is embedded through the gateway's own API as the
|
||||
// caller, in batches, each vector in its chunk's place.
|
||||
func TestTheGatewayEmbedsADocumentSentAsText(t *testing.T) {
|
||||
e := newFakeEngine(t, "qwen2.5-14b-instruct")
|
||||
rt, secret := pricedRouter(t, e.srv.URL)
|
||||
_ = rt.Handler() // mounts the API the embedding goes through
|
||||
texts := make([]string, kbEmbedBatch*2+5)
|
||||
for i := range texts {
|
||||
texts[i] = strings.Repeat("söz ", i+1)
|
||||
}
|
||||
req := httptest.NewRequestWithContext(t.Context(), http.MethodPut, "/v1/kb/collections/x/documents", nil)
|
||||
req.Header.Set("Authorization", "Bearer "+secret)
|
||||
c := store.KBCollection{Name: "x", EmbedModel: "qwen2.5-14b-instruct", Dims: 3}
|
||||
vecs, err := rt.kbEmbed(req, c, texts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i, v := range vecs {
|
||||
// The fake engine's vector is the text's bytes, its words, and 1.
|
||||
if len(v) != 3 || int(v[1]) != i+1 || int(v[0]) != len(texts[i]) {
|
||||
t.Fatalf("chunk %d got %v", i, v)
|
||||
}
|
||||
}
|
||||
|
||||
// A model of other dimensions is refused, saying so.
|
||||
c.Dims = 4
|
||||
if _, err := rt.kbEmbed(req, c, texts[:2]); err == nil || !strings.Contains(err.Error(), "dimensions") {
|
||||
t.Errorf("4 dimensions: %v", err)
|
||||
}
|
||||
// Without the caller's key, nothing is embedded.
|
||||
bare := httptest.NewRequestWithContext(t.Context(), http.MethodPut, "/v1/kb/collections/x/documents", nil)
|
||||
c.Dims = 3
|
||||
if _, err := rt.kbEmbed(bare, c, texts[:1]); err == nil {
|
||||
t.Error("embedded without a key")
|
||||
}
|
||||
}
|
||||
@@ -206,3 +206,36 @@ func TestAPersonalKeyReadsOnlyInTheProjectItNames(t *testing.T) {
|
||||
t.Errorf("a project the person is not in: %d", code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADocumentSentAsTextIsEmbeddedByTheGateway(t *testing.T) {
|
||||
rt := kbRouter(t)
|
||||
h := rt.Handler()
|
||||
ctx := t.Context()
|
||||
s := rt.store
|
||||
sfx := strings.ToLower(store.NewID(""))
|
||||
org, _ := s.CreateOrg(ctx, "e-"+sfx)
|
||||
team, _ := s.CreateTeam(ctx, org, "t")
|
||||
prj, _ := s.CreateProject(ctx, team, "p")
|
||||
// The fake engine embeds in three dimensions.
|
||||
if _, err := s.CreateCollection(ctx, store.KBCollection{OwnerScope: store.ScopeProject, OwnerID: prj,
|
||||
Name: "metin", EmbedModel: "qwen2.5-14b-instruct", Dims: 3}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key := kbKey(t, s, prj, "")
|
||||
base := "/v1/kb/collections/metin/"
|
||||
if code, _, body := call(t, h, key, http.MethodGet, base+"manifest", nil); !strings.Contains(body, `"server_embeds":true`) {
|
||||
t.Errorf("manifest: %d %s", code, body)
|
||||
}
|
||||
textOnly := map[string]any{"name": "a.md", "digest": "d1", "bytes": 10, "chunks": []map[string]any{
|
||||
{"text": "birinci parça"}, {"text": "ikinci parça burada"}}}
|
||||
code, out, body := call(t, h, key, http.MethodPut, base+"documents", textOnly)
|
||||
if code != http.StatusOK || out["chunks"] != float64(2) {
|
||||
t.Fatalf("text only: %d %s", code, body)
|
||||
}
|
||||
// Some with a vector and some without is refused.
|
||||
mixed := map[string]any{"name": "b.md", "digest": "d2", "bytes": 10, "chunks": []map[string]any{
|
||||
{"text": "bir", "embedding": []float32{1, 0, 0}}, {"text": "iki"}}}
|
||||
if code, _, body := call(t, h, key, http.MethodPut, base+"documents", mixed); code != http.StatusBadRequest {
|
||||
t.Errorf("mixed: %d %s", code, body)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user