diff --git a/BACKLOG.md b/BACKLOG.md index 8a73c094..e7d671d9 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1784,6 +1784,62 @@ A customer that already runs Kong puts llmgw behind it: `ai-proxy` with route. A page in docs/ with the configuration, tried against Kong 3.9 in a container, and what Kong's token accounting sees. +## M. Models from Hugging Face we can trust (written 2026-10-25) + +The user asked how we know a model fetched from Hugging Face is safe. Two +questions: whether its files can run code on our machines, which we can +close, and whether the model itself was trained to misbehave, which no scan +proves — only who published it, an eval before it serves, and the +guardrails narrow that. What existed: `--trust-remote-code` refused, no +repository Python fetched, ComfyUI nodes pinned to a commit, engines +hash-pinned and run as `llmgw-engine`, mirror versions pinned to a commit +and never overwritten. What did not: everything below. + +### M1. Weights in a pickle format refused (S) — done 2026-10-25 +`.bin`, `.pt`, `.pth`, `.ckpt`, `.pkl` run code as they are loaded, which is +how the malicious models found on Hugging Face work. The node refuses such a +source, or a mirror set holding one, before fetching anything; a model being +added or given such a source is refused; vLLM and SGLang downloading a model +by its id are given `--load-format safetensors`; the mirror page says so on +the file. `allow_pickle: true` on a model lets it through, on purpose. On the +PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that +needs it. + +### M2. Every download pinned and checked against Hugging Face (M) +- `hf:org/repo/file` follows `main` and is checked by nothing; a bare + `org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is. + Both: resolve to the commit when the model is added, keep it in the + catalog's origin, and fetch that commit only (`--revision` for vLLM). +- The sha256 Hugging Face publishes for each LFS file compared on every + path, and a file that differs removed: `mirror add`, the manager, the + node's `hf:` and http sources, mirror set files with no sha recorded. +- Done when: a file changed on Hugging Face after a model was added is + refused by the node with both checksums named. + +### M3. Hugging Face's own scan (S) +Hugging Face scans every file (pickle imports, ClamAV, partners' scanners) +and marks a file unsafe. Ask for it with the repository listing; refuse an +unsafe file, show the result on the mirror page and in the catalog's origin. +The API field has to be confirmed first. + +### M4. Publishers we trust, and the rest asked about (S) +An allow-list of organisations in the cluster's settings (Qwen, meta-llama, +mistralai, google, microsoft, …). A model from anyone else is added only by +an admin who says so, and who said it is kept in the origin. The PoC already +serves four community models whose training nobody here checked. + +### M5. Engines without the internet (M) +The firewall is inbound only, so an engine that was made to run code can +send what it reads anywhere. With weights from the mirror the engines need +no egress: an nft output rule for the engine user (allow the mirror, the +cluster, loopback), systemd hardening of the agent's engines +(`NoNewPrivileges`, `ProtectSystem`, private /tmp). Agent itself still root. + +### M6. An eval before a new model serves (S) +`eval` and `canary` exist; make a new model from an untrusted publisher +(M4) serve only after its eval suite passes, so a model trained to answer +wrongly on a trigger has at least been asked the suite. + ## D. Facts worth knowing before starting - Admin API: `127.0.0.1:9090`, behind the token in diff --git a/CHANGELOG.md b/CHANGELOG.md index 1e20de87..a5a87d3b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,30 @@ is only released once its **exit criterion** is met, not when its code lands. Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. +## Unreleased + +**Upgrading.** +- A model whose weights are in a pickle format (`.bin`, `.pt`, `.pth`, + `.ckpt`, `.pkl`) stops being placed: its node refuses the source. Give it + its safetensors or GGUF, or add it again with `allow_pickle: true` + (`llmgw ctl model add -f`) if its files were checked. Look at the + models' sources before upgrading the nodes: one ending in + `.ckpt` or `.bin`, or a mirror version holding `pytorch_model.bin`. Upgrade the control planes first: a node on + this version refuses those weights until its control plane can say + `allow_pickle`. + +### Added + +- **Weights that run code are refused.** Python pickles — `.bin`, `.pt`, + `.pth`, `.ckpt` — run whatever they hold when loaded, which is how the + malicious models on Hugging Face work. The node refuses such a source, + or a mirror version with one in it, before fetching it; adding such a + model is refused; vLLM and SGLang downloading a model by its id read its + safetensors only; the mirror page marks the file. `allow_pickle: true` + on a model lets it through on purpose. BACKLOG M lists what is next: + checksums against Hugging Face everywhere, its own scan, trusted + publishers, engines without the internet. + ## v0.15.0 — 2026-10-25 The company's tools and editors through the gateway, and guardrails that diff --git a/api/gen/llmgw/v1/agent.pb.go b/api/gen/llmgw/v1/agent.pb.go index d9f42a66..5f25bc75 100644 --- a/api/gen/llmgw/v1/agent.pb.go +++ b/api/gen/llmgw/v1/agent.pb.go @@ -3790,7 +3790,10 @@ type EngineProfile struct { Adapters []*LoraAdapter `protobuf:"bytes,12,rep,name=adapters,proto3" json:"adapters,omitempty"` // kv_events has a vLLM engine publish its KV cache events to the agent, for // the router's precise index. - KvEvents bool `protobuf:"varint,13,opt,name=kv_events,json=kvEvents,proto3" json:"kv_events,omitempty"` + KvEvents bool `protobuf:"varint,13,opt,name=kv_events,json=kvEvents,proto3" json:"kv_events,omitempty"` + // allow_pickle lets the node fetch weights in a pickle format, which run + // code when loaded: the catalog's allow_pickle, set on purpose. + AllowPickle bool `protobuf:"varint,14,opt,name=allow_pickle,json=allowPickle,proto3" json:"allow_pickle,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -3916,6 +3919,13 @@ func (x *EngineProfile) GetKvEvents() bool { return false } +func (x *EngineProfile) GetAllowPickle() bool { + if x != nil { + return x.AllowPickle + } + return false +} + // RemoveWeights deletes cached model files a node no longer needs. // // The paths are computed by the control plane from the catalog, but the agent @@ -5462,7 +5472,7 @@ const file_llmgw_v1_agent_proto_rawDesc = "" + "profile_fp\x18\x0f \x01(\tR\tprofileFp\x12\x17\n" + "\aapi_key\x18\x10 \x01(\tR\x06apiKey\x12\x1f\n" + "\vcpu_threads\x18\x11 \x01(\x05R\n" + - "cpuThreads\"\xcd\x03\n" + + "cpuThreads\"\xf0\x03\n" + "\rEngineProfile\x12\x12\n" + "\x04name\x18\x01 \x01(\tR\x04name\x12\x16\n" + "\x06engine\x18\x02 \x01(\tR\x06engine\x12\x1b\n" + @@ -5478,7 +5488,8 @@ const file_llmgw_v1_agent_proto_rawDesc = "" + " \x01(\tR\tmodelPath\x12*\n" + "\x11served_model_name\x18\v \x01(\tR\x0fservedModelName\x121\n" + "\badapters\x18\f \x03(\v2\x15.llmgw.v1.LoraAdapterR\badapters\x12\x1b\n" + - "\tkv_events\x18\r \x01(\bR\bkvEvents\x1a6\n" + + "\tkv_events\x18\r \x01(\bR\bkvEvents\x12!\n" + + "\fallow_pickle\x18\x0e \x01(\bR\vallowPickle\x1a6\n" + "\bEnvEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"%\n" + diff --git a/api/proto/llmgw/v1/agent.proto b/api/proto/llmgw/v1/agent.proto index 8be40c78..4afe00da 100644 --- a/api/proto/llmgw/v1/agent.proto +++ b/api/proto/llmgw/v1/agent.proto @@ -749,6 +749,9 @@ message EngineProfile { // kv_events has a vLLM engine publish its KV cache events to the agent, for // the router's precise index. bool kv_events = 13; + // allow_pickle lets the node fetch weights in a pickle format, which run + // code when loaded: the catalog's allow_pickle, set on purpose. + bool allow_pickle = 14; } // RemoveWeights deletes cached model files a node no longer needs. diff --git a/docs/admin-api.openapi.json b/docs/admin-api.openapi.json index e4818d39..2b5fb867 100644 --- a/docs/admin-api.openapi.json +++ b/docs/admin-api.openapi.json @@ -9930,6 +9930,10 @@ "type": "string", "description": "Source is the weights path or a HuggingFace id." }, + "allow_pickle": { + "type": "boolean", + "description": "AllowPickle lets the nodes fetch this model's weights, or its adapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code as they are loaded. Off, such a source is refused when the model is added and by the node before it fetches anything. For a model that has no safetensors or GGUF form and whose files were checked." + }, "profiles": { "type": "array", "items": { @@ -10309,6 +10313,10 @@ "type": "string", "description": "HFRepo is the repository its node loads the weights by name from." }, + "allow_pickle": { + "type": "boolean", + "description": "AllowPickle is the member model's allow_pickle." + }, "workflow": { "type": "string", "description": "Workflow and Checkpoint are what a request for it is sent." diff --git a/docs/install.md b/docs/install.md index f7ccd7b9..0a2e59df 100644 --- a/docs/install.md +++ b/docs/install.md @@ -876,6 +876,24 @@ Nothing to do here if you built the mirror in step 3 — the agent fetches what model's `source` names, and that source is your mirror. This section is for the arrangements where the files are already on the machine. +**Weights in a pickle format are refused.** `.bin`, `.pt`, `.pth`, `.ckpt` and +`.pkl` are Python pickles, and loading one runs whatever it holds, as the +engine's user, on the node — which is how the malicious models found on Hugging +Face work. A model whose source is one, or a mirror version holding one, is +refused when it is added and by the node before it fetches anything; vLLM and +SGLang given a model by its Hugging Face id read only its safetensors. Take the +repository's safetensors or GGUF. A model that has no other form, and whose +files were checked, can say so: + +```yaml +- name: triposr + source: http://mirror:8081/triposr//model.ckpt + allow_pickle: true +``` + +A file put on the machine by hand, as a path, is the operator's own and is not +looked at. + Put them where the node can read them. `scp`, a shared filesystem, or the engine's own downloader are all valid, and which you want depends on how many nodes you have. diff --git a/docs/reference.md b/docs/reference.md index e01bd62b..04a5e8a6 100644 --- a/docs/reference.md +++ b/docs/reference.md @@ -188,6 +188,7 @@ rather than repeated. | `aliases` | `[]string` | [Model](#model) | | `description` | `string` | [Model](#model) | | `source` | `string` | [Model](#model) | +| `allow_pickle` | `bool` | [Model](#model) | | `profiles` | `[]Profile` | [Model](#model) | | `scaling` | `Scaling` | [Model](#model) | | `class` | `string` | [Model](#model) | @@ -1790,6 +1791,16 @@ choice by guesswork. Source is the weights path or a HuggingFace id. +### `allow_pickle` + +`bool` + +AllowPickle lets the nodes fetch this model's weights, or its +adapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code +as they are loaded. Off, such a source is refused when the model is +added and by the node before it fetches anything. For a model that has +no safetensors or GGUF form and whose files were checked. + ### `profiles` `[]Profile` diff --git a/internal/agent/agent.go b/internal/agent/agent.go index 91cac305..107e39a0 100644 --- a/internal/agent/agent.go +++ b/internal/agent/agent.go @@ -1124,7 +1124,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan< }, }) }) - if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath); err != nil { + if path, err := a.weights().Ensure(ctx, spec.Profile.ModelPath, spec.Profile.AllowPickle); err != nil { stop() report(supervisor.Failed, err.Error(), nil) return @@ -1145,7 +1145,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan< send(&llmgwv1.ShardStatus{State: agentsvc.StateToWire(supervisor.Starting), Fetch: &llmgwv1.FetchProgress{Source: p.Source, BytesDone: p.BytesDone, BytesTotal: p.BytesTotal}}) }) - path, err := a.weights().Ensure(ctx, m["source"]) + path, err := a.weights().Ensure(ctx, m["source"], m["allow_pickle"] == "true") stop() if err != nil { report(supervisor.Failed, fmt.Sprintf("model %s: %v", m["name"], err), nil) @@ -1161,7 +1161,7 @@ func (a *Agent) startShard(ctx context.Context, m *llmgwv1.StartShard, out chan< // loads them at start, and one missing is a replica that answers for the // base model under the adapter's name. for i, ad := range spec.Profile.Adapters { - path, err := a.weights().Ensure(ctx, ad.Path) + path, err := a.weights().Ensure(ctx, ad.Path, spec.Profile.AllowPickle) if err != nil { report(supervisor.Failed, fmt.Sprintf("adapter %s: %v", ad.Name, err), nil) return diff --git a/internal/agent/agent_test.go b/internal/agent/agent_test.go index 441bbf5c..04e8683d 100644 --- a/internal/agent/agent_test.go +++ b/internal/agent/agent_test.go @@ -671,7 +671,7 @@ func TestAShardWithMissingWeightsFailsBeforeLaunching(t *testing.T) { // nobody is watching — so it is caught here, where the message can say // what is actually wrong. a := &Agent{cfg: Config{WeightsCache: t.TempDir()}, log: slog.New(slog.DiscardHandler)} - _, err := a.weights().Ensure(t.Context(), filepath.Join(t.TempDir(), "not-there.gguf")) + _, err := a.weights().Ensure(t.Context(), filepath.Join(t.TempDir(), "not-there.gguf"), false) if err == nil { t.Fatal("a missing weights file was accepted") } diff --git a/internal/agent/engines/engine.go b/internal/agent/engines/engine.go index 35952a8d..cf086121 100644 --- a/internal/agent/engines/engine.go +++ b/internal/agent/engines/engine.go @@ -47,6 +47,9 @@ type Profile struct { // KVEvents has the engine publish its KV cache events to the agent, // for the router's precise index. vLLM only. KVEvents bool `json:"kv_events,omitempty"` + // AllowPickle lets the node fetch weights in a pickle format, which run + // code when loaded; the catalog's allow_pickle. + AllowPickle bool `json:"allow_pickle,omitempty"` } // ShardSpec is one shard of one replica on this node. diff --git a/internal/agent/engines/pickle_test.go b/internal/agent/engines/pickle_test.go new file mode 100644 index 00000000..3f91851f --- /dev/null +++ b/internal/agent/engines/pickle_test.go @@ -0,0 +1,22 @@ +package engines + +import ( + "slices" + "testing" +) + +func TestAModelDownloadedByTheEngineIsReadAsSafetensorsOnly(t *testing.T) { + for path, want := range map[string]bool{ + "Qwen/Qwen2.5-7B-Instruct": true, + "/var/lib/llmgw/weights/abc/": false, "./local": false, + "http://m/q/v/": false, "q.gguf": false, + } { + got := slices.Contains(safetensorsOnly(Profile{ModelPath: path}), "safetensors") + if got != want { + t.Errorf("%q: safetensors only = %v, want %v", path, got, want) + } + } + if len(safetensorsOnly(Profile{ModelPath: "Qwen/Qwen2.5-7B-Instruct", AllowPickle: true})) != 0 { + t.Error("allow_pickle still forced safetensors") + } +} diff --git a/internal/agent/engines/sglang.go b/internal/agent/engines/sglang.go index fa24a8c6..9befa66c 100644 --- a/internal/agent/engines/sglang.go +++ b/internal/agent/engines/sglang.go @@ -102,6 +102,7 @@ func (g *SGLang) serverCommand(s ShardSpec) (Command, error) { if p.ServedModelName != "" { args = append(args, "--served-model-name", p.ServedModelName) } + args = append(args, safetensorsOnly(p)...) if s.NNodes > 1 { args = append(args, "--nnodes", strconv.Itoa(s.NNodes), diff --git a/internal/agent/engines/unsafeenv.go b/internal/agent/engines/unsafeenv.go index c1d267b5..fb3a171f 100644 --- a/internal/agent/engines/unsafeenv.go +++ b/internal/agent/engines/unsafeenv.go @@ -2,6 +2,7 @@ package engines import ( "fmt" + "os" "strings" ) @@ -185,3 +186,28 @@ func UnsafeArgs(engine string, args []string) error { } return nil } + +// safetensorsOnly has vLLM or SGLang read only safetensors from a model it +// downloads itself, by its Hugging Face id: left to choose, either takes +// the repository's pickle files (pytorch_model.bin) when it has no +// safetensors, and loading those runs whatever they hold. A path was +// fetched by the agent, which checked it, or put there by the operator. +func safetensorsOnly(p Profile) []string { + if p.AllowPickle || !hubID(p.ModelPath) { + return nil + } + return []string{"--load-format", "safetensors"} +} + +// hubID reports whether a model path is a Hugging Face id, org/name, rather +// than a file or directory. +func hubID(s string) bool { + if s == "" || strings.HasPrefix(s, "/") || strings.HasPrefix(s, ".") || strings.Contains(s, "://") { + return false + } + if strings.Count(s, "/") != 1 { + return false + } + _, err := os.Stat(s) + return err != nil +} diff --git a/internal/agent/engines/vllm.go b/internal/agent/engines/vllm.go index f0e12f21..b69c9594 100644 --- a/internal/agent/engines/vllm.go +++ b/internal/agent/engines/vllm.go @@ -119,6 +119,7 @@ func (v *VLLM) serverCommand(s ShardSpec) (Command, error) { if p.ServedModelName != "" { args = append(args, "--served-model-name", p.ServedModelName) } + args = append(args, safetensorsOnly(p)...) env := baseEnv(s) diff --git a/internal/agent/weights/fetch.go b/internal/agent/weights/fetch.go index 63e48919..962d2057 100644 --- a/internal/agent/weights/fetch.go +++ b/internal/agent/weights/fetch.go @@ -90,14 +90,20 @@ var ErrNotLocal = errors.New("the weights are not on this node and the source sa // already cached, downloaded, or downloaded by another shard this one waited // for — since that is most of the difference between a replica that starts // in a minute and one that takes an hour. -func (f *Fetcher) Ensure(ctx context.Context, source string) (path string, err error) { +// +// A source in a pickle format, or a set with such a file in it, is refused +// before anything is fetched unless allowPickle; see Pickled. +func (f *Fetcher) Ensure(ctx context.Context, source string, allowPickle bool) (path string, err error) { ctx, span := obs.Tracer(tracerName).Start(ctx, "fetch weights", trace.WithAttributes(attribute.String("llmgw.weights.source", redact(source)))) defer func() { span.SetAttributes(attribute.String("llmgw.weights.path", path)) obs.End(span, err) }() - return f.ensure(ctx, source) + if !allowPickle && !IsSet(source) && Pickled(source) { + return "", refusePickle(source) + } + return f.ensure(ctx, source, allowPickle) } // tracerName names the spans this package opens. @@ -119,7 +125,7 @@ func redact(source string) string { return u.String() } -func (f *Fetcher) ensure(ctx context.Context, source string) (string, error) { +func (f *Fetcher) ensure(ctx context.Context, source string, allowPickle bool) (string, error) { if source == "" { return "", fmt.Errorf("no source given") } @@ -127,7 +133,7 @@ func (f *Fetcher) ensure(ctx context.Context, source string) (string, error) { // file, and a set has to be handled before any of it. if IsSet(source) { outcome(ctx, "set") - return f.ensureSet(ctx, source) + return f.ensureSet(ctx, source, allowPickle) } if !isRemote(source) { if !isPath(source) { diff --git a/internal/agent/weights/fetch_test.go b/internal/agent/weights/fetch_test.go index 82bed689..ed172772 100644 --- a/internal/agent/weights/fetch_test.go +++ b/internal/agent/weights/fetch_test.go @@ -37,7 +37,7 @@ func TestALocalPathIsUsedNeverCopied(t *testing.T) { } f := &Fetcher{CacheDir: t.TempDir()} - got, err := f.Ensure(t.Context(), path) + got, err := f.Ensure(t.Context(), path, false) if err != nil { t.Fatal(err) } @@ -53,7 +53,7 @@ func TestAMissingLocalPathSaysWhatIsWrong(t *testing.T) { // The commonest setup mistake, and "no such file" from an engine twenty // minutes into a load is a poor way to learn it. f := &Fetcher{CacheDir: t.TempDir()} - _, err := f.Ensure(t.Context(), filepath.Join(t.TempDir(), "absent.gguf")) + _, err := f.Ensure(t.Context(), filepath.Join(t.TempDir(), "absent.gguf"), false) if !errors.Is(err, ErrNotLocal) { t.Fatalf("err = %v, want ErrNotLocal", err) } @@ -67,7 +67,7 @@ func TestARemoteFileIsFetchedOnceAndReused(t *testing.T) { srv := serveBytes(t, []byte("the weights"), &hits) f := &Fetcher{CacheDir: t.TempDir()} - first, err := f.Ensure(t.Context(), srv.URL+"/model.gguf") + first, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false) if err != nil { t.Fatal(err) } @@ -76,7 +76,7 @@ func TestARemoteFileIsFetchedOnceAndReused(t *testing.T) { t.Fatalf("fetched content = %q, err %v", body, err) } - second, err := f.Ensure(t.Context(), srv.URL+"/model.gguf") + second, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false) if err != nil { t.Fatal(err) } @@ -103,7 +103,7 @@ func TestConcurrentRequestsProduceOneDownload(t *testing.T) { wg.Add(1) go func() { defer wg.Done() - paths[i], errs[i] = f.Ensure(t.Context(), srv.URL+"/model.gguf") + paths[i], errs[i] = f.Ensure(t.Context(), srv.URL+"/model.gguf", false) }() } wg.Wait() @@ -139,7 +139,7 @@ func TestAPartialDownloadIsNeverVisibleAsTheModel(t *testing.T) { cache := t.TempDir() f := &Fetcher{CacheDir: cache} - if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf"); err == nil { + if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false); err == nil { t.Fatal("a truncated download was reported as success") } entries, _ := os.ReadDir(cache) @@ -169,7 +169,7 @@ func TestAnInterruptedDownloadResumes(t *testing.T) { t.Fatal(err) } - got, err := f.Ensure(t.Context(), srv.URL+"/model.gguf") + got, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false) if err != nil { t.Fatal(err) } @@ -232,7 +232,7 @@ func TestProgressIsReportedWithAPercentage(t *testing.T) { CacheDir: t.TempDir(), OnProgress: func(p Progress) { seen = append(seen, p) }, } - if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf"); err != nil { + if _, err := f.Ensure(t.Context(), srv.URL+"/model.gguf", false); err != nil { t.Fatal(err) } if len(seen) == 0 { @@ -254,7 +254,7 @@ func TestPercentIsUnknownRatherThanZeroWithoutATotal(t *testing.T) { func TestNoCacheDirectoryIsAClearRefusal(t *testing.T) { f := &Fetcher{} - _, err := f.Ensure(t.Context(), "https://example.invalid/model.gguf") + _, err := f.Ensure(t.Context(), "https://example.invalid/model.gguf", false) if err == nil || !strings.Contains(err.Error(), "cache directory") { t.Errorf("err = %v, want a complaint about the cache directory", err) } @@ -269,7 +269,7 @@ func TestABareModelIdIsLeftToTheEngine(t *testing.T) { "Qwen/Qwen2.5-7B-Instruct", "meta-llama/Llama-3.3-70B-Instruct", } { - got, err := f.Ensure(t.Context(), id) + got, err := f.Ensure(t.Context(), id, false) if err != nil { t.Errorf("%s was refused: %v", id, err) } @@ -283,7 +283,7 @@ func TestAnAbsolutePathThatIsMissingIsStillAnError(t *testing.T) { // The other half: an operator writing a path means a file, and a missing // one should be said plainly rather than handed to an engine. f := &Fetcher{CacheDir: t.TempDir()} - if _, err := f.Ensure(t.Context(), "/models/definitely-not-here.gguf"); !errors.Is(err, ErrNotLocal) { + if _, err := f.Ensure(t.Context(), "/models/definitely-not-here.gguf", false); !errors.Is(err, ErrNotLocal) { t.Errorf("err = %v, want ErrNotLocal", err) } } diff --git a/internal/agent/weights/pickle.go b/internal/agent/weights/pickle.go new file mode 100644 index 00000000..f66610cb --- /dev/null +++ b/internal/agent/weights/pickle.go @@ -0,0 +1,41 @@ +package weights + +import ( + "errors" + "fmt" + "path" + "strings" +) + +// Weights in Python's pickle format are a program as much as a file: loading +// one runs whatever it says, as the engine's user, on the node. That is how +// the malicious models found on Hugging Face work, and nothing in the file's +// name or size says so. safetensors and GGUF are only tensors and are read, +// never run, so those are what a node fetches unless the catalog says +// otherwise for a model, with allow_pickle, on purpose. + +// pickleExts are the extensions of PyTorch's pickle formats, and of pickle +// itself. A .bin beside safetensors is the same model in pickle form. +var pickleExts = []string{".bin", ".pt", ".pth", ".ckpt", ".pkl", ".pickle"} + +// Pickled reports whether a file name or URL is in a pickle format. +func Pickled(name string) bool { + if i := strings.IndexAny(name, "?#"); i >= 0 && strings.Contains(name, "://") { + name = name[:i] + } + ext := strings.ToLower(path.Ext(name)) + for _, e := range pickleExts { + if ext == e { + return true + } + } + return false +} + +// ErrPickle is a source refused for being in a pickle format. +var ErrPickle = errors.New("weights in a pickle format run code when they are loaded") + +func refusePickle(name string) error { + return fmt.Errorf("%w: %s is refused; use the model's safetensors or GGUF, or set allow_pickle: true "+ + "on the model if it was checked and has no other form", ErrPickle, redact(name)) +} diff --git a/internal/agent/weights/pickle_test.go b/internal/agent/weights/pickle_test.go new file mode 100644 index 00000000..eb9d50f3 --- /dev/null +++ b/internal/agent/weights/pickle_test.go @@ -0,0 +1,53 @@ +package weights + +import ( + "errors" + "sync/atomic" + "testing" +) + +func TestPickleFormatsAreKnownByTheirNames(t *testing.T) { + for name, want := range map[string]bool{ + "pytorch_model.bin": true, "model.ckpt": true, "x.PT": true, "w.pth": true, "a.pkl": true, + "https://h/r/resolve/main/model.ckpt?download=true": true, + "model.safetensors": false, "m-Q4_K_M.gguf": false, "Qwen/Qwen2.5-7B-Instruct": false, + "config.json": false, "": false, + } { + if got := Pickled(name); got != want { + t.Errorf("Pickled(%q) = %v, want %v", name, got, want) + } + } +} + +func TestAPickleFileIsRefusedBeforeItIsFetched(t *testing.T) { + var hits atomic.Int64 + srv := serveBytes(t, []byte("not really a pickle"), &hits) + f := &Fetcher{CacheDir: t.TempDir()} + + _, err := f.Ensure(t.Context(), srv.URL+"/model.ckpt", false) + if !errors.Is(err, ErrPickle) { + t.Fatalf("a .ckpt was not refused: %v", err) + } + if hits.Load() != 0 { + t.Errorf("the refused file was fetched %d times", hits.Load()) + } + if _, err := f.Ensure(t.Context(), srv.URL+"/model.ckpt", true); err != nil { + t.Errorf("allowed, the .ckpt was still refused: %v", err) + } +} + +func TestASetWithAPickleFileIsRefusedWhole(t *testing.T) { + f, srv, hits := mirrorOf(t, map[string]string{ + "pytorch_model.bin": "pickle", + "config.json": "{}", + }) + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); !errors.Is(err, ErrPickle) { + t.Fatalf("a set holding pytorch_model.bin was not refused: %v", err) + } + if *hits != 0 { + t.Errorf("%d files of the refused set were fetched", *hits) + } + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", true); err != nil { + t.Errorf("allowed, the set was still refused: %v", err) + } +} diff --git a/internal/agent/weights/set.go b/internal/agent/weights/set.go index e0ebf765..d1fd3693 100644 --- a/internal/agent/weights/set.go +++ b/internal/agent/weights/set.go @@ -46,7 +46,7 @@ type manifest struct { // The whole set or nothing. A directory with three of four shards in it loads // as a broken model rather than failing to load, so a partial set is left // unfinished under its temporary name and the next attempt starts again. -func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error) { +func (f *Fetcher) ensureSet(ctx context.Context, source string, allowPickle bool) (string, error) { dir, err := f.cachePath(strings.TrimSuffix(source, "/")) if err != nil { return "", err @@ -66,6 +66,15 @@ func (f *Fetcher) ensureSet(ctx context.Context, source string) (string, error) if len(m.Files) == 0 { return "", fmt.Errorf("%s holds no files", source) } + // Refused whole rather than skipped: an engine given the directory + // loads the pickle when there is no safetensors beside it. + if !allowPickle { + for _, file := range m.Files { + if Pickled(file.Name) { + return "", refusePickle(source + file.Name) + } + } + } if err := os.MkdirAll(dir, 0o755); err != nil { return "", err diff --git a/internal/agent/weights/set_test.go b/internal/agent/weights/set_test.go index 93e9063e..f32e0523 100644 --- a/internal/agent/weights/set_test.go +++ b/internal/agent/weights/set_test.go @@ -53,7 +53,7 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) { "config.json": "{}", }) - got, err := f.Ensure(t.Context(), srv.URL+"/m/v/") + got, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false) if err != nil { t.Fatal(err) } @@ -74,11 +74,11 @@ func TestASetIsFetchedIntoOneDirectory(t *testing.T) { func TestAFinishedSetIsNotFetchedAgain(t *testing.T) { f, srv, hits := mirrorOf(t, map[string]string{"a.gguf": "aaaa", "b.gguf": "bb"}) - if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil { + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil { t.Fatal(err) } after := *hits - if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil { + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil { t.Fatal(err) } if *hits != after { @@ -100,7 +100,7 @@ func TestAHalfFetchedSetIsNotTreatedAsFinished(t *testing.T) { t.Fatal(err) } - got, err := f.Ensure(t.Context(), srv.URL+"/m/v/") + got, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false) if err != nil { t.Fatal(err) } @@ -117,7 +117,7 @@ func TestAFileAlreadyThereAtTheRightSizeIsNotRefetched(t *testing.T) { _ = os.MkdirAll(dir, 0o755) _ = os.WriteFile(filepath.Join(dir, "a.gguf"), []byte("aaaa"), 0o644) - if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil { + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil { t.Fatal(err) } // Resuming a set means not paying for the shards that already landed. @@ -133,7 +133,7 @@ func TestAFilenameThatEscapesTheDirectoryIsRefused(t *testing.T) { defer srv.Close() f := &Fetcher{CacheDir: t.TempDir()} - _, err := f.Ensure(t.Context(), srv.URL+"/m/v/") + _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false) // The mirror is a place an operator copies files into by hand, so what it // reports is not automatically safe to join onto a path. if err == nil || !strings.Contains(err.Error(), "offered a file named") { @@ -148,7 +148,7 @@ func TestADirectoryListingIsNotAManifest(t *testing.T) { defer srv.Close() f := &Fetcher{CacheDir: t.TempDir()} - _, err := f.Ensure(t.Context(), srv.URL+"/m/v/") + _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false) // "invalid character '<'" sends somebody looking at the wrong thing. if err == nil || !strings.Contains(err.Error(), "has to be a mirror version") { t.Errorf("the reason should name the cause, got %v", err) @@ -182,12 +182,12 @@ func TestASetFileThatDoesNotMatchTheMirrorsChecksumIsRefused(t *testing.T) { })) t.Cleanup(srv.Close) f := &Fetcher{CacheDir: t.TempDir()} - if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err == nil || !strings.Contains(err.Error(), "sha256") { + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err == nil || !strings.Contains(err.Error(), "sha256") { t.Fatalf("a damaged shard was accepted: %v", err) } served = good // The size differs, so it is fetched again; this time it matches. - if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/"); err != nil { + if _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false); err != nil { t.Fatalf("the shard as mirrored: %v", err) } } @@ -198,7 +198,7 @@ func TestASetStillDownloadingOnTheMirrorSaysSo(t *testing.T) { })) t.Cleanup(srv.Close) f := &Fetcher{CacheDir: t.TempDir()} - _, err := f.Ensure(t.Context(), srv.URL+"/m/v/") + _, err := f.Ensure(t.Context(), srv.URL+"/m/v/", false) if err == nil || !strings.Contains(err.Error(), "not complete on the mirror yet") || !strings.Contains(err.Error(), "model-00002-of-00002") { t.Fatalf("got %v", err) diff --git a/internal/agent/weights/trace_test.go b/internal/agent/weights/trace_test.go index 0fc5242b..1a273657 100644 --- a/internal/agent/weights/trace_test.go +++ b/internal/agent/weights/trace_test.go @@ -13,7 +13,7 @@ func TestAFetchSaysInItsTraceWhetherItDownloadedAndHowMuch(t *testing.T) { f := &Fetcher{CacheDir: t.TempDir()} src := strings.Replace(srv.URL, "http://", "http://user:secret@", 1) + "/m/model.gguf?token=hunter2" - if _, err := f.Ensure(t.Context(), src); err != nil { + if _, err := f.Ensure(t.Context(), src, false); err != nil { t.Fatal(err) } fetch := rec.One(t, [16]byte{}, "fetch weights") @@ -30,7 +30,7 @@ func TestAFetchSaysInItsTraceWhetherItDownloadedAndHowMuch(t *testing.T) { } // The second start on this node finds it here. - if _, err := f.Ensure(t.Context(), src); err != nil { + if _, err := f.Ensure(t.Context(), src, false); err != nil { t.Fatal(err) } again := rec.Named("fetch weights") diff --git a/internal/control/agentsvc/convert.go b/internal/control/agentsvc/convert.go index d386cc55..a045bcda 100644 --- a/internal/control/agentsvc/convert.go +++ b/internal/control/agentsvc/convert.go @@ -201,8 +201,9 @@ func StartShardFromCommand(cmd reconciler.ShardCommand) *llmgwv1.StartShard { GpusPerNode: int32(p.GPUsPerNode), Nodes: int32(p.Nodes), Args: p.Args, Env: p.Env, ModelPath: p.ModelPath, ServedModelName: p.ServedModelName, - Adapters: adaptersToWire(p.Adapters), - KvEvents: p.KVEvents, + Adapters: adaptersToWire(p.Adapters), + KvEvents: p.KVEvents, + AllowPickle: p.AllowPickle, }, StartupTimeoutMs: cmd.StartupTimeout.Milliseconds(), Pool: cmd.Pool, @@ -238,8 +239,9 @@ func ShardSpecFromStart(m *llmgwv1.StartShard) (engines.ShardSpec, time.Duration GPUsPerNode: int(p.GetGpusPerNode()), Nodes: int(p.GetNodes()), Args: p.GetArgs(), Env: p.GetEnv(), ModelPath: p.GetModelPath(), ServedModelName: p.GetServedModelName(), - Adapters: adaptersFromWire(p.GetAdapters()), - KVEvents: p.GetKvEvents(), + Adapters: adaptersFromWire(p.GetAdapters()), + KVEvents: p.GetKvEvents(), + AllowPickle: p.GetAllowPickle(), }, } return spec, time.Duration(m.GetStartupTimeoutMs()) * time.Millisecond diff --git a/internal/control/catalog/catalog.go b/internal/control/catalog/catalog.go index 528c8384..964863a3 100644 --- a/internal/control/catalog/catalog.go +++ b/internal/control/catalog/catalog.go @@ -323,9 +323,15 @@ type Model struct { // choice by guesswork. Description string `json:"description,omitempty" koanf:"description"` // Source is the weights path or a HuggingFace id. - Source string `json:"source,omitempty" koanf:"source"` - Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"` - Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"` + Source string `json:"source,omitempty" koanf:"source"` + // AllowPickle lets the nodes fetch this model's weights, or its + // adapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code + // as they are loaded. Off, such a source is refused when the model is + // added and by the node before it fetches anything. For a model that has + // no safetensors or GGUF form and whose files were checked. + AllowPickle bool `json:"allow_pickle,omitempty" koanf:"allow_pickle"` + Profiles []Profile `json:"profiles,omitempty" koanf:"profiles"` + Scaling Scaling `json:"scaling,omitempty" koanf:"scaling"` // Class is how much this model's capacity is worth against another's, // when there is not enough for both: critical, standard or opportunistic. // Empty is standard, or opportunistic for a model loaded on demand. See @@ -1336,10 +1342,16 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) { Name string `json:"name"` Source string `json:"source"` HFRepo string `json:"hf_repo,omitempty"` + // A string: the node reads the members as strings. + AllowPickle string `json:"allow_pickle,omitempty"` } var ms []member for _, sm := range m.Shared.Members { - ms = append(ms, member{Name: sm.Name, Source: sm.Source, HFRepo: sm.HFRepo}) + mb := member{Name: sm.Name, Source: sm.Source, HFRepo: sm.HFRepo} + if sm.AllowPickle { + mb.AllowPickle = "true" + } + ms = append(ms, mb) } b, _ := json.Marshal(ms) env = maps.Clone(env) @@ -1369,6 +1381,7 @@ func (m Model) EngineProfile(name string) (engines.Profile, bool) { ServedModelName: m.Name, Adapters: m.engineAdapters(), KVEvents: p.KVEvents, + AllowPickle: m.AllowPickle, }.WithRuntimeLoRA(m.MaxLoadedAdapters), true } return engines.Profile{}, false diff --git a/internal/control/catalog/pickle.go b/internal/control/catalog/pickle.go new file mode 100644 index 00000000..85e0e275 --- /dev/null +++ b/internal/control/catalog/pickle.go @@ -0,0 +1,47 @@ +package catalog + +import ( + "fmt" + "slices" + + "gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights" +) + +// PickleSources are the sources of a model's weights and adapters that are +// in a pickle format, which run code as they are loaded (weights.Pickled). +func (m Model) PickleSources() []string { + var out []string + add := func(s string) { + if s != "" && weights.Pickled(s) && !slices.Contains(out, s) { + out = append(out, s) + } + } + add(m.Source) + for _, p := range m.Profiles { + add(p.Source) + } + for _, a := range m.Adapters { + add(a.Source) + } + return out +} + +// CheckPickle refuses a model whose weights are in a pickle format, unless +// it says allow_pickle. Asked when a model is added or its sources change, +// not when the catalog is built: a model already running is refused by its +// node, with the same words, rather than dropped from the catalog. +// Sources in except were there before and are not asked about again. +func (m Model) CheckPickle(except ...string) error { + if m.AllowPickle { + return nil + } + for _, s := range m.PickleSources() { + if slices.Contains(except, s) { + continue + } + return fmt.Errorf("model %q: %s is in a pickle format, which runs code when it is loaded; "+ + "use the model's safetensors or GGUF, or set allow_pickle: true if it was checked and has no other form", + m.Name, s) + } + return nil +} diff --git a/internal/control/catalog/pickle_test.go b/internal/control/catalog/pickle_test.go new file mode 100644 index 00000000..ca0da0dd --- /dev/null +++ b/internal/control/catalog/pickle_test.go @@ -0,0 +1,30 @@ +package catalog + +import ( + "strings" + "testing" +) + +func TestAModelInAPickleFormatIsRefusedUnlessItSaysSo(t *testing.T) { + m := Model{Name: "triposr", Source: "hf:stabilityai/TripoSR/model.ckpt", + Adapters: []Adapter{{Name: "a", Source: "http://m/a/adapter_model.bin"}}} + err := m.CheckPickle() + if err == nil || !strings.Contains(err.Error(), "model.ckpt") || !strings.Contains(err.Error(), "allow_pickle") { + t.Fatalf("not refused, or not said why: %v", err) + } + if got := m.PickleSources(); len(got) != 2 { + t.Errorf("pickle sources %v, want the model's and the adapter's", got) + } + // A source that was there before is not asked about again: changing + // the scaling of a model that runs does not need allow_pickle. + if err := m.CheckPickle(m.PickleSources()...); err != nil { + t.Errorf("sources already there were refused: %v", err) + } + m.AllowPickle = true + if err := m.CheckPickle(); err != nil { + t.Errorf("allow_pickle was not honoured: %v", err) + } + if err := (Model{Name: "q", Source: "http://m/q/v/q-Q4_K_M.gguf"}).CheckPickle(); err != nil { + t.Errorf("a GGUF was refused: %v", err) + } +} diff --git a/internal/control/catalog/shared.go b/internal/control/catalog/shared.go index 51ced8a8..3cb88238 100644 --- a/internal/control/catalog/shared.go +++ b/internal/control/catalog/shared.go @@ -34,6 +34,8 @@ type SharedMember struct { Source string `json:"source"` // HFRepo is the repository its node loads the weights by name from. HFRepo string `json:"hf_repo,omitempty"` + // AllowPickle is the member model's allow_pickle. + AllowPickle bool `json:"allow_pickle,omitempty"` // Workflow and Checkpoint are what a request for it is sent. Workflow string `json:"workflow"` Checkpoint string `json:"checkpoint"` @@ -80,7 +82,7 @@ func envModel(env Environment, members []Model) Model { hf = m.ComfyUI.HuggingFaceRepo } shared.Members = append(shared.Members, SharedMember{Name: m.Name, Aliases: m.Aliases, - Source: m.Source, HFRepo: hf, Workflow: workflow, Checkpoint: ckpt}) + Source: m.Source, HFRepo: hf, AllowPickle: m.AllowPickle, Workflow: workflow, Checkpoint: ckpt}) idle = max(idle, m.Scaling.IdleTimeout) minN, maxN = max(minN, m.Scaling.Min), max(maxN, m.Scaling.Max) if ClassRank(m.EffectiveClass()) > ClassRank(class) || class == "" { diff --git a/internal/control/modelsvc/modelsvc.go b/internal/control/modelsvc/modelsvc.go index e1bb55c0..3f003f83 100644 --- a/internal/control/modelsvc/modelsvc.go +++ b/internal/control/modelsvc/modelsvc.go @@ -154,6 +154,9 @@ func (s *Service) add(ctx context.Context, m catalog.Model, by string, onlyNew b next = append(next, e) } next = append(next, m) + if err := m.CheckPickle(); err != nil { + return err + } built, err := catalog.New(next) if err != nil { @@ -406,9 +409,13 @@ func (s *Service) update(ctx context.Context, name, by, what string, change func found := false for _, e := range stored { if e.Name == name { + before := e.PickleSources() if err := change(&e); err != nil { return catalog.Model{}, err } + if err := e.CheckPickle(before...); err != nil { + return catalog.Model{}, err + } found = true } next = append(next, e) diff --git a/internal/docgen/reference_gen.go b/internal/docgen/reference_gen.go index 8ac2744b..b0e52afd 100644 --- a/internal/docgen/reference_gen.go +++ b/internal/docgen/reference_gen.go @@ -240,6 +240,7 @@ var Reference = []Block{ {Key: "aliases", Type: "[]string", Doc: "", Nested: ""}, {Key: "description", Type: "string", Doc: "Description is what this model is for, in a line.\n\nWritten by whoever added it, because nobody else can say it. A registry\noffers a pipeline tag and a wall of README, and neither answers the\nquestion a person picking from a list is asking: which of these nine\nnames should I send this request to.\n\nIt reaches clients through /v1/models, which is the only view most of\nthem ever see. A gateway serving nine models by name alone makes that\nchoice by guesswork.", Nested: ""}, {Key: "source", Type: "string", Doc: "Source is the weights path or a HuggingFace id.", Nested: ""}, + {Key: "allow_pickle", Type: "bool", Doc: "AllowPickle lets the nodes fetch this model's weights, or its\nadapters', in a pickle format (.bin, .pt, .pth, .ckpt), which run code\nas they are loaded. Off, such a source is refused when the model is\nadded and by the node before it fetches anything. For a model that has\nno safetensors or GGUF form and whose files were checked.", Nested: ""}, {Key: "profiles", Type: "[]Profile", Doc: "", Nested: ""}, {Key: "scaling", Type: "Scaling", Doc: "", Nested: "catalog.Scaling"}, {Key: "class", Type: "string", Doc: "Class is how much this model's capacity is worth against another's,\nwhen there is not enough for both: critical, standard or opportunistic.\nEmpty is standard, or opportunistic for a model loaded on demand. See\nEffectiveClass.", Nested: ""}, diff --git a/internal/web/admin_mirror.go b/internal/web/admin_mirror.go index c1c41aa8..ae7175ec 100644 --- a/internal/web/admin_mirror.go +++ b/internal/web/admin_mirror.go @@ -14,6 +14,7 @@ import ( "sync" "time" + "gitea.mynodes.xyz/ilker/llmgw/internal/agent/weights" "gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi" "gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog" "gitea.mynodes.xyz/ilker/llmgw/internal/gguf" @@ -876,6 +877,11 @@ func defaultModelName(repoID string) string { return hf.ModelName(repoID) } // verdictFor judges a registry file set by the repository's architecture. A // vision projector is its own architecture whatever the repository says. func (p *AccountPages) verdictFor(l *Lang, arch, name string) gguf.Verdict { + // Not refused here — the mirror keeps what it is given — but a node + // refuses to fetch it for a model without allow_pickle. + if weights.Pickled(name) { + return gguf.Verdict{Checked: true, Reason: l.T("card.pickle")} + } if strings.Contains(strings.ToLower(name), "mmproj") { return gguf.LlamaCpp(p.llamaVersion, "clip") } diff --git a/internal/web/locales/en/mirror.json b/internal/web/locales/en/mirror.json index 19a4f85c..dec737bf 100644 --- a/internal/web/locales/en/mirror.json +++ b/internal/web/locales/en/mirror.json @@ -137,6 +137,7 @@ "card.model": "model", "card.cannot_serve": "cannot be served", "card.imatrix": "it is an importance matrix for quantising, not a model", + "card.pickle": "it is in a pickle format, which runs code when it is loaded; nodes refuse it unless the model says allow_pickle — take the safetensors or GGUF", "card.fetch": "Fetch", "card.weights_note": "Only the weights are listed; a repository is mostly a README and a tokenizer. A safetensors set is fetched with the files vLLM reads beside it — config.json, the tokenizer, the shard index — and never with code. Shards of one file are one row and one button — a fifteen-shard model is one decision, not fifteen. Two separate files are two rows because they are usually alternatives: one quantisation or the other, not both.", "card.pinned_note": "The source URLs are pinned to %s rather than to a branch, so what arrives is what is listed here even if the branch moves in between — and a single file's checksum above is what the registry publishes, to compare with what the mirror computes once it has it.", diff --git a/internal/web/locales/tr/mirror.json b/internal/web/locales/tr/mirror.json index ab8d1a53..a065c07a 100644 --- a/internal/web/locales/tr/mirror.json +++ b/internal/web/locales/tr/mirror.json @@ -132,6 +132,7 @@ "card.model": "model", "card.cannot_serve": "sunulamaz", "card.imatrix": "bu bir model değil, nicemleme için bir önem matrisi", + "card.pickle": "pickle biçiminde; yüklenirken kod çalıştırır. Model allow_pickle demedikçe düğümler onu almaz — safetensors ya da GGUF olanı seçin", "card.fetch": "İndir", "card.weights_note": "Yalnızca ağırlıklar listelenir; bir deponun çoğu bir README ve bir tokenizer'dır. Bir safetensors kümesi, vLLM'in yanında okuduğu dosyalarla — config.json, tokenizer, parça dizini — birlikte indirilir, asla kodla birlikte değil. Bir dosyanın parçaları tek satır ve tek düğmedir — on beş parçalı bir model on beş değil, tek bir karardır. Ayrı iki dosya iki satırdır, çünkü genellikle birbirinin seçeneğidir: ya bir nicemleme ya öteki, ikisi birden değil.", "card.pinned_note": "Kaynak URL'ler bir dala değil %s sürümüne sabitlenmiştir; böylece dal arada değişse bile gelen, burada listelenendir — yukarıdaki tek dosya sağlama toplamı da model deposunun yayımladığıdır ve ayna dosyayı aldığında hesapladığıyla karşılaştırılabilir.",