Weights that run code are refused
Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when loaded, as the engine's user, on the node; that is how the malicious models found on Hugging Face work. The node refuses such a source, or a mirror set holding one, before fetching anything. A model added with one, or given one, is refused, while one already in the catalog still loads and is refused by its node with the same words. vLLM and SGLang downloading a model by its id get --load-format safetensors. The mirror page marks the file. allow_pickle: true on a model lets it through on purpose, carried to the node in EngineProfile.allow_pickle. BACKLOG M lists the rest: checksums against Hugging Face on every path, its own scan, trusted publishers, engines without the internet, an eval before an untrusted model serves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
6752938e6d
commit
4466d2d15a
31 files changed
+447
-44
No files matched your search
+56
@@ -1784,6 +1784,62 @@ A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
|
||||
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
|
||||
container, and what Kong's token accounting sees.
|
||||
|
||||
## M. Models from Hugging Face we can trust (written 2026-10-25)
|
||||
|
||||
The user asked how we know a model fetched from Hugging Face is safe. Two
|
||||
questions: whether its files can run code on our machines, which we can
|
||||
close, and whether the model itself was trained to misbehave, which no scan
|
||||
proves — only who published it, an eval before it serves, and the
|
||||
guardrails narrow that. What existed: `--trust-remote-code` refused, no
|
||||
repository Python fetched, ComfyUI nodes pinned to a commit, engines
|
||||
hash-pinned and run as `llmgw-engine`, mirror versions pinned to a commit
|
||||
and never overwritten. What did not: everything below.
|
||||
|
||||
### M1. Weights in a pickle format refused (S) — done 2026-10-25
|
||||
`.bin`, `.pt`, `.pth`, `.ckpt`, `.pkl` run code as they are loaded, which is
|
||||
how the malicious models found on Hugging Face work. The node refuses such a
|
||||
source, or a mirror set holding one, before fetching anything; a model being
|
||||
added or given such a source is refused; vLLM and SGLang downloading a model
|
||||
by its id are given `--load-format safetensors`; the mirror page says so on
|
||||
the file. `allow_pickle: true` on a model lets it through, on purpose. On the
|
||||
PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that
|
||||
needs it.
|
||||
|
||||
### M2. Every download pinned and checked against Hugging Face (M)
|
||||
- `hf:org/repo/file` follows `main` and is checked by nothing; a bare
|
||||
`org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is.
|
||||
Both: resolve to the commit when the model is added, keep it in the
|
||||
catalog's origin, and fetch that commit only (`--revision` for vLLM).
|
||||
- The sha256 Hugging Face publishes for each LFS file compared on every
|
||||
path, and a file that differs removed: `mirror add`, the manager, the
|
||||
node's `hf:` and http sources, mirror set files with no sha recorded.
|
||||
- Done when: a file changed on Hugging Face after a model was added is
|
||||
refused by the node with both checksums named.
|
||||
|
||||
### M3. Hugging Face's own scan (S)
|
||||
Hugging Face scans every file (pickle imports, ClamAV, partners' scanners)
|
||||
and marks a file unsafe. Ask for it with the repository listing; refuse an
|
||||
unsafe file, show the result on the mirror page and in the catalog's origin.
|
||||
The API field has to be confirmed first.
|
||||
|
||||
### M4. Publishers we trust, and the rest asked about (S)
|
||||
An allow-list of organisations in the cluster's settings (Qwen, meta-llama,
|
||||
mistralai, google, microsoft, …). A model from anyone else is added only by
|
||||
an admin who says so, and who said it is kept in the origin. The PoC already
|
||||
serves four community models whose training nobody here checked.
|
||||
|
||||
### M5. Engines without the internet (M)
|
||||
The firewall is inbound only, so an engine that was made to run code can
|
||||
send what it reads anywhere. With weights from the mirror the engines need
|
||||
no egress: an nft output rule for the engine user (allow the mirror, the
|
||||
cluster, loopback), systemd hardening of the agent's engines
|
||||
(`NoNewPrivileges`, `ProtectSystem`, private /tmp). Agent itself still root.
|
||||
|
||||
### M6. An eval before a new model serves (S)
|
||||
`eval` and `canary` exist; make a new model from an untrusted publisher
|
||||
(M4) serve only after its eval suite passes, so a model trained to answer
|
||||
wrongly on a trigger has at least been asked the suite.
|
||||
|
||||
## D. Facts worth knowing before starting
|
||||
|
||||
- Admin API: `127.0.0.1:9090`, behind the token in
|
||||
|
||||
Reference in new issue
Block a user