Weights that run code are refused
CI / test (push) Successful in 8m57s

Python pickles (.bin, .pt, .pth, .ckpt, .pkl) run whatever they hold when
loaded, as the engine's user, on the node; that is how the malicious models
found on Hugging Face work. The node refuses such a source, or a mirror set
holding one, before fetching anything. A model added with one, or given
one, is refused, while one already in the catalog still loads and is
refused by its node with the same words. vLLM and SGLang downloading a
model by its id get --load-format safetensors. The mirror page marks the
file. allow_pickle: true on a model lets it through on purpose, carried to
the node in EngineProfile.allow_pickle.

BACKLOG M lists the rest: checksums against Hugging Face on every path,
its own scan, trusted publishers, engines without the internet, an eval
before an untrusted model serves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 23:17:22 +01:00
1 parent 6752938e6d
commit 4466d2d15a
31 files changed
+447 -44

No files matched your search

+56
View File
@@ -1784,6 +1784,62 @@ A customer that already runs Kong puts llmgw behind it: `ai-proxy` with
route. A page in docs/ with the configuration, tried against Kong 3.9 in a
container, and what Kong's token accounting sees.
## M. Models from Hugging Face we can trust (written 2026-10-25)
The user asked how we know a model fetched from Hugging Face is safe. Two
questions: whether its files can run code on our machines, which we can
close, and whether the model itself was trained to misbehave, which no scan
proves — only who published it, an eval before it serves, and the
guardrails narrow that. What existed: `--trust-remote-code` refused, no
repository Python fetched, ComfyUI nodes pinned to a commit, engines
hash-pinned and run as `llmgw-engine`, mirror versions pinned to a commit
and never overwritten. What did not: everything below.
### M1. Weights in a pickle format refused (S) — done 2026-10-25
`.bin`, `.pt`, `.pth`, `.ckpt`, `.pkl` run code as they are loaded, which is
how the malicious models found on Hugging Face work. The node refuses such a
source, or a mirror set holding one, before fetching anything; a model being
added or given such a source is refused; vLLM and SGLang downloading a model
by its id are given `--load-format safetensors`; the mirror page says so on
the file. `allow_pickle: true` on a model lets it through, on purpose. On the
PoC, `triposr` (`hf:stabilityai/TripoSR/model.ckpt`) is the one model that
needs it.
### M2. Every download pinned and checked against Hugging Face (M)
- `hf:org/repo/file` follows `main` and is checked by nothing; a bare
`org/repo` given to vLLM is downloaded by vLLM, of whatever `main` is.
Both: resolve to the commit when the model is added, keep it in the
catalog's origin, and fetch that commit only (`--revision` for vLLM).
- The sha256 Hugging Face publishes for each LFS file compared on every
path, and a file that differs removed: `mirror add`, the manager, the
node's `hf:` and http sources, mirror set files with no sha recorded.
- Done when: a file changed on Hugging Face after a model was added is
refused by the node with both checksums named.
### M3. Hugging Face's own scan (S)
Hugging Face scans every file (pickle imports, ClamAV, partners' scanners)
and marks a file unsafe. Ask for it with the repository listing; refuse an
unsafe file, show the result on the mirror page and in the catalog's origin.
The API field has to be confirmed first.
### M4. Publishers we trust, and the rest asked about (S)
An allow-list of organisations in the cluster's settings (Qwen, meta-llama,
mistralai, google, microsoft, …). A model from anyone else is added only by
an admin who says so, and who said it is kept in the origin. The PoC already
serves four community models whose training nobody here checked.
### M5. Engines without the internet (M)
The firewall is inbound only, so an engine that was made to run code can
send what it reads anywhere. With weights from the mirror the engines need
no egress: an nft output rule for the engine user (allow the mirror, the
cluster, loopback), systemd hardening of the agent's engines
(`NoNewPrivileges`, `ProtectSystem`, private /tmp). Agent itself still root.
### M6. An eval before a new model serves (S)
`eval` and `canary` exist; make a new model from an untrusted publisher
(M4) serve only after its eval suite passes, so a model trained to answer
wrongly on a trigger has at least been asked the suite.
## D. Facts worth knowing before starting
- Admin API: `127.0.0.1:9090`, behind the token in