A hardened install by default: baseline guardrails with Llama Guard 3 1B judging on a CPU, keys at most 90 days
server init (and the setup screens) write guard.default [baseline] — secrets refused, cards masked, injection recorded, the judge refusing S1-S4, S7, S9-S12 in prompts and answers, fail open after 5s — fetch the judge into the mirror and run it on a node's CPU. --no-guard-model leaves it out. The closing text names what init cannot do: TLS on the client port and backup recipients. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
6abd510628
commit
ddb0472f9b
9 files changed
+284
-37
No files matched your search
+5
-2
@@ -3,8 +3,11 @@
|
||||
Written for the operator deciding what the gateway lets through: what a
|
||||
policy can check, what it can do, and how one is attached to a team.
|
||||
|
||||
Without a `guard:` section in `server.yaml` nothing is checked and nothing
|
||||
costs anything. With one, every request is judged once it is authorised and
|
||||
`llmgw server init` writes a `baseline` policy that applies to every
|
||||
request, with Llama Guard 3 1B as its judge on a CPU; see
|
||||
[install.md](install.md#2-the-control-plane) for what it checks. Without a
|
||||
`guard:` section in `server.yaml` nothing is checked and nothing costs
|
||||
anything. With one, every request is judged once it is authorised and
|
||||
within its limits, with its stored conversation spliced in, and before it is
|
||||
sent to a model; and the answer is judged before the client sees it.
|
||||
|
||||
|
||||
+34
-2
@@ -117,7 +117,8 @@ That is the whole step. In order, it:
|
||||
3. issues the control plane's certificate for this machine's hostname and
|
||||
every address it has, and copies `ca.pem` beside it
|
||||
4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
|
||||
and a price, and a control plane with an empty catalog
|
||||
and a price, keys that expire after at most 90 days, the baseline
|
||||
guardrails policy, and a control plane with an empty catalog
|
||||
5. creates the store, the `default` project, and your account, already approved
|
||||
and promoted to administrator — the password is generated and **printed
|
||||
once**
|
||||
@@ -129,8 +130,39 @@ That is the whole step. In order, it:
|
||||
engine
|
||||
9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers
|
||||
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU
|
||||
of a node with room — this one, or the first that joins
|
||||
11. closes llmgw's internal ports to everything but the cluster's networks
|
||||
|
||||
`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA
|
||||
`--no-mirror`, `--no-node` and `--no-model` leave out 7 to 9,
|
||||
`--no-guard-model` 10, and `--no-firewall` 11.
|
||||
|
||||
**Secure by default.** What init writes is the hardened configuration, not a
|
||||
demonstration one:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Agents | mutual TLS with the cluster's own authority; the agent port refuses plaintext |
|
||||
| Admin API | loopback only, behind a token; pprof off |
|
||||
| Store | between machines, TLS everywhere (PostgreSQL, etcd, Patroni) |
|
||||
| Firewall | llmgw's internal ports answer the cluster's networks only |
|
||||
| Accounts | signup closed; administrators create accounts or approve them |
|
||||
| Keys | at most 90 days (`auth.max_key_ttl`), shorter per part of the tree with `llmgw keys ttl` |
|
||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||
|
||||
The judge runs on a CPU, about half a second for a short text, and reads a
|
||||
streamed answer every 600 characters. When it is slower than 5 seconds or
|
||||
not running, the request is served and recorded as flagged rather than
|
||||
refused (`fail: open`), so a judge that is down does not take the gateway
|
||||
with it; `fail: closed` turns that round. Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
Two things are left to you because they need something only you have: a
|
||||
certificate for the client port — port 8080 is plain HTTP; put the proxy in
|
||||
[deploy/proxy/](../deploy/proxy/) in front — and the keys backups are
|
||||
encrypted to (`backup_recipients`, see [Backing up](#backing-up)). An NVIDIA
|
||||
card with no working driver is reported rather than silently ignored; install
|
||||
the driver and run init again to serve on it.
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ explains every choice.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each. |
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 6 GB RAM when it is also a node running the starter model and the judge. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||
@@ -51,7 +51,10 @@ install:
|
||||
Patroni);
|
||||
- your administrator account. **The password is shown once. Copy it now.**
|
||||
- a model mirror, which downloads model files once for the whole cluster;
|
||||
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
|
||||
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
|
||||
- guardrails on every request, with a judge model (Llama Guard 3 1B) on a
|
||||
CPU, and API keys that expire within 90 days. What is hardened, and how
|
||||
to loosen it: [install.md](install.md#2-the-control-plane).
|
||||
|
||||
Run again later, `sudo llmgw` is the console: the cluster's state and
|
||||
everything you can do to it ([console.md](console.md)). Every step below is
|
||||
|
||||
Reference in new issue
Block a user