A hardened install by default: baseline guardrails with Llama Guard 3 1B judging on a CPU, keys at most 90 days

server init (and the setup screens) write guard.default [baseline] —
secrets refused, cards masked, injection recorded, the judge refusing
S1-S4, S7, S9-S12 in prompts and answers, fail open after 5s — fetch the
judge into the mirror and run it on a node's CPU. --no-guard-model leaves
it out. The closing text names what init cannot do: TLS on the client
port and backup recipients.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-04 17:45:31 +02:00
1 parent 6abd510628
commit ddb0472f9b
9 files changed
+284 -37

No files matched your search

+5 -2
View File
@@ -3,8 +3,11 @@
Written for the operator deciding what the gateway lets through: what a
policy can check, what it can do, and how one is attached to a team.
Without a `guard:` section in `server.yaml` nothing is checked and nothing
costs anything. With one, every request is judged once it is authorised and
`llmgw server init` writes a `baseline` policy that applies to every
request, with Llama Guard 3 1B as its judge on a CPU; see
[install.md](install.md#2-the-control-plane) for what it checks. Without a
`guard:` section in `server.yaml` nothing is checked and nothing costs
anything. With one, every request is judged once it is authorised and
within its limits, with its stored conversation spliced in, and before it is
sent to a model; and the answer is judged before the client sees it.
+34 -2
View File
@@ -117,7 +117,8 @@ That is the whole step. In order, it:
3. issues the control plane's certificate for this machine's hostname and
every address it has, and copies `ca.pem` beside it
4. writes `/etc/llmgw/server.yaml`: mutual TLS, authentication, default limits
and a price, and a control plane with an empty catalog
and a price, keys that expire after at most 90 days, the baseline
guardrails policy, and a control plane with an empty catalog
5. creates the store, the `default` project, and your account, already approved
and promoted to administrator — the password is generated and **printed
once**
@@ -129,8 +130,39 @@ That is the whole step. In order, it:
engine
9. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers
10. fetches Llama Guard 3 1B, the guardrails' judge, and runs it on the CPU
of a node with room — this one, or the first that joins
11. closes llmgw's internal ports to everything but the cluster's networks
`--no-mirror`, `--no-node` and `--no-model` leave the last three out. An NVIDIA
`--no-mirror`, `--no-node` and `--no-model` leave out 7 to 9,
`--no-guard-model` 10, and `--no-firewall` 11.
**Secure by default.** What init writes is the hardened configuration, not a
demonstration one:
| | |
|---|---|
| Agents | mutual TLS with the cluster's own authority; the agent port refuses plaintext |
| Admin API | loopback only, behind a token; pprof off |
| Store | between machines, TLS everywhere (PostgreSQL, etcd, Patroni) |
| Firewall | llmgw's internal ports answer the cluster's networks only |
| Accounts | signup closed; administrators create accounts or approve them |
| Keys | at most 90 days (`auth.max_key_ttl`), shorter per part of the tree with `llmgw keys ttl` |
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
The judge runs on a CPU, about half a second for a short text, and reads a
streamed answer every 600 characters. When it is slower than 5 seconds or
not running, the request is served and recorded as flagged rather than
refused (`fail: open`), so a judge that is down does not take the gateway
with it; `fail: closed` turns that round. Defamation, specialized advice,
intellectual property and elections are left out of what it refuses: they
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
Two things are left to you because they need something only you have: a
certificate for the client port — port 8080 is plain HTTP; put the proxy in
[deploy/proxy/](../deploy/proxy/) in front — and the keys backups are
encrypted to (`backup_recipients`, see [Backing up](#backing-up)). An NVIDIA
card with no working driver is reported rather than silently ignored; install
the driver and run init again to serve on it.
+5 -2
View File
@@ -8,7 +8,7 @@ explains every choice.
| | |
|---|---|
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each. |
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 6 GB RAM when it is also a node running the starter model and the judge. |
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
@@ -51,7 +51,10 @@ install:
Patroni);
- your administrator account. **The password is shown once. Copy it now.**
- a model mirror, which downloads model files once for the whole cluster;
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it.
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
- guardrails on every request, with a judge model (Llama Guard 3 1B) on a
CPU, and API keys that expire within 90 days. What is hardened, and how
to loosen it: [install.md](install.md#2-the-control-plane).
Run again later, `sudo llmgw` is the console: the cluster's state and
everything you can do to it ([console.md](console.md)). Every step below is