Everything in server.yaml that is not one machine's own is now a versioned document in the store (cluster_settings), the same on every control plane. The first start against a store imports the file's cluster keys, with the baseline guard policy when there is none; after that those keys in a file are ignored, with a warning. Each control plane polls the document's version and its own server.yaml's hash. A change is validated and applied: guard live (Gate.Replace), anything else by a graceful restart that systemd brings back, standbys first and the leader 30s later. A version that does not load on a machine is not applied; at start the newest one that loads is used, so a bad edit cannot keep every control plane down. llmgw ctl config show/get/set/unset/edit/history/rollback, the admin API under /api/v1/settings, and the console's Settings screen, which layers the store's document over the machine's file, write it. Commands beside the control plane that read cluster keys (init, node add, server add, backup, upgrade) read the layered configuration; init --bundle writes offline to the store once it holds the settings. A standby that becomes the leader re-reads models, engine choices and facility holds from the store. Retires appendGuardIfMissing: the import does it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
227 lines
9.2 KiB
Markdown
227 lines
9.2 KiB
Markdown
# Guardrails
|
|
|
|
Written for the operator deciding what the gateway lets through: what a
|
|
policy can check, what it can do, and how one is attached to a team.
|
|
|
|
`llmgw server init` writes a `baseline` policy that applies to every
|
|
request, with Llama Guard 3 1B as its judge on a CPU; see
|
|
[install.md](install.md#2-the-control-plane) for what it checks. The
|
|
policies are part of the cluster's settings, kept in the store: `llmgw ctl
|
|
config edit` (or the console's Settings) changes them, and the change is in
|
|
force on every control plane within seconds, without a restart. A cluster
|
|
whose settings were brought in from a server.yaml without a `guard:` section
|
|
gets the `baseline` policy, without a judge. With no policies, nothing is
|
|
checked and nothing costs anything. With one, every request is judged once it is authorised and
|
|
within its limits, with its stored conversation spliced in, and before it is
|
|
sent to a model; and the answer is judged before the client sees it.
|
|
|
|
## A policy
|
|
|
|
```yaml
|
|
guard:
|
|
# Applied to every request.
|
|
default: [baseline]
|
|
policies:
|
|
baseline:
|
|
input:
|
|
pii: {secret: block, card: redact}
|
|
injection: flag
|
|
output:
|
|
pii: {secret: block}
|
|
finance:
|
|
mode: enforce # enforce (default), shadow, off
|
|
fail: closed # open (default) or closed, for checks that can fail
|
|
no_store: true # no conversations, notes, prefix routing or external models
|
|
max_output_tokens: 2048
|
|
system_prompt: "You answer questions about Acme's products only."
|
|
drop_client_system: true
|
|
stream: sentence # sentence (default) or buffer_all
|
|
require_capabilities: true # no tools/images to models whose catalog entry lacks them
|
|
validate_structured: true # answers must fit response_format's json_schema
|
|
input:
|
|
pii: {email: redact, phone: redact, iban: block, tckn: redact}
|
|
injection: block
|
|
max_chars: 40000
|
|
deny:
|
|
- {name: codename, regex: "(?i)project falcon", action: block}
|
|
output:
|
|
pii: {iban: redact, tckn: redact, secret: block}
|
|
tools:
|
|
allow: [search, fetch]
|
|
deny: [shell]
|
|
on_denied: strip # strip (default) or block
|
|
max_calls_per_turn: 4
|
|
validate_arguments: true
|
|
```
|
|
|
|
**What is found.**
|
|
|
|
- **Personal data:** email, phone, iban, card, tckn, ip. Each is checked
|
|
before it counts: IBANs by their mod-97 check, cards by Luhn and issuer
|
|
prefix, TCKNs by their two check digits, phone numbers only when written
|
|
to be dialled (`+90…`, `00…`, `0532…`).
|
|
- **Secrets** (`secret`): OpenAI- and Anthropic-style keys, AWS access key
|
|
ids, GitHub, Slack and Google tokens, JWTs, PEM private keys and llmgw's
|
|
own keys.
|
|
- **Prompt injection:** instructions to drop instructions (English and
|
|
Turkish), chat-template control tokens in user text, invisible Unicode,
|
|
long encoded blobs. These are found through disguises: fullwidth letters,
|
|
Cyrillic look-alikes, zero-width characters, and base64, hex and URL
|
|
encoding two levels deep.
|
|
- **`deny` patterns:** matched against the text as written, and as a model
|
|
reads it once those disguises are removed.
|
|
|
|
**What is done.**
|
|
|
|
- **`flag`** lets the request through and records it.
|
|
- **`redact`** rewrites the value as a placeholder (`[EMAIL_1]`; the same
|
|
value is the same placeholder throughout a request).
|
|
- **`block`** stops it.
|
|
|
|
**Shapes.**
|
|
|
|
- `system_prompt` is put first.
|
|
- `drop_client_system` removes the caller's own system and developer
|
|
messages.
|
|
- `max_output_tokens` is set where a request asks for none, and lowers one
|
|
that asks for more.
|
|
- A tool `deny`/`allow` list strips the tool from the request, or refuses
|
|
it.
|
|
- In the answer, a call to a tool that is not allowed, more calls than
|
|
`max_calls_per_turn`, or arguments that do not fit the parameters the
|
|
request declared (`validate_arguments`) stop it.
|
|
- `require_capabilities` refuses tools offered to a model whose catalog
|
|
entry does not say `capabilities.tools`, and images to one without
|
|
`capabilities.vision`. It is opt-in because an entry that says nothing is
|
|
read as "cannot".
|
|
- `validate_structured` checks an answer against the `json_schema` the
|
|
request asked for in `response_format`. Buffered, or streamed with
|
|
`buffer_all`, an answer that does not fit is stopped; streamed a sentence
|
|
at a time it has gone out already, and is flagged.
|
|
|
|
## A judge model
|
|
|
|
Patterns find what has a shape. For what does not — violence, self-harm,
|
|
abuse, whatever a moderation model was trained on — a policy can have a
|
|
**judge model on the cluster** read the text:
|
|
|
|
```yaml
|
|
# a catalog model (server.yaml models:, or ctl model add)
|
|
- name: llama-guard-3-1b
|
|
source: http://mirror:8081/llama-guard-3-1b/…/Llama-Guard-3-1B-Q4_K_M.gguf
|
|
role: guard
|
|
guard:
|
|
format: llamaguard3 # llamaguard3, shieldgemma or json
|
|
categories: [Violent Crimes, Non-Violent Crimes, Sex-Related Crimes, …] # names for S1, S2, …
|
|
scaling: {min: 2, max: 2}
|
|
profiles: [{name: cpu, engine: llamacpp}]
|
|
|
|
# in a guard policy
|
|
judge:
|
|
model: llama-guard-3-1b
|
|
input: block # flag or block; empty does not ask about input
|
|
output: flag
|
|
categories: [S1, S10] # only these count; empty: any unsafe verdict
|
|
latency_budget: 250ms # then the policy's fail: open serves (flagged), closed refuses
|
|
```
|
|
|
|
A model with `role: guard` is the guardrails' alone:
|
|
|
|
- it is not in `/v1/models` or the chat page, and is refused (404) to
|
|
every key, as a name and as a fallback;
|
|
- it is `critical` by default and must always run — on demand is refused.
|
|
Give it two replicas on different nodes, because every judged request
|
|
waits on it.
|
|
|
|
The gateway's calls to it go past admission, accounting and the guardrails
|
|
themselves.
|
|
|
|
- **Formats:**
|
|
- `llamaguard3` reads Meta's "safe" / "unsafe
|
|
S1,S10".
|
|
- `shieldgemma` asks Google's model whether the text breaks the listed
|
|
principles (`categories`, each a sentence) and reads Yes or No.
|
|
- `json` expects `{"safe": false, "categories": [...]}`.
|
|
- **What is read:**
|
|
- **Input:** the request's last user turn, after the patterns.
|
|
- **Output:** the answer, together with that user turn. A streamed answer
|
|
is read every 600 characters and at the end, so text between two
|
|
readings has gone out already; `stream: buffer_all` holds it until it
|
|
is read.
|
|
- **Cache:** the same text is not asked about twice within five minutes.
|
|
|
|
## Where a policy applies
|
|
|
|
`guard.default` applies everywhere. Others are attached to a part of the
|
|
tenancy tree, and apply to everything below it:
|
|
|
|
```
|
|
llmgw ctl guard attach project:Acme/finance/payroll finance
|
|
llmgw ctl guard attach key:key-7f3a baseline
|
|
llmgw ctl guard list
|
|
llmgw ctl guard detach project:Acme/finance/payroll finance
|
|
```
|
|
|
|
Policies narrow, as limits do. A policy attached lower can tighten what
|
|
applies above it, never loosen it. Combined, they take:
|
|
|
|
- the strictest mode and failure;
|
|
- for each kind, the strictest action;
|
|
- every deny pattern and denied tool;
|
|
- only the tools every allow list allows;
|
|
- the smallest limits;
|
|
- `no_store` if any of them asks;
|
|
- every pinned prompt, widest first.
|
|
|
|
An attachment applies at once.
|
|
|
|
## What a client sees
|
|
|
|
**A refused request** is answered with 400:
|
|
|
|
```json
|
|
{"error": {"type": "invalid_request_error", "code": "content_policy_violation",
|
|
"message": "this request was refused by the gateway's policy (secret)"}}
|
|
```
|
|
|
|
It names the category, never the text, and costs nothing.
|
|
|
|
**A stopped answer** ends with `finish_reason: "content_filter"`. A stream
|
|
ends with that chunk and `[DONE]`, and the engine is cancelled. The
|
|
Responses API reports it as `incomplete` with
|
|
`incomplete_details.reason: "content_filter"`.
|
|
|
|
**Headers:** `x-llmgw-guard` (`flagged`, `redacted`, `blocked`) and
|
|
`x-llmgw-guard-policy` say what was done.
|
|
|
|
**Streams.** A streamed answer is judged a sentence at a time. Text is held
|
|
until a sentence, a line or 200 characters ends, released up to its last
|
|
whitespace — so a value still arriving is never sent in two halves — and
|
|
rewritten if needed. **What was released cannot be taken back**: a policy
|
|
that must never let a word through uses `stream: buffer_all`, which holds
|
|
the whole answer until it is judged.
|
|
|
|
## Shadow mode
|
|
|
|
`mode: shadow` judges and records — the usage row says `shadow-blocked`,
|
|
`shadow-redacted` — and changes nothing. Run a new policy in shadow first,
|
|
and read what it would have done before enforcing it.
|
|
|
|
## What is recorded
|
|
|
|
Each request's usage row has the action, policy, category and the time the
|
|
checks took. The access log has the action, stage and policy. Neither has
|
|
the text, a matched value, or the placeholder map: the map exists only in
|
|
the gateway's memory while the request is served.
|
|
|
|
## In the console
|
|
|
|
`sudo llmgw` has a **Guardrails** view: each policy with its mode, what it
|
|
does when it cannot judge, and what it checks; whether each judge model it
|
|
leans on has replicas ready — a policy that fails closed refuses everything
|
|
while its judge has none; where policies are attached, with A to attach one
|
|
and Enter on a row to detach it; and what they did in the last day (W for an
|
|
hour or a week), totals first and the newest requests after, each with the
|
|
request id `llmgw ctl request` reads. The same is at `GET /api/v1/guard` and
|
|
`GET /api/v1/guard/events?since=24h`, and on the portal's Guardrails page.
|