llmgw/docs/guardrails.md

# Guardrails

Written for the operator deciding what the gateway lets through: what a
policy can check, what it can do, and how one is attached to a team.

`llmgw server init` writes a `baseline` policy that applies to every
request, with Llama Guard 3 1B as its judge on a CPU; see
[install.md](install.md#2-the-control-plane) for what it checks. The
policies are part of the cluster's settings, kept in the store: `llmgw ctl
config edit` (or the console's Settings) changes them, and the change is in
force on every control plane within seconds, without a restart. A cluster
whose settings were brought in from a server.yaml without a `guard:` section
gets the `baseline` policy, without a judge. With no policies, nothing is
checked and nothing costs anything. With one, every request is judged once it is authorised and
within its limits, with its stored conversation spliced in, and before it is
sent to a model; and the answer is judged before the client sees it.

## A policy

```yaml
guard:
  # Applied to every request.
  default: [baseline]
  policies:
    baseline:
      input:
        pii: {secret: block, card: redact}
        injection: flag
      output:
        pii: {secret: block}
    finance:
      mode: enforce          # enforce (default), shadow, off
      fail: closed           # open (default) or closed, for checks that can fail
      no_store: true         # no conversations, notes, prefix routing or external models
      max_output_tokens: 2048
      system_prompt: "You answer questions about Acme's products only."
      drop_client_system: true
      stream: sentence       # sentence (default) or buffer_all
      require_capabilities: true   # no tools/images to models whose catalog entry lacks them
      validate_structured: true    # answers must fit response_format's json_schema
      input:
        pii: {email: redact, phone: redact, iban: block, tckn: redact}
        injection: block
        max_chars: 40000
        deny:
          - {name: codename, regex: "(?i)project falcon", action: block}
      output:
        pii: {iban: redact, tckn: redact, secret: block}
      tools:
        allow: [search, fetch]
        deny: [shell]
        on_denied: strip     # strip (default) or block
        max_calls_per_turn: 4
        validate_arguments: true
```

**What is found.**

- **Personal data:** email, phone, iban, card, tckn, ip. Each is checked
  before it counts: IBANs by their mod-97 check, cards by Luhn and issuer
  prefix, TCKNs by their two check digits, phone numbers only when written
  to be dialled (`+90…`, `00…`, `0532…`).
- **Secrets** (`secret`): OpenAI- and Anthropic-style keys, AWS access key
  ids, GitHub, Slack and Google tokens, JWTs, PEM private keys and llmgw's
  own keys.
- **Prompt injection:** instructions to drop instructions (English and
  Turkish), chat-template control tokens in user text, invisible Unicode,
  long encoded blobs. These are found through disguises: fullwidth letters,
  Cyrillic look-alikes, zero-width characters, and base64, hex and URL
  encoding two levels deep.
- **`deny` patterns:** matched against the text as written, and as a model
  reads it once those disguises are removed.

**What is done.**

- **`flag`** lets the request through and records it.
- **`redact`** rewrites the value as a placeholder (`[EMAIL_1]`; the same
  value is the same placeholder throughout a request).
- **`block`** stops it.

**Shapes.**

- `system_prompt` is put first.
- `drop_client_system` removes the caller's own system and developer
  messages.
- `max_output_tokens` is set where a request asks for none, and lowers one
  that asks for more.
- A tool `deny`/`allow` list strips the tool from the request, or refuses
  it.
- In the answer, a call to a tool that is not allowed, more calls than
  `max_calls_per_turn`, or arguments that do not fit the parameters the
  request declared (`validate_arguments`) stop it.
- `require_capabilities` refuses tools offered to a model whose catalog
  entry does not say `capabilities.tools`, and images to one without
  `capabilities.vision`. It is opt-in because an entry that says nothing is
  read as "cannot".
- `validate_structured` checks an answer against the `json_schema` the
  request asked for in `response_format`. Buffered, or streamed with
  `buffer_all`, an answer that does not fit is stopped; streamed a sentence
  at a time it has gone out already, and is flagged.

## A judge model

Patterns find what has a shape. For what does not — violence, self-harm,
abuse, whatever a moderation model was trained on — a policy can have a
**judge model on the cluster** read the text.

The judge must be in the catalog under the name the policy gives, as a
judge (`role: guard`), with a replica ready. When it is not, what the
policy judges is refused with `guard_unavailable` (`fail: closed`) or served
unjudged (`fail: open`), and `llmgw ctl status` and the console's Overview
say which judge and why. To serve it:

```
llmgw ctl guard judge status           # each judge the policies ask, and what is wrong with it
llmgw ctl guard judge enable           # Llama Guard 3 1B on CPUs, under the name the policies ask
llmgw ctl guard judge enable --replace # when a model of that name is there, but not as a judge
```

or **Do → Serve the guardrails' judge** in the console. A Llama Guard file
served from the mirror page is served as a judge too, under the name the
policies ask. By hand, the catalog entry is:

```yaml
# a catalog model (server.yaml models:, or ctl model add)
- name: llama-guard-3-1b
  source: http://mirror:8081/llama-guard-3-1b/…/Llama-Guard-3-1B-Q4_K_M.gguf
  role: guard
  guard:
    format: llamaguard3        # llamaguard3, shieldgemma or json
    categories: [Violent Crimes, Non-Violent Crimes, Sex-Related Crimes, …]  # names for S1, S2, …
  scaling: {min: 2, max: 2}
  profiles: [{name: cpu, engine: llamacpp}]

# in a guard policy
judge:
  model: llama-guard-3-1b
  input: block                 # flag or block; empty does not ask about input
  output: flag
  categories: [S1, S10]        # only these count; empty: any unsafe verdict
  latency_budget: 250ms        # then the policy's fail: open serves (flagged), closed refuses
```

A model with `role: guard` is the guardrails' alone:

- it is not in `/v1/models` or the chat page, and is refused (404) to
  every key, as a name and as a fallback;
- it is `critical` by default and must always run — on demand is refused.
  Give it two replicas on different nodes, because every judged request
  waits on it.

The gateway's calls to it go past admission, accounting and the guardrails
themselves.

- **Formats:**
  - `llamaguard3` reads Meta's "safe" / "unsafe
S1,S10".
  - `shieldgemma` asks Google's model whether the text breaks the listed
    principles (`categories`, each a sentence) and reads Yes or No.
  - `json` expects `{"safe": false, "categories": [...]}`.
- **What is read:**
  - **Input:** the request's last user turn, after the patterns.
  - **Output:** the answer, together with that user turn. A streamed answer
    is read every 600 characters and at the end, so text between two
    readings has gone out already; `stream: buffer_all` holds it until it
    is read.
- **Cache:** the same text is not asked about twice within five minutes.

## Where a policy applies

`guard.default` applies everywhere. Others are attached to a part of the
tenancy tree, and apply to everything below it:

```
llmgw ctl guard attach project:Acme/finance/payroll finance
llmgw ctl guard attach key:key-7f3a baseline
llmgw ctl guard list
llmgw ctl guard detach project:Acme/finance/payroll finance
```

Policies narrow, as limits do. A policy attached lower can tighten what
applies above it, never loosen it. Combined, they take:

- the strictest mode and failure;
- for each kind, the strictest action;
- every deny pattern and denied tool;
- only the tools every allow list allows;
- the smallest limits;
- `no_store` if any of them asks;
- every pinned prompt, widest first.

An attachment applies at once.

## What a client sees

**A refused request** is answered with 400:

```json
{"error": {"type": "invalid_request_error", "code": "content_policy_violation",
           "message": "this request was refused by the gateway's policy (secret)"}}
```

It names the category, never the text, and costs nothing.

**A stopped answer** ends with `finish_reason: "content_filter"`. A stream
ends with that chunk and `[DONE]`, and the engine is cancelled. The
Responses API reports it as `incomplete` with
`incomplete_details.reason: "content_filter"`.

**Headers:** `x-llmgw-guard` (`flagged`, `redacted`, `blocked`) and
`x-llmgw-guard-policy` say what was done.

**Streams.** A streamed answer is judged a sentence at a time. Text is held
until a sentence, a line or 200 characters ends, released up to its last
whitespace — so a value still arriving is never sent in two halves — and
rewritten if needed. **What was released cannot be taken back**: a policy
that must never let a word through uses `stream: buffer_all`, which holds
the whole answer until it is judged.

## Shadow mode

`mode: shadow` judges and records — the usage row says `shadow-blocked`,
`shadow-redacted` — and changes nothing. Run a new policy in shadow first,
and read what it would have done before enforcing it.

## What is recorded

Each request's usage row has the action, policy, category and the time the
checks took. The access log has the action, stage and policy. Neither has
the text, a matched value, or the placeholder map: the map exists only in
the gateway's memory while the request is served.

## In the console

`sudo llmgw` has a **Guardrails** view: each policy with its mode, what it
does when it cannot judge, and what it checks; whether each judge model it
leans on has replicas ready — a policy that fails closed refuses everything
while its judge has none; where policies are attached, with A to attach one
and Enter on a row to detach it; and what they did in the last day (W for an
hour or a week), totals first and the newest requests after, each with the
request id `llmgw ctl request` reads. The same is at `GET /api/v1/guard` and
`GET /api/v1/guard/events?since=24h`, and on the portal's Guardrails page.