Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
5.4 KiB
Guardrails
Written for the operator deciding what the gateway lets through: what a policy can check, what it can do, and how one is attached to a team.
Without a guard: section in server.yaml nothing is checked and nothing
costs anything. With one, every request is judged once it is authorised and
within its limits, with its stored conversation spliced in, and before it is
sent to a model; and the answer is judged before the client sees it.
A policy
guard:
# Applied to every request.
default: [baseline]
policies:
baseline:
input:
pii: {secret: block, card: redact}
injection: flag
output:
pii: {secret: block}
finance:
mode: enforce # enforce (default), shadow, off
fail: closed # open (default) or closed, for checks that can fail
no_store: true # no conversations, notes, prefix routing or external models
max_output_tokens: 2048
system_prompt: "You answer questions about Acme's products only."
drop_client_system: true
stream: sentence # sentence (default) or buffer_all
input:
pii: {email: redact, phone: redact, iban: block, tckn: redact}
injection: block
max_chars: 40000
deny:
- {name: codename, regex: "(?i)project falcon", action: block}
output:
pii: {iban: redact, tckn: redact, secret: block}
tools:
allow: [search, fetch]
deny: [shell]
on_denied: strip # strip (default) or block
max_calls_per_turn: 4
validate_arguments: true
What is found.
- Personal data: email, phone, iban, card, tckn, ip. Each is checked
before it counts: IBANs by their mod-97 check, cards by Luhn and issuer
prefix, TCKNs by their two check digits, phone numbers only when written
to be dialled (
+90…,00…,0532…). - Secrets (
secret): OpenAI- and Anthropic-style keys, AWS access key ids, GitHub, Slack and Google tokens, JWTs, PEM private keys and llmgw's own keys. - Prompt injection: instructions to drop instructions (English and Turkish), chat-template control tokens in user text, invisible Unicode, long encoded blobs. These are found through disguises: fullwidth letters, Cyrillic look-alikes, zero-width characters, and base64, hex and URL encoding two levels deep.
denypatterns: matched against the text as written, and as a model reads it once those disguises are removed.
What is done.
flaglets the request through and records it.redactrewrites the value as a placeholder ([EMAIL_1]; the same value is the same placeholder throughout a request).blockstops it.
Shapes.
system_promptis put first.drop_client_systemremoves the caller's own system and developer messages.max_output_tokensis set where a request asks for none, and lowers one that asks for more.- A tool
deny/allowlist strips the tool from the request, or refuses it. - In the answer, a call to a tool that is not allowed, more calls than
max_calls_per_turn, or arguments that do not fit the parameters the request declared (validate_arguments) stop it.
Where a policy applies
guard.default applies everywhere. Others are attached to a part of the
tenancy tree, and apply to everything below it:
llmgw ctl guard attach project:Acme/finance/payroll finance
llmgw ctl guard attach key:key-7f3a baseline
llmgw ctl guard list
llmgw ctl guard detach project:Acme/finance/payroll finance
Policies narrow, as limits do. A policy attached lower can tighten what applies above it, never loosen it. Combined, they take:
- the strictest mode and failure;
- for each kind, the strictest action;
- every deny pattern and denied tool;
- only the tools every allow list allows;
- the smallest limits;
no_storeif any of them asks;- every pinned prompt, widest first.
An attachment applies at once.
What a client sees
A refused request is answered with 400:
{"error": {"type": "invalid_request_error", "code": "content_policy_violation",
"message": "this request was refused by the gateway's policy (secret)"}}
It names the category, never the text, and costs nothing.
A stopped answer ends with finish_reason: "content_filter". A stream
ends with that chunk and [DONE], and the engine is cancelled. The
Responses API reports it as incomplete with
incomplete_details.reason: "content_filter".
Headers: x-llmgw-guard (flagged, redacted, blocked) and
x-llmgw-guard-policy say what was done.
Streams. A streamed answer is judged a sentence at a time. Text is held
until a sentence, a line or 200 characters ends, released up to its last
whitespace — so a value still arriving is never sent in two halves — and
rewritten if needed. What was released cannot be taken back: a policy
that must never let a word through uses stream: buffer_all, which holds
the whole answer until it is judged.
Shadow mode
mode: shadow judges and records — the usage row says shadow-blocked,
shadow-redacted — and changes nothing. Run a new policy in shadow first,
and read what it would have done before enforcing it.
What is recorded
Each request's usage row has the action, policy, category and the time the checks took. The access log has the action, stage and policy. Neither has the text, a matched value, or the placeholder map: the map exists only in the gateway's memory while the request is served.