Document guardrails, correct what data.md said leaves the cluster, and regenerate the reference
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ec737cffb5
commit
2f07b2ba0b
7 files changed
+189
-3
No files matched your search
@@ -37,6 +37,23 @@ APIs change between minors without a deprecation period.
|
||||
`incomplete_details.reason: "content_filter"`. `x-llmgw-guard` and
|
||||
`x-llmgw-guard-policy` say what was done; the usage row records the
|
||||
action, policy, category and time taken — never the text.
|
||||
- **Guardrail policies** (`guard:` in server.yaml; see docs/guardrails.md).
|
||||
- **What is found:** personal data, each kind validated before it counts
|
||||
(email, phone, IBAN by mod-97, card by Luhn, TCKN by its check digits,
|
||||
IP); secrets (API keys, AWS, GitHub, Slack, Google, JWT, PEM); prompt
|
||||
injection seen through disguises (look-alike letters, invisible
|
||||
characters, base64/hex/URL encoding); deny patterns.
|
||||
- **What is done:** flag, redact as `[EMAIL_1]`, or block. A request can
|
||||
be shaped too: a pinned system prompt, the caller's own system messages
|
||||
dropped, an output limit, tools stripped or refused.
|
||||
- **Answers:** judged buffered, or a sentence at a time when streamed,
|
||||
never releasing half a value; tool calls are checked against allow
|
||||
lists, a count and the request's own schemas.
|
||||
- **Modes:** `shadow` records what it would do; `no_store` refuses
|
||||
conversations, notes and models outside the cluster.
|
||||
- **Where they apply:** `guard.default` everywhere; others attached with
|
||||
`llmgw ctl guard attach` to an organization, team, project, user or
|
||||
key, and narrowed like limits.
|
||||
|
||||
### Security
|
||||
|
||||
|
||||
+10
-1
@@ -16,7 +16,16 @@ who made a request, when, how many tokens, and what it cost.
|
||||
Two features change that, and both are off until somebody turns them on for a
|
||||
named project, and neither can be turned on without also saying for how long.
|
||||
|
||||
Nothing on the serving path leaves the cluster.
|
||||
Nothing on the serving path leaves the cluster — **unless the catalog has a
|
||||
model served outside it** (`external:` in a model definition): a request for
|
||||
that model, or one that falls back to it, sends its prompt to that provider.
|
||||
A guardrail policy with `no_store` refuses both for the scopes it covers.
|
||||
|
||||
**Guardrails** read every prompt and answer they are configured for, in the
|
||||
gateway's own memory, to judge them. What they record — in the usage row and
|
||||
the log — is the action, the policy, the category and how long it took: never
|
||||
the text, the value that matched, or the placeholder map of a redaction. See
|
||||
[guardrails.md](guardrails.md).
|
||||
|
||||
## What is stored, exactly
|
||||
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
# Guardrails
|
||||
|
||||
Written for the operator deciding what the gateway lets through: what a
|
||||
policy can check, what it can do, and how one is attached to a team.
|
||||
|
||||
Without a `guard:` section in `server.yaml` nothing is checked and nothing
|
||||
costs anything. With one, every request is judged once it is authorised and
|
||||
within its limits, with its stored conversation spliced in, and before it is
|
||||
sent to a model; and the answer is judged before the client sees it.
|
||||
|
||||
## A policy
|
||||
|
||||
```yaml
|
||||
guard:
|
||||
# Applied to every request.
|
||||
default: [baseline]
|
||||
policies:
|
||||
baseline:
|
||||
input:
|
||||
pii: {secret: block, card: redact}
|
||||
injection: flag
|
||||
output:
|
||||
pii: {secret: block}
|
||||
finance:
|
||||
mode: enforce # enforce (default), shadow, off
|
||||
fail: closed # open (default) or closed, for checks that can fail
|
||||
no_store: true # no conversations, notes, prefix routing or external models
|
||||
max_output_tokens: 2048
|
||||
system_prompt: "You answer questions about Acme's products only."
|
||||
drop_client_system: true
|
||||
stream: sentence # sentence (default) or buffer_all
|
||||
input:
|
||||
pii: {email: redact, phone: redact, iban: block, tckn: redact}
|
||||
injection: block
|
||||
max_chars: 40000
|
||||
deny:
|
||||
- {name: codename, regex: "(?i)project falcon", action: block}
|
||||
output:
|
||||
pii: {iban: redact, tckn: redact, secret: block}
|
||||
tools:
|
||||
allow: [search, fetch]
|
||||
deny: [shell]
|
||||
on_denied: strip # strip (default) or block
|
||||
max_calls_per_turn: 4
|
||||
validate_arguments: true
|
||||
```
|
||||
|
||||
**What is found.**
|
||||
|
||||
- **Personal data:** email, phone, iban, card, tckn, ip. Each is checked
|
||||
before it counts: IBANs by their mod-97 check, cards by Luhn and issuer
|
||||
prefix, TCKNs by their two check digits, phone numbers only when written
|
||||
to be dialled (`+90…`, `00…`, `0532…`).
|
||||
- **Secrets** (`secret`): OpenAI- and Anthropic-style keys, AWS access key
|
||||
ids, GitHub, Slack and Google tokens, JWTs, PEM private keys and llmgw's
|
||||
own keys.
|
||||
- **Prompt injection:** instructions to drop instructions (English and
|
||||
Turkish), chat-template control tokens in user text, invisible Unicode,
|
||||
long encoded blobs. These are found through disguises: fullwidth letters,
|
||||
Cyrillic look-alikes, zero-width characters, and base64, hex and URL
|
||||
encoding two levels deep.
|
||||
- **`deny` patterns:** matched against the text as written, and as a model
|
||||
reads it once those disguises are removed.
|
||||
|
||||
**What is done.**
|
||||
|
||||
- **`flag`** lets the request through and records it.
|
||||
- **`redact`** rewrites the value as a placeholder (`[EMAIL_1]`; the same
|
||||
value is the same placeholder throughout a request).
|
||||
- **`block`** stops it.
|
||||
|
||||
**Shapes.**
|
||||
|
||||
- `system_prompt` is put first.
|
||||
- `drop_client_system` removes the caller's own system and developer
|
||||
messages.
|
||||
- `max_output_tokens` is set where a request asks for none, and lowers one
|
||||
that asks for more.
|
||||
- A tool `deny`/`allow` list strips the tool from the request, or refuses
|
||||
it.
|
||||
- In the answer, a call to a tool that is not allowed, more calls than
|
||||
`max_calls_per_turn`, or arguments that do not fit the parameters the
|
||||
request declared (`validate_arguments`) stop it.
|
||||
|
||||
## Where a policy applies
|
||||
|
||||
`guard.default` applies everywhere. Others are attached to a part of the
|
||||
tenancy tree, and apply to everything below it:
|
||||
|
||||
```
|
||||
llmgw ctl guard attach project:Acme/finance/payroll finance
|
||||
llmgw ctl guard attach key:key-7f3a baseline
|
||||
llmgw ctl guard list
|
||||
llmgw ctl guard detach project:Acme/finance/payroll finance
|
||||
```
|
||||
|
||||
Policies narrow, as limits do. A policy attached lower can tighten what
|
||||
applies above it, never loosen it. Combined, they take:
|
||||
|
||||
- the strictest mode and failure;
|
||||
- for each kind, the strictest action;
|
||||
- every deny pattern and denied tool;
|
||||
- only the tools every allow list allows;
|
||||
- the smallest limits;
|
||||
- `no_store` if any of them asks;
|
||||
- every pinned prompt, widest first.
|
||||
|
||||
An attachment applies at once.
|
||||
|
||||
## What a client sees
|
||||
|
||||
**A refused request** is answered with 400:
|
||||
|
||||
```json
|
||||
{"error": {"type": "invalid_request_error", "code": "content_policy_violation",
|
||||
"message": "this request was refused by the gateway's policy (secret)"}}
|
||||
```
|
||||
|
||||
It names the category, never the text, and costs nothing.
|
||||
|
||||
**A stopped answer** ends with `finish_reason: "content_filter"`. A stream
|
||||
ends with that chunk and `[DONE]`, and the engine is cancelled. The
|
||||
Responses API reports it as `incomplete` with
|
||||
`incomplete_details.reason: "content_filter"`.
|
||||
|
||||
**Headers:** `x-llmgw-guard` (`flagged`, `redacted`, `blocked`) and
|
||||
`x-llmgw-guard-policy` say what was done.
|
||||
|
||||
**Streams.** A streamed answer is judged a sentence at a time. Text is held
|
||||
until a sentence, a line or 200 characters ends, released up to its last
|
||||
whitespace — so a value still arriving is never sent in two halves — and
|
||||
rewritten if needed. **What was released cannot be taken back**: a policy
|
||||
that must never let a word through uses `stream: buffer_all`, which holds
|
||||
the whole answer until it is judged.
|
||||
|
||||
## Shadow mode
|
||||
|
||||
`mode: shadow` judges and records — the usage row says `shadow-blocked`,
|
||||
`shadow-redacted` — and changes nothing. Run a new policy in shadow first,
|
||||
and read what it would have done before enforcing it.
|
||||
|
||||
## What is recorded
|
||||
|
||||
Each request's usage row has the action, policy, category and the time the
|
||||
checks took. The access log has the action, stage and policy. Neither has
|
||||
the text, a matched value, or the placeholder map: the map exists only in
|
||||
the gateway's memory while the request is served.
|
||||
@@ -185,6 +185,7 @@ rather than repeated.
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
| `backup_dir` | `string` | [Server](#server) |
|
||||
| `guard` | `guard.Config` | [Server](#server) |
|
||||
| `firewall_networks` | `[]string` | [Server](#server) |
|
||||
| `backup_recipients` | `[]string` | [Server](#server) |
|
||||
| `releases` | `string` | [Server](#server) |
|
||||
@@ -1494,6 +1495,15 @@ BackupDir is where `llmgw server backup` writes, and where the newest
|
||||
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
||||
Empty means /var/backups/llmgw-server.
|
||||
|
||||
### `guard`
|
||||
|
||||
`guard.Config`
|
||||
|
||||
Guard is the library of guardrail policies: what requests and answers
|
||||
are checked for, and what is done. Applied to every request through
|
||||
guard.default, and to parts of the tenancy tree by attaching them
|
||||
(llmgw ctl guard attach). Empty checks nothing.
|
||||
|
||||
### `firewall_networks`
|
||||
|
||||
`[]string`
|
||||
|
||||
@@ -33,7 +33,6 @@ import (
|
||||
llmgwv1 "gitea.mynodes.xyz/ilker/llmgw/api/gen/llmgw/v1"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/adminapi"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/agentsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/catalog"
|
||||
@@ -41,6 +40,7 @@ import (
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/fleetmetrics"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/modelsvc"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/control/reconciler"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/guard"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/hf"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/mirror"
|
||||
"gitea.mynodes.xyz/ilker/llmgw/internal/obs"
|
||||
|
||||
@@ -225,6 +225,7 @@ var Reference = []Block{
|
||||
{Key: "engines", Type: "map[string]string", Doc: "Engines are the engines every node installs, and at which version. The\nagents install them themselves — the right build for their hardware,\nchecked against its publisher's checksum and seen to run — so a node is\nnever trusted to already have what somebody once put there.\n\nUnset keeps DefaultEngines. A version replaces the default; an empty\none leaves that engine to be installed by hand.", Nested: ""},
|
||||
{Key: "engine_overrides", Type: "[]EngineOverride", Doc: "EngineOverrides give some nodes different engines from the rest: a\nvLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at\nall on nodes meant only for llama.cpp. Each picks nodes and lays its\nengines over the cluster's, in order, later ones last; an empty version\nremoves that engine from the nodes it picks.", Nested: ""},
|
||||
{Key: "backup_dir", Type: "string", Doc: "BackupDir is where `llmgw server backup` writes, and where the newest\narchive's age is read for llmgw_backup_last_success_timestamp_seconds.\nEmpty means /var/backups/llmgw-server.", Nested: ""},
|
||||
{Key: "guard", Type: "guard.Config", Doc: "Guard is the library of guardrail policies: what requests and answers\nare checked for, and what is done. Applied to every request through\nguard.default, and to parts of the tenancy tree by attaching them\n(llmgw ctl guard attach). Empty checks nothing.", Nested: "guard.Config"},
|
||||
{Key: "firewall_networks", Type: "[]string", Doc: "FirewallNetworks are networks, besides those each machine reaches the\ncluster on, that llmgw's internal ports answer: a monitoring host on\nanother subnet, nodes behind a router. Written into every machine's\nfirewall by `server init`, `node add` and `server add`.", Nested: ""},
|
||||
{Key: "backup_recipients", Type: "[]string", Doc: "BackupRecipients encrypt every archive `llmgw server backup` writes, to\nwhoever holds one of their keys: age recipients (age1…) or SSH public\nkeys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds\nthe certificate authority's key and every secret in the store, and is\ncopied off the machine; encrypted, the copy is safe wherever it lands.\nEmpty writes archives unencrypted, readable by root only.", Nested: ""},
|
||||
{Key: "releases", Type: "string", Doc: "Releases is where `llmgw ctl upgrade` fetches release binaries from,\nfor this machine and — through their agents — every node. Empty means\nthe project's own releases.", Nested: ""},
|
||||
|
||||
@@ -47,7 +47,9 @@ func (g *wordGuard) Output(context.Context, guard.Subject) guard.OutputFilter {
|
||||
return &stopHi{}
|
||||
}
|
||||
|
||||
func (g *wordGuard) Directives(guard.Subject) guard.Directives { return guard.Directives{NoStore: g.noStore} }
|
||||
func (g *wordGuard) Directives(guard.Subject) guard.Directives {
|
||||
return guard.Directives{NoStore: g.noStore}
|
||||
}
|
||||
|
||||
type stopHi struct{ v guard.Verdict }
|
||||
|
||||
|
||||
Reference in new issue
Block a user