Commit Graph
11 Commits
Author SHA1 Message Date
Ilker ManapandClaude Opus 5.5 c09d5ad4ee Outside services read answers redacted and whole; a card's own lanes
CI / test (push) Successful in 9m16s
From the second review.
- An output service reads the answer with the values restore put back
  taken out again (guard.Subject.Restored), as an input service reads the
  request redacted.
- With n above 1, the service reads the first choice once it is whole,
  not at whichever choice finished first.
- Restored values go into an answer asked for as JSON (response_format)
  escaped for it, as into tool-call arguments.
- The embedding failure is logged once a minute through an outage, not
  once.
- A card's own lane count and fastest generation come from the kernel
  (max_link_width, max_link_speed): the driver's width.max is what card
  and slot allow together, so an x16 card in an x8 slot read x8 of x8 and
  was never said.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 13:22:39 +01:00
Ilker ManapandClaude Opus 5.5 54c6b66cb9 Fixes from review: the similar cache, restore, outside services, latency
CI / test (push) Successful in 9m48s
- The similar-question cache embeds a question as its project would: a
  project kept in the cluster has external fallbacks refused, and an
  embedding model outside the cluster is not asked at all. The vectors are
  compared outside the cache's lock. The question's bound is in characters.
- The restorer puts values back in streamed tool calls' arguments (escaped
  for the JSON they are), completions' text and reasoning, each held across
  events on its own, and releases a held tail with the finish event. A
  whole answer is walked as JSON, so a value with a quote no longer breaks
  tool-call arguments.
- An outside guardrail service reads the user's turn as redacted, and a
  streamed answer once rather than at the finish chunk and again at the
  end. guardrails.md says it can stop a stream only under buffer_all.
- A replica's smoothed time to first token is forgotten after five minutes
  without an answer: one found slow and then given nothing was never tried
  again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 08:15:43 +01:00
Ilker ManapandClaude Opus 5.5 783dc9e9f6 A guardrail policy can ask someone else's service
CI / test (push) Failing after 7m39s
A site may be required to use a particular guardrail: Azure Content Safety,
Lakera, NeMo, a classifier of its own. A policy's services asks one over
HTTP — a body template given the text, header values from files read each
call, dotted paths to where the answer says unsafe and why — on input,
output or both, to block or flag. On input it reads the latest user turn
before the judge; on output the whole answer once at its end. A service
that does not answer is what the policy's fail says. It is told the text
only, never who sent it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 16:49:56 +01:00
Ilker ManapandClaude Opus 5.5 b0273ffb5f Qwen3Guard is the judge, and judges on a GPU and on CPUs are asked in turn
Llama Guard 3 1B could not tell ordinary coding requests from harmful ones:
on the PoC it scored "continue" 0.51, "continue the code above" 0.89 and
"how do I kill all child processes in bash" 0.92 unsafe, against 0.97 to
1.00 for harmful requests, at Q4_K_M and Q8_0 alike, with its own template.
A site had "continue" refused. Qwen3Guard-Gen 0.6B gave the ordinary ones
0.15 at the most and the harmful ones 1.00, in 0.8 GB on CPUs.

- format qwen3guard: its prompt written out as its template writes it, for
  requests and answers, and its Safe / Unsafe / Controversial verdict read;
  controversial counts as the policy's judge.controversial says, safe by
  default.
- judge.models: judges asked in turn, the first that answers deciding; a
  failure is recorded against the judge that failed. guard judge use sets
  the order, replacing categories none of the judges names.
- known judges (Qwen3Guard 0.6B/4B/8B, Llama Guard 1B/8B): guard judge
  enable serves any of them, --gpu the 4B on the roomiest GPU class as a
  standard model; the leader fetches and serves one a policy names that the
  catalog lacks.
- server init installs qwen3guard-0.6b with the baseline's categories.
- a judge reads a long text by its start and end, which fit its context,
  rather than overflowing it and refusing the request under fail: closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-08 10:22:47 +02:00
Ilker ManapandClaude Opus 5.5 c553f2041a The baseline fails closed with two judges, and a judge not answering is a notification
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 19:19:49 +02:00
Ilker ManapandClaude Opus 5.5 742d6633e2 Read a judge's verdict after the answer llama.cpp goes on from, and log a verdict that cannot be read
Found on the PoC: llama.cpp takes a conversation ending on the assistant's
turn as a reply to continue, so Llama Guard answered "Rome.safe" and every
answer judged was served unjudged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:33:23 +02:00
Ilker ManapandClaude Opus 5.5 62038484d6 Never lose a word held back at one release when the next has nothing to rewrite
Found on the PoC: a streamed list came out as "1. 2. 3." — each item cut at
one release and dropped at the next, which sent the engine's own events.
A property test now cuts an answer every way and checks it arrives whole.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:24:22 +02:00
Ilker ManapandClaude Opus 5.5 47e847423d Guardrail metrics and alerts: checks, latency, judge failures, fail-open, redactions, stopped streams, tenant blocks
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:11:50 +02:00
Ilker ManapandClaude Opus 5.5 43f6a585a6 A judge model on the cluster: role guard in the catalog, judge in a guard policy
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 15:09:31 +02:00
Ilker ManapandClaude Opus 5.5 dc788e5cda Opt-in capability checks and structured-answer validation in guard policies
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:39:40 +02:00
Ilker ManapandClaude Opus 5.5 b4bdd250af The gate: judge requests and answers by the policies in force, buffered or a sentence at a time
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-04 12:31:55 +02:00