One machine runs one judge, on up to four cores
CI / test (push) Successful in 9m1s

A second judge replica on the same machine shares its cores and is lost
with it, so it bought nothing but half the speed of each. On a 4-core,
8 GB single-machine install, one judge on four cores passed 20 of 20
requests at about 3.5 s each, judged in and out. Two judges on two cores
each had passed 19 of 20.

server init now serves one judge when this machine is the only node, and
two, on two cores each and spread over the nodes, where there will be
others (HA setups, or a control plane that is not a node).
--judge-replicas and the setup's Copies still choose. install.md gives
the measured times in place of "half a second".

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-16 19:35:42 +02:00
1 parent f8206c3540
commit 3e7dd20bad
7 files changed
+91 -29

No files matched your search

+15 -11
View File
@@ -152,9 +152,9 @@ That is the whole step. In order, it:
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
engine
9. closes llmgw's internal ports to everything but the cluster's networks
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
replicas on the CPUs of nodes with room — this one, or the first that
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
CPUs: one replica on up to four cores when this machine is the only
node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
first thing anyone opens `/chat` to find is a model that answers. A
machine without room for it is told so within two minutes, and why.
@@ -187,15 +187,19 @@ demonstration one:
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
The judge runs as two replicas, each on a CPU — about half a second for a
short text on any CPU with AVX2 — and reads a streamed answer every 600
characters. The policy fails **closed**: a request the judge cannot read
within 5 seconds is refused with 503 rather than served unjudged, and
`judge_unavailable` goes to the notification webhooks. Two replicas are what
make that bearable; on one small machine, or a VM without AVX2 (see
The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
streamed answer every 600 characters. The policy fails **closed**: a request
the judge cannot read within 5 seconds is refused with 503 rather than
served unjudged, and `judge_unavailable` goes to the notification webhooks.
So a judge gets cores of its own: on one machine one replica on up to four,
and on a larger cluster two replicas of two cores each, spread over its nodes
so that losing one leaves the other. On a machine of fewer than four cores,
a busy one, or a VM without AVX2 (see
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
`--no-guard-model` is the choice there. Defamation, specialized advice,
first thing to fail. The choices there are a longer `latency_budget`,
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
or `--no-guard-model`. Defamation, specialized advice,
intellectual property and elections are left out of what it refuses: they
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.