A second judge replica on the same machine shares its cores and is lost with it, so it bought nothing but half the speed of each. On a 4-core, 8 GB single-machine install, one judge on four cores passed 20 of 20 requests at about 3.5 s each, judged in and out. Two judges on two cores each had passed 19 of 20. server init now serves one judge when this machine is the only node, and two, on two cores each and spread over the nodes, where there will be others (HA setups, or a control plane that is not a node). --judge-replicas and the setup's Copies still choose. install.md gives the measured times in place of "half a second". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f8206c3540
commit
3e7dd20bad
7 files changed
+91
-29
No files matched your search
+15
-11
@@ -152,9 +152,9 @@ That is the whole step. In order, it:
|
||||
`nvidia-smi` finds any, on CPU otherwise — and waits for it to install its
|
||||
engine
|
||||
9. closes llmgw's internal ports to everything but the cluster's networks
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs two
|
||||
replicas on the CPUs of nodes with room — this one, or the first that
|
||||
join ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
10. fetches Qwen3Guard-Gen 0.6B, the guardrails' judge, and runs it on
|
||||
CPUs: one replica on up to four cores when this machine is the only
|
||||
node, two on two cores each on the nodes of a larger cluster ([guardrails.md](guardrails.md#a-judge-model) has one on a GPU too)
|
||||
11. fetches Qwen2.5 0.5B into the mirror and serves it on this node, so the
|
||||
first thing anyone opens `/chat` to find is a model that answers. A
|
||||
machine without room for it is told so within two minutes, and why.
|
||||
@@ -187,15 +187,19 @@ demonstration one:
|
||||
| Requests | every key limited (60 a minute, 200 000 tokens a minute) and priced, so budgets bind |
|
||||
| Guardrails | the `baseline` policy on every request: secrets in prompts and answers refused, card numbers masked, injection attempts recorded, and the judge refusing violence, crime, weapons, child abuse, privacy violations, hate, self-harm and sexual content, in prompts and answers |
|
||||
|
||||
The judge runs as two replicas, each on a CPU — about half a second for a
|
||||
short text on any CPU with AVX2 — and reads a streamed answer every 600
|
||||
characters. The policy fails **closed**: a request the judge cannot read
|
||||
within 5 seconds is refused with 503 rather than served unjudged, and
|
||||
`judge_unavailable` goes to the notification webhooks. Two replicas are what
|
||||
make that bearable; on one small machine, or a VM without AVX2 (see
|
||||
The judge runs on CPUs. A judgment of about 280 tokens took 2.3 s on four
|
||||
cores, 4.2 s on two and 8.0 s on one (a 4-core VM). It also reads a
|
||||
streamed answer every 600 characters. The policy fails **closed**: a request
|
||||
the judge cannot read within 5 seconds is refused with 503 rather than
|
||||
served unjudged, and `judge_unavailable` goes to the notification webhooks.
|
||||
So a judge gets cores of its own: on one machine one replica on up to four,
|
||||
and on a larger cluster two replicas of two cores each, spread over its nodes
|
||||
so that losing one leaves the other. On a machine of fewer than four cores,
|
||||
a busy one, or a VM without AVX2 (see
|
||||
[step 6](#6-setting-the-nodes-cpu-model-if-it-is-a-vm)), the judge is the
|
||||
first thing to fail, so `fail: open` — serve it, recorded as flagged — or
|
||||
`--no-guard-model` is the choice there. Defamation, specialized advice,
|
||||
first thing to fail. The choices there are a longer `latency_budget`,
|
||||
`fail: open` (serve the request, recorded as flagged), the judge on a GPU,
|
||||
or `--no-guard-model`. Defamation, specialized advice,
|
||||
intellectual property and elections are left out of what it refuses: they
|
||||
catch too much ordinary work. [guardrails.md](guardrails.md) has the rest.
|
||||
|
||||
|
||||
Reference in new issue
Block a user