The baseline fails closed with two judges, and a judge not answering is a notification
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
07af03aa93
commit
c553f2041a
11 files changed
+67
-22
No files matched your search
@@ -8,7 +8,7 @@ explains every choice.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 6 GB RAM when it is also a node running the starter model and the judge. |
|
||||
| **A control plane** | One machine, or three for high availability (the store replicated, either one able to take over). Linux, x86-64 or ARM64: Debian 12, Ubuntu 22.04+ or RHEL 10. No GPU needed; 2 GB RAM, 20 GB disk each — 8 GB RAM when it is also a node running the starter model and the two judges. |
|
||||
| **GPU nodes** (optional to start) | The same Linux, with the NVIDIA driver installed. vLLM needs driver 525 or newer and a Turing GPU or newer; a V100 gets an older vLLM. |
|
||||
| **root** | on the control plane. On the other two of a three-machine setup, `sudo` (its password is asked for if it needs one); on nodes, `sudo` without a password. |
|
||||
| **SSH** | from the control plane to the other machines. The setup screens take a key, the ssh agent or a password; `node add` uses ssh keys. A machine nothing may log into joins with a code instead (`llmgw join`). |
|
||||
@@ -52,8 +52,8 @@ install:
|
||||
- your administrator account. **The password is shown once. Copy it now.**
|
||||
- a model mirror, which downloads model files once for the whole cluster;
|
||||
- this machine as a first node, and a small model (Qwen2.5 0.5B) on it;
|
||||
- guardrails on every request, with a judge model (Llama Guard 3 1B) on a
|
||||
CPU, and API keys that expire within 90 days. What is hardened, and how
|
||||
- guardrails on every request, with a judge model (Llama Guard 3 1B, two
|
||||
replicas) on CPUs — a request it cannot check is refused, not let through — and API keys that expire within 90 days. What is hardened, and how
|
||||
to loosen it: [install.md](install.md#2-the-control-plane).
|
||||
|
||||
Run again later, `sudo llmgw` is the console: the cluster's state and
|
||||
|
||||
Reference in new issue
Block a user