A second judge replica on the same machine shares its cores and is lost with it, so it bought nothing but half the speed of each. On a 4-core, 8 GB single-machine install, one judge on four cores passed 20 of 20 requests at about 3.5 s each, judged in and out. Two judges on two cores each had passed 19 of 20. server init now serves one judge when this machine is the only node, and two, on two cores each and spread over the nodes, where there will be others (HA setups, or a control plane that is not a node). --judge-replicas and the setup's Copies still choose. install.md gives the measured times in place of "half a second". Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f8206c3540
commit
3e7dd20bad
7 files changed
+91
-29
No files matched your search
@@ -719,13 +719,13 @@ type cpuSplit struct{ judge, starter int }
|
||||
// the starter on one 4-core control plane all ran on all four: under load
|
||||
// most judgments missed their budget and the requests were refused.
|
||||
//
|
||||
// A judge needs two cores: a judgment of about 280 tokens took 8.0 s on
|
||||
// one, 4.2 s on two and 2.3 s on four, against a 5 s budget. When this
|
||||
// machine is the node, each judge replica gets two (one each only below four
|
||||
// CPUs, where two would not fit both), and the starter what is left, up to
|
||||
// 8, or no reservation at all when nothing is, rather than a starter that
|
||||
// fits nowhere. Otherwise two a judge, on whichever node takes it, and the
|
||||
// starter, on a node not known yet, none.
|
||||
// A judge needs cores: a judgment of about 280 tokens took 8.0 s on one,
|
||||
// 4.2 s on two and 2.3 s on four, against a 5 s budget. On one machine its
|
||||
// one replica gets up to four; where there are two, two each (one each only
|
||||
// where two would not fit both). The starter has what is left, up to 8, or
|
||||
// no reservation at all when nothing is, rather than a starter that fits
|
||||
// nowhere. A machine that is not the node reserves two a judge on whichever
|
||||
// node takes it, and the starter, on a node not known yet, none.
|
||||
func (in *serverInit) cpuPlan() cpuSplit {
|
||||
if in.noNode {
|
||||
return cpuSplit{judge: judgeCores}
|
||||
@@ -742,7 +742,10 @@ func (in *serverInit) cpuPlan() cpuSplit {
|
||||
judges = in.judgeFor().Replicas
|
||||
}
|
||||
judge := judgeCores
|
||||
if judges > 0 && judges*judge > n {
|
||||
switch {
|
||||
case judges == 1:
|
||||
judge = min(n, judgeCoresAlone)
|
||||
case judges > 0 && judges*judge > n:
|
||||
judge = max(1, n/judges)
|
||||
}
|
||||
left := max(n-judges*judge, 0)
|
||||
@@ -751,8 +754,10 @@ func (in *serverInit) cpuPlan() cpuSplit {
|
||||
|
||||
// judgeCores and judgeMemoryGB are what a judge replica needs: two cores
|
||||
// for a judgment within its budget, and 2 GB for its 1.6 with no prompt
|
||||
// cache.
|
||||
// cache. judgeCoresAlone is what the one judge of a single machine takes,
|
||||
// past which a model this small hardly gets faster.
|
||||
const (
|
||||
judgeCores = 2
|
||||
judgeMemoryGB = 2
|
||||
judgeCores = 2
|
||||
judgeCoresAlone = 4
|
||||
judgeMemoryGB = 2
|
||||
)
|
||||
Reference in new issue
Block a user