Checked by a single-machine install of v0.14.6 on a 4-core, 8 GB VM. The second judge fitted nowhere: v0.14.6 reserved 4 GB a judge, a figure from one replica that grew to 3.2 GB through llama.cpp's prompt cache (up to 8 GB by default), which a judge never reuses. Once it fitted, every judgment timed out. v0.14.6 gave a judge one core below 8 CPUs, and a judgment of about 280 tokens takes 8.0 s on one core, 4.2 s on two and 2.3 s on four, against a 5 s budget: 0 of 20 requests went through. A judge now runs with --cache-ram 0, uses about 1.6 GB and reserves 2, and reserves two cores (one only below four CPUs). On the same VM, 19 of 20 went through. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
0def6eaba1
commit
f8206c3540
4 files changed
+48
-23
No files matched your search
@@ -496,10 +496,13 @@ func guardModel(source string, replicas, cores int) catalog.Model {
|
||||
// judge that is not there refuses everything.
|
||||
Scaling: catalog.Scaling{Min: replicas, Max: replicas},
|
||||
Profiles: []catalog.Profile{{
|
||||
// It uses about 3.2 GB with its context: reserving 2 filled nodes
|
||||
// past their memory.
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: 4, CPUCores: cores,
|
||||
Args: []string{"--ctx-size", "8192"},
|
||||
// About 1.6 GB with its context, which stays so with no prompt
|
||||
// cache: llama.cpp keeps up to 8 GB of prompts by default, and a
|
||||
// judge's, each a different text, are not asked again. With the
|
||||
// cache one grew to 3.2 GB over a day of traffic; reserving 4 for
|
||||
// that left a 7 GB machine room for only one of the two.
|
||||
Name: "cpu", Engine: "llamacpp", Priority: 100, MinMemoryGB: judgeMemoryGB, CPUCores: cores,
|
||||
Args: []string{"--ctx-size", "8192", "--cache-ram", "0"},
|
||||
}},
|
||||
}
|
||||
}
|
||||
@@ -716,14 +719,16 @@ type cpuSplit struct{ judge, starter int }
|
||||
// the starter on one 4-core control plane all ran on all four: under load
|
||||
// most judgments missed their budget and the requests were refused.
|
||||
//
|
||||
// When this machine is the node, by its own CPUs: a core a judge under 8,
|
||||
// two from 8, and the starter what is left, up to 8, or no reservation at
|
||||
// all when nothing is, rather than a starter that fits nowhere. Otherwise
|
||||
// two a judge, on whichever node takes it, and the starter, on a node not
|
||||
// known yet, none.
|
||||
// A judge needs two cores: a judgment of about 280 tokens took 8.0 s on
|
||||
// one, 4.2 s on two and 2.3 s on four, against a 5 s budget. When this
|
||||
// machine is the node, each judge replica gets two (one each only below four
|
||||
// CPUs, where two would not fit both), and the starter what is left, up to
|
||||
// 8, or no reservation at all when nothing is, rather than a starter that
|
||||
// fits nowhere. Otherwise two a judge, on whichever node takes it, and the
|
||||
// starter, on a node not known yet, none.
|
||||
func (in *serverInit) cpuPlan() cpuSplit {
|
||||
if in.noNode {
|
||||
return cpuSplit{judge: 2}
|
||||
return cpuSplit{judge: judgeCores}
|
||||
}
|
||||
n := in.cpus
|
||||
if n <= 0 {
|
||||
@@ -736,13 +741,18 @@ func (in *serverInit) cpuPlan() cpuSplit {
|
||||
if !in.noGuardModel {
|
||||
judges = in.judgeFor().Replicas
|
||||
}
|
||||
judge := 1
|
||||
if n >= 8 {
|
||||
judge = 2
|
||||
}
|
||||
left := n - judges*judge
|
||||
if left < 1 {
|
||||
left = 0
|
||||
judge := judgeCores
|
||||
if judges > 0 && judges*judge > n {
|
||||
judge = max(1, n/judges)
|
||||
}
|
||||
left := max(n-judges*judge, 0)
|
||||
return cpuSplit{judge: judge, starter: min(left, 8)}
|
||||
}
|
||||
|
||||
// judgeCores and judgeMemoryGB are what a judge replica needs: two cores
|
||||
// for a judgment within its budget, and 2 GB for its 1.6 with no prompt
|
||||
// cache.
|
||||
const (
|
||||
judgeCores = 2
|
||||
judgeMemoryGB = 2
|
||||
)
|
||||
Reference in new issue
Block a user