Make the planned entry use an engine that can serve the precision
CI / test (push) Successful in 1m39s

Reading the tool's own output against the real fleet: it chose q8_0 as the
best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF
quantisation that only llama.cpp serves, and from a file rather than a
repository — so following that entry loads full-precision weights, which
the same output had just listed as not fitting one card. The planner would
have caused the out-of-memory it exists to prevent, from a recommendation
that looked right.

Choosing a precision and choosing an engine turn out to be one decision, so
a Precision now carries the engine that serves it, the quantisation flag
where one is needed, and whether it lives in a file. A GGUF fit prints a
path and says to download it; a vLLM fit prints the repository and passes
--quantization so the engine does not quietly load bf16.

The arithmetic was right and the recommendation was wrong, which is the
failure mode worth having tests for: everything reported a number, and the
numbers agreed with each other.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 07:48:20 +02:00
1 parent 88a5949f7c
commit bd134a7ab9
3 files changed
+75 -10

No files matched your search

+16 -3
View File
@@ -190,7 +190,17 @@ func catalogEntry(m modelplan.Model, f modelplan.Fit, source string, context int
var b strings.Builder
name := strings.ToLower(m.Name)
fmt.Fprintf(&b, "name: %s\n", name)
fmt.Fprintf(&b, "source: %s\n", source)
if f.Precision.GGUF {
// A GGUF quantisation lives in a file, not a repository, and llama.cpp
// is given a path. Printing the repository id here would produce an
// entry that looks right and cannot load — which is the failure this
// whole command exists to prevent.
fmt.Fprintf(&b, "# %s is a GGUF quantisation: download the %s file and give its path.\n",
f.Precision.Name, f.Precision.Name)
fmt.Fprintf(&b, "source: /var/lib/llmgw-models/%s-%s.gguf\n", name, f.Precision.Name)
} else {
fmt.Fprintf(&b, "source: %s\n", source)
}
fmt.Fprintf(&b, "routing_profile: chat\n")
fmt.Fprintf(&b, "context_length: %d\n", context)
fmt.Fprintf(&b, "scaling: { min: 1, max: 1 }\n")
@@ -198,14 +208,17 @@ func catalogEntry(m modelplan.Model, f modelplan.Fit, source string, context int
switch {
case f.CPUOnly:
fmt.Fprintf(&b, " - name: cpu\n engine: llamacpp\n")
fmt.Fprintf(&b, " - name: cpu\n engine: %s\n", f.Precision.Engine)
fmt.Fprintf(&b, " min_memory_gb: %d\n", int(f.PerGPUGB)+2)
default:
fmt.Fprintf(&b, " - name: tp%d\n engine: vllm\n", f.TP)
fmt.Fprintf(&b, " - name: tp%d\n engine: %s\n", f.TP, f.Precision.Engine)
fmt.Fprintf(&b, " gpu_class: %s\n tp: %d\n", f.Class, f.TP)
if f.Nodes > 1 {
fmt.Fprintf(&b, " nodes: %d\n gpus_per_node: %d\n", f.Nodes, f.GPUsPerNode)
}
if q := f.Precision.Quantization; q != "" {
fmt.Fprintf(&b, " args: [\"--quantization\", \"%s\"]\n", q)
}
}
fmt.Fprintf(&b, " priority: 100\n weight: 1\n")
// Measured rather than guessed is the rule everywhere else in this system,