Make the planned entry use an engine that can serve the precision
CI / test (push) Successful in 1m39s

Reading the tool's own output against the real fleet: it chose q8_0 as the
best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF
quantisation that only llama.cpp serves, and from a file rather than a
repository — so following that entry loads full-precision weights, which
the same output had just listed as not fitting one card. The planner would
have caused the out-of-memory it exists to prevent, from a recommendation
that looked right.

Choosing a precision and choosing an engine turn out to be one decision, so
a Precision now carries the engine that serves it, the quantisation flag
where one is needed, and whether it lives in a file. A GGUF fit prints a
path and says to download it; a vLLM fit prints the repository and passes
--quantization so the engine does not quietly load bf16.

The arithmetic was right and the recommendation was wrong, which is the
failure mode worth having tests for: everything reported a number, and the
numbers agreed with each other.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-17 07:48:20 +02:00
1 parent 88a5949f7c
commit bd134a7ab9
3 files changed
+75 -10

No files matched your search

+24 -7
View File
@@ -40,25 +40,42 @@ type Precision struct {
Name string
BytesPerParam float64
BytesPerKVElem float64
// Engine is the one that can serve this precision. Carried here because
// choosing a precision and choosing an engine are the same decision, and
// separating them produced a plan recommending a GGUF quantisation with a
// vLLM profile — which would have loaded full-precision weights that the
// same plan had just shown do not fit.
Engine string
// Quantization is the engine's own flag value, where it needs one.
Quantization string
// GGUF marks a precision that lives in a file rather than a repository, so
// the source is a path and not a model id.
GGUF bool
}
// Known precisions, from full to the quantisations worth serving.
//
// Engine matters as much as the number of bytes. q8_0 and q4_k_m are GGUF
// quantisations and only llama.cpp serves them, from a .gguf file rather than a
// repository; bf16 and fp8 are what vLLM and SGLang load from the weights as
// published. A plan that picks a precision without saying which engine can
// serve it is a plan that does not work — and worse, one that looks like it
// should.
var (
BF16 = Precision{"bf16", 2, 2}
FP8 = Precision{"fp8", 1, 1}
INT4 = Precision{"int4", 0.5, 2} // weights quantised, KV usually is not
Q8GGUF = Precision{"q8_0", 1.06, 2}
Q4GGUF = Precision{"q4_k_m", 0.61, 2}
BF16 = Precision{Name: "bf16", BytesPerParam: 2, BytesPerKVElem: 2, Engine: "vllm"}
FP8 = Precision{Name: "fp8", BytesPerParam: 1, BytesPerKVElem: 1, Engine: "vllm", Quantization: "fp8"}
Q8GGUF = Precision{Name: "q8_0", BytesPerParam: 1.06, BytesPerKVElem: 2, Engine: "llamacpp", GGUF: true}
Q4GGUF = Precision{Name: "q4_k_m", BytesPerParam: 0.61, BytesPerKVElem: 2, Engine: "llamacpp", GGUF: true}
)
// PrecisionByName resolves a precision, reporting what it knows when it cannot.
func PrecisionByName(s string) (Precision, error) {
for _, p := range []Precision{BF16, FP8, INT4, Q8GGUF, Q4GGUF} {
for _, p := range []Precision{BF16, FP8, Q8GGUF, Q4GGUF} {
if p.Name == s {
return p, nil
}
}
return Precision{}, fmt.Errorf("unknown precision %q; known: bf16, fp8, int4, q8_0, q4_k_m", s)
return Precision{}, fmt.Errorf("unknown precision %q; known: bf16, fp8, q8_0, q4_k_m", s)
}
// Demand is what one replica needs, in bytes.