Make the planned entry use an engine that can serve the precision
CI / test (push) Successful in 1m39s
CI / test (push) Successful in 1m39s
Reading the tool's own output against the real fleet: it chose q8_0 as the best fit and emitted a vLLM profile with no quantisation. q8_0 is a GGUF quantisation that only llama.cpp serves, and from a file rather than a repository — so following that entry loads full-precision weights, which the same output had just listed as not fitting one card. The planner would have caused the out-of-memory it exists to prevent, from a recommendation that looked right. Choosing a precision and choosing an engine turn out to be one decision, so a Precision now carries the engine that serves it, the quantisation flag where one is needed, and whether it lives in a file. A GGUF fit prints a path and says to download it; a vLLM fit prints the repository and passes --quantization so the engine does not quietly load bf16. The arithmetic was right and the recommendation was wrong, which is the failure mode worth having tests for: everything reported a number, and the numbers agreed with each other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
88a5949f7c
commit
bd134a7ab9
3 files changed
+75
-10
No files matched your search
@@ -40,25 +40,42 @@ type Precision struct {
|
||||
Name string
|
||||
BytesPerParam float64
|
||||
BytesPerKVElem float64
|
||||
// Engine is the one that can serve this precision. Carried here because
|
||||
// choosing a precision and choosing an engine are the same decision, and
|
||||
// separating them produced a plan recommending a GGUF quantisation with a
|
||||
// vLLM profile — which would have loaded full-precision weights that the
|
||||
// same plan had just shown do not fit.
|
||||
Engine string
|
||||
// Quantization is the engine's own flag value, where it needs one.
|
||||
Quantization string
|
||||
// GGUF marks a precision that lives in a file rather than a repository, so
|
||||
// the source is a path and not a model id.
|
||||
GGUF bool
|
||||
}
|
||||
|
||||
// Known precisions, from full to the quantisations worth serving.
|
||||
//
|
||||
// Engine matters as much as the number of bytes. q8_0 and q4_k_m are GGUF
|
||||
// quantisations and only llama.cpp serves them, from a .gguf file rather than a
|
||||
// repository; bf16 and fp8 are what vLLM and SGLang load from the weights as
|
||||
// published. A plan that picks a precision without saying which engine can
|
||||
// serve it is a plan that does not work — and worse, one that looks like it
|
||||
// should.
|
||||
var (
|
||||
BF16 = Precision{"bf16", 2, 2}
|
||||
FP8 = Precision{"fp8", 1, 1}
|
||||
INT4 = Precision{"int4", 0.5, 2} // weights quantised, KV usually is not
|
||||
Q8GGUF = Precision{"q8_0", 1.06, 2}
|
||||
Q4GGUF = Precision{"q4_k_m", 0.61, 2}
|
||||
BF16 = Precision{Name: "bf16", BytesPerParam: 2, BytesPerKVElem: 2, Engine: "vllm"}
|
||||
FP8 = Precision{Name: "fp8", BytesPerParam: 1, BytesPerKVElem: 1, Engine: "vllm", Quantization: "fp8"}
|
||||
Q8GGUF = Precision{Name: "q8_0", BytesPerParam: 1.06, BytesPerKVElem: 2, Engine: "llamacpp", GGUF: true}
|
||||
Q4GGUF = Precision{Name: "q4_k_m", BytesPerParam: 0.61, BytesPerKVElem: 2, Engine: "llamacpp", GGUF: true}
|
||||
)
|
||||
|
||||
// PrecisionByName resolves a precision, reporting what it knows when it cannot.
|
||||
func PrecisionByName(s string) (Precision, error) {
|
||||
for _, p := range []Precision{BF16, FP8, INT4, Q8GGUF, Q4GGUF} {
|
||||
for _, p := range []Precision{BF16, FP8, Q8GGUF, Q4GGUF} {
|
||||
if p.Name == s {
|
||||
return p, nil
|
||||
}
|
||||
}
|
||||
return Precision{}, fmt.Errorf("unknown precision %q; known: bf16, fp8, int4, q8_0, q4_k_m", s)
|
||||
return Precision{}, fmt.Errorf("unknown precision %q; known: bf16, fp8, q8_0, q4_k_m", s)
|
||||
}
|
||||
|
||||
// Demand is what one replica needs, in bytes.
|
||||
|
||||
Reference in new issue
Block a user