Ilker ManapandClaude Opus 5.5
4a652fdcc1
The serve form asked for weights plus half again plus two gigabytes. That ignores the one thing that varies most: the KV cache. At the same 16k pool, the models on the live mirror need from 0.44 GB of cache (Qwen2.5-1.5B, two KV heads) to 5.25 GB (Gemma-2-9B, 256-wide heads) — so the flat guess overcounted the small models, which on 7 GB nodes means refusing placements that fit, and undercounted the ones with wide or many KV heads, which means an engine killed for memory. The GGUF header already carries the shape. The reader now also takes each architecture's block_count, head counts, embedding length and key/value lengths, reading past general.* until the tokenizer — and a header whose tokenizer comes first and runs past the fetched bytes still returns what it has rather than failing. The estimate is weights, plus the f16 KV cache for the context pool (llama.cpp's --ctx-size, which --parallel divides rather than multiplies), plus a gigabyte and five percent for the engine. It reuses the header the servability check already fetched, and the form says what it is made of. Checked against all twelve GGUF models on the live mirror, whose shapes read back as their published architectures. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
31 lines
918 B
Go
31 lines
918 B
Go
package gguf
|
|
|
|
import (
|
|
"os"
|
|
"strings"
|
|
"testing"
|
|
)
|
|
|
|
// TestRealHeaders reads real files when LLMGW_GGUF_URLS names them (space
|
|
// separated), and prints the shape and memory estimate for a 16384-token pool.
|
|
// Skipped otherwise: CI has no mirror to read.
|
|
func TestRealHeaders(t *testing.T) {
|
|
urls := strings.Fields(os.Getenv("LLMGW_GGUF_URLS"))
|
|
if len(urls) == 0 {
|
|
t.Skip("LLMGW_GGUF_URLS not set")
|
|
}
|
|
for _, u := range urls {
|
|
h, err := FromURL(t.Context(), nil, u)
|
|
if err != nil {
|
|
t.Errorf("%s: %v", u, err)
|
|
continue
|
|
}
|
|
t.Logf("%s: arch=%s layers=%d heads=%d kv_heads=%d embd=%d key_len=%d kv/token=%d B, kv@16k=%.2f GiB",
|
|
u[strings.LastIndex(u, "/")+1:], h.Architecture, h.BlockCount, h.HeadCount, h.HeadCountKV,
|
|
h.EmbeddingLength, h.KeyLength, h.KVBytesPerToken(), float64(16384*h.KVBytesPerToken())/(1<<30))
|
|
if h.KVBytesPerToken() == 0 {
|
|
t.Errorf("%s: no KV shape read", u)
|
|
}
|
|
}
|
|
}
|