Files
llmgw/internal/modelplan/size.go
T
Ilker ManapandClaude Opus 5 300e298053
CI / test (push) Has been cancelled
Work out whether a model fits the cluster, and how to split it
llmgw ctl model plan Qwen/Qwen2.5-Coder-32B-Instruct

The arithmetic this replaces is the last thing between having GPUs and
serving a model on them. Choosing tp, gpu_class and a context that will
actually fit means knowing the layer count, the KV head count, what the
weights weigh at a chosen quantisation and how much of a card the runtime
keeps — and getting it wrong is not an error message, it is an
out-of-memory twenty minutes into a load. I computed those by hand for
every model on this cluster.

Both halves are read rather than retyped: the architecture from the model's
own config.json and parameter count, the hardware from the running fleet.
Attention heads in place of KV heads is the classic slip and is worth a
factor of five on a model with grouped-query attention, so it is not a
number anyone should be entering.

Every candidate is listed, not one answer. With cards to spare you want the
full-precision fit; with one card free you want to know a quantised one
exists. Choosing between those is not the planner's job.

The tests are calibrated against files actually downloaded here — llama.cpp
reported 14.62 GiB and 8.37 GiB for the 14B at q8_0 and q4_K_M, and the
model reproduces both — and they caught two errors in my own arithmetic.
The overhead was charged per rank from the whole model's weights rather
than each rank's share, and it reserved a tenth of a card that
usableFraction had already reserved. Between them, Llama-3.3-70B at bf16
across two 80 GB cards came out as not fitting, which is a configuration
people run every day. Slack reserved in two places, each unaware of the
other, reads as caution and behaves as a bug.

Pools are grouped by class and fabric together, because a replica may never
span two fabrics: eight free cards split across two of them are not eight
cards for a model that needs eight, and merging them would plan something
that starts, passes its health checks and hangs on its first all-reduce.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 07:46:53 +02:00

139 lines
5.4 KiB
Go

// Package modelplan works out whether a model fits some hardware, and how it
// should be split when it does.
//
// It exists because that arithmetic was the real barrier to running a model
// here. Everything else about adding one is typing: the name, the path, the
// engine. Deciding tp, gpus_per_node, gpu_class and the context that will
// actually fit means knowing the model's layer count, its KV head count, what
// its weights weigh at a given quantisation, and how much of a card is left
// after the runtime takes its share — and getting it wrong is not an error
// message, it is an out-of-memory twenty minutes into a load.
package modelplan
import "fmt"
// Model is what a model's own configuration says about it.
//
// Every field here is read from the model rather than guessed: config.json
// carries the architecture, and the repository's index carries the parameter
// count. A guess would defeat the point, because the answer is only worth
// having if it is the one the hardware will agree with.
type Model struct {
Name string
// Params is the total parameter count.
Params int64
// Layers, KVHeads and HeadDim size the KV cache, which is what scales with
// context and is usually what actually runs a card out of memory.
Layers int
KVHeads int
HeadDim int
// MaxContext is the model's trained context length.
MaxContext int
}
// Precision is how many bytes one parameter weighs.
//
// Named rather than numeric because the names are what people use, and because
// the difference between them is the difference between a model fitting and
// not.
type Precision struct {
Name string
BytesPerParam float64
BytesPerKVElem float64
}
// Known precisions, from full to the quantisations worth serving.
var (
BF16 = Precision{"bf16", 2, 2}
FP8 = Precision{"fp8", 1, 1}
INT4 = Precision{"int4", 0.5, 2} // weights quantised, KV usually is not
Q8GGUF = Precision{"q8_0", 1.06, 2}
Q4GGUF = Precision{"q4_k_m", 0.61, 2}
)
// PrecisionByName resolves a precision, reporting what it knows when it cannot.
func PrecisionByName(s string) (Precision, error) {
for _, p := range []Precision{BF16, FP8, INT4, Q8GGUF, Q4GGUF} {
if p.Name == s {
return p, nil
}
}
return Precision{}, fmt.Errorf("unknown precision %q; known: bf16, fp8, int4, q8_0, q4_k_m", s)
}
// Demand is what one replica needs, in bytes.
type Demand struct {
Weights int64
KVCache int64
// Overhead is what the runtime costs on a single GPU. Across a tensor
// parallel group it is recomputed from each rank's share — see PerGPU.
Overhead int64
}
// Total is everything one replica needs across all its GPUs.
func (d Demand) Total() int64 { return d.Weights + d.KVCache + d.Overhead }
// PerGPU is what one GPU of a replica must hold.
//
// Tensor parallelism splits the weights and the KV cache, so both divide. The
// overhead does not divide — each process pays its own CUDA context and
// allocator — but neither is it constant: activations scale with the shard the
// rank actually holds, so it is recomputed from the divided weights.
//
// Charging the whole model's overhead to every rank was the first version of
// this, and it was badly wrong in exactly the case that matters. Llama-3.3-70B
// at bf16 across two 80 GB cards came out at 85.9 GB a card and was reported as
// not fitting, which is a configuration people run every day.
func (d Demand) PerGPU(gpus int) int64 {
if gpus <= 0 {
gpus = 1
}
n := int64(gpus)
return (d.Weights+d.KVCache)/n + overheadBytes(d.Weights/n)
}
// KVBytesPerToken is what one token of context costs, for the whole model.
//
// Two for the key and the value, times the layers, times the heads the cache
// actually keeps. Grouped-query attention is why KVHeads and not the attention
// head count: a 14B model with 40 attention heads and 8 KV heads pays for
// eight, and the difference is a factor of five.
func (m Model) KVBytesPerToken(p Precision) int64 {
return int64(2 * float64(m.Layers) * float64(m.KVHeads) * float64(m.HeadDim) * p.BytesPerKVElem)
}
// Needs computes what one replica of this model needs.
//
// context is the total token pool the engine will hold, which is the context
// length times however many requests it serves at once — the number an engine
// is configured with, not the number a single request sees.
func (m Model) Needs(p Precision, context int) Demand {
weights := int64(float64(m.Params) * p.BytesPerParam)
kv := int64(context) * m.KVBytesPerToken(p)
return Demand{
Weights: weights,
KVCache: kv,
Overhead: overheadBytes(weights),
}
}
// overheadBytes is what the runtime costs beyond weights and cache, per GPU.
//
// A flat allowance for the CUDA context, the framework and its allocator —
// which a small model pays as much as a large one — plus a small fraction of
// the shard for activations, which do scale with it.
//
// The fraction is two percent rather than the ten it started at, because the
// ten was counted twice. usableFraction already holds back a tenth of every
// card for exactly this, so charging another tenth on top made a plan reject
// Llama-3.3-70B at bf16 across two 80 GB cards — a configuration people run
// every day. Reserving slack in two places, each unaware of the other, reads as
// caution and behaves as a bug.
func overheadBytes(weights int64) int64 {
const flat = 2 << 30 // 2 GiB
return flat + weights/50
}
// GB renders bytes the way people discuss hardware.
func GB(b int64) float64 { return float64(b) / (1 << 30) }