CI / test (push) Has been cancelled
llmgw ctl model plan Qwen/Qwen2.5-Coder-32B-Instruct The arithmetic this replaces is the last thing between having GPUs and serving a model on them. Choosing tp, gpu_class and a context that will actually fit means knowing the layer count, the KV head count, what the weights weigh at a chosen quantisation and how much of a card the runtime keeps — and getting it wrong is not an error message, it is an out-of-memory twenty minutes into a load. I computed those by hand for every model on this cluster. Both halves are read rather than retyped: the architecture from the model's own config.json and parameter count, the hardware from the running fleet. Attention heads in place of KV heads is the classic slip and is worth a factor of five on a model with grouped-query attention, so it is not a number anyone should be entering. Every candidate is listed, not one answer. With cards to spare you want the full-precision fit; with one card free you want to know a quantised one exists. Choosing between those is not the planner's job. The tests are calibrated against files actually downloaded here — llama.cpp reported 14.62 GiB and 8.37 GiB for the 14B at q8_0 and q4_K_M, and the model reproduces both — and they caught two errors in my own arithmetic. The overhead was charged per rank from the whole model's weights rather than each rank's share, and it reserved a tenth of a card that usableFraction had already reserved. Between them, Llama-3.3-70B at bf16 across two 80 GB cards came out as not fitting, which is a configuration people run every day. Slack reserved in two places, each unaware of the other, reads as caution and behaves as a bug. Pools are grouped by class and fabric together, because a replica may never span two fabrics: eight free cards split across two of them are not eight cards for a model that needs eight, and merging them would plan something that starts, passes its health checks and hangs on its first all-reduce. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
139 lines
5.4 KiB
Go
139 lines
5.4 KiB
Go
// Package modelplan works out whether a model fits some hardware, and how it
|
|
// should be split when it does.
|
|
//
|
|
// It exists because that arithmetic was the real barrier to running a model
|
|
// here. Everything else about adding one is typing: the name, the path, the
|
|
// engine. Deciding tp, gpus_per_node, gpu_class and the context that will
|
|
// actually fit means knowing the model's layer count, its KV head count, what
|
|
// its weights weigh at a given quantisation, and how much of a card is left
|
|
// after the runtime takes its share — and getting it wrong is not an error
|
|
// message, it is an out-of-memory twenty minutes into a load.
|
|
package modelplan
|
|
|
|
import "fmt"
|
|
|
|
// Model is what a model's own configuration says about it.
|
|
//
|
|
// Every field here is read from the model rather than guessed: config.json
|
|
// carries the architecture, and the repository's index carries the parameter
|
|
// count. A guess would defeat the point, because the answer is only worth
|
|
// having if it is the one the hardware will agree with.
|
|
type Model struct {
|
|
Name string
|
|
// Params is the total parameter count.
|
|
Params int64
|
|
// Layers, KVHeads and HeadDim size the KV cache, which is what scales with
|
|
// context and is usually what actually runs a card out of memory.
|
|
Layers int
|
|
KVHeads int
|
|
HeadDim int
|
|
// MaxContext is the model's trained context length.
|
|
MaxContext int
|
|
}
|
|
|
|
// Precision is how many bytes one parameter weighs.
|
|
//
|
|
// Named rather than numeric because the names are what people use, and because
|
|
// the difference between them is the difference between a model fitting and
|
|
// not.
|
|
type Precision struct {
|
|
Name string
|
|
BytesPerParam float64
|
|
BytesPerKVElem float64
|
|
}
|
|
|
|
// Known precisions, from full to the quantisations worth serving.
|
|
var (
|
|
BF16 = Precision{"bf16", 2, 2}
|
|
FP8 = Precision{"fp8", 1, 1}
|
|
INT4 = Precision{"int4", 0.5, 2} // weights quantised, KV usually is not
|
|
Q8GGUF = Precision{"q8_0", 1.06, 2}
|
|
Q4GGUF = Precision{"q4_k_m", 0.61, 2}
|
|
)
|
|
|
|
// PrecisionByName resolves a precision, reporting what it knows when it cannot.
|
|
func PrecisionByName(s string) (Precision, error) {
|
|
for _, p := range []Precision{BF16, FP8, INT4, Q8GGUF, Q4GGUF} {
|
|
if p.Name == s {
|
|
return p, nil
|
|
}
|
|
}
|
|
return Precision{}, fmt.Errorf("unknown precision %q; known: bf16, fp8, int4, q8_0, q4_k_m", s)
|
|
}
|
|
|
|
// Demand is what one replica needs, in bytes.
|
|
type Demand struct {
|
|
Weights int64
|
|
KVCache int64
|
|
// Overhead is what the runtime costs on a single GPU. Across a tensor
|
|
// parallel group it is recomputed from each rank's share — see PerGPU.
|
|
Overhead int64
|
|
}
|
|
|
|
// Total is everything one replica needs across all its GPUs.
|
|
func (d Demand) Total() int64 { return d.Weights + d.KVCache + d.Overhead }
|
|
|
|
// PerGPU is what one GPU of a replica must hold.
|
|
//
|
|
// Tensor parallelism splits the weights and the KV cache, so both divide. The
|
|
// overhead does not divide — each process pays its own CUDA context and
|
|
// allocator — but neither is it constant: activations scale with the shard the
|
|
// rank actually holds, so it is recomputed from the divided weights.
|
|
//
|
|
// Charging the whole model's overhead to every rank was the first version of
|
|
// this, and it was badly wrong in exactly the case that matters. Llama-3.3-70B
|
|
// at bf16 across two 80 GB cards came out at 85.9 GB a card and was reported as
|
|
// not fitting, which is a configuration people run every day.
|
|
func (d Demand) PerGPU(gpus int) int64 {
|
|
if gpus <= 0 {
|
|
gpus = 1
|
|
}
|
|
n := int64(gpus)
|
|
return (d.Weights+d.KVCache)/n + overheadBytes(d.Weights/n)
|
|
}
|
|
|
|
// KVBytesPerToken is what one token of context costs, for the whole model.
|
|
//
|
|
// Two for the key and the value, times the layers, times the heads the cache
|
|
// actually keeps. Grouped-query attention is why KVHeads and not the attention
|
|
// head count: a 14B model with 40 attention heads and 8 KV heads pays for
|
|
// eight, and the difference is a factor of five.
|
|
func (m Model) KVBytesPerToken(p Precision) int64 {
|
|
return int64(2 * float64(m.Layers) * float64(m.KVHeads) * float64(m.HeadDim) * p.BytesPerKVElem)
|
|
}
|
|
|
|
// Needs computes what one replica of this model needs.
|
|
//
|
|
// context is the total token pool the engine will hold, which is the context
|
|
// length times however many requests it serves at once — the number an engine
|
|
// is configured with, not the number a single request sees.
|
|
func (m Model) Needs(p Precision, context int) Demand {
|
|
weights := int64(float64(m.Params) * p.BytesPerParam)
|
|
kv := int64(context) * m.KVBytesPerToken(p)
|
|
return Demand{
|
|
Weights: weights,
|
|
KVCache: kv,
|
|
Overhead: overheadBytes(weights),
|
|
}
|
|
}
|
|
|
|
// overheadBytes is what the runtime costs beyond weights and cache, per GPU.
|
|
//
|
|
// A flat allowance for the CUDA context, the framework and its allocator —
|
|
// which a small model pays as much as a large one — plus a small fraction of
|
|
// the shard for activations, which do scale with it.
|
|
//
|
|
// The fraction is two percent rather than the ten it started at, because the
|
|
// ten was counted twice. usableFraction already holds back a tenth of every
|
|
// card for exactly this, so charging another tenth on top made a plan reject
|
|
// Llama-3.3-70B at bf16 across two 80 GB cards — a configuration people run
|
|
// every day. Reserving slack in two places, each unaware of the other, reads as
|
|
// caution and behaves as a bug.
|
|
func overheadBytes(weights int64) int64 {
|
|
const flat = 2 << 30 // 2 GiB
|
|
return flat + weights/50
|
|
}
|
|
|
|
// GB renders bytes the way people discuss hardware.
|
|
func GB(b int64) float64 { return float64(b) / (1 << 30) }
|