Files
llmgw/internal/agent/engines/llamacpp.go
T
Ilker ManapandClaude Opus 5 501db3d3e8 Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same
OpenAI API on CPU, so the whole path — placement, supervision, scraping,
scheduling, admission — can be exercised end to end on ordinary machines.

Three things had to give way for it.

A profile may now omit its GPU class, and that is how it says it runs on the
CPU. Only engines that can actually serve without a card may omit it; for
vLLM an omitted class stays an error, because otherwise a typo becomes a
replica placed on a machine with no hardware and diagnosed minutes later
during model loading. Placement spreads CPU replicas instead of bin packing
them, since unlike GPUs there is no allocation to stop two of them sharing a
machine's cores.

Metric mappings are now per engine rather than per cluster. llama.cpp calls
the same three numbers requests_processing, requests_deferred and
kv_cache_usage_ratio, and a single global mapping would leave half a mixed
fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that
names its own Running metric is taken as a complete statement, so the fields
it leaves blank stay blank rather than being filled with names that engine
cannot have.

The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits
across local GPUs by a different mechanism than this adapter drives, and
accepting either would produce a replica that starts and then serves from one
machine while the catalog believes it has several.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 08:14:05 +02:00

118 lines
4.0 KiB
Go

package engines
import (
"fmt"
"strconv"
"strings"
)
// LlamaCppConfig locates the llama.cpp installation.
type LlamaCppConfig struct {
// Bin is the llama-server binary. It is the server, not the CLI: the
// gateway talks OpenAI over HTTP and needs the one that serves it.
Bin string `koanf:"bin"`
// Parallel is how many requests a server handles at once, when a profile
// does not say. llama.cpp defaults to one, which would make every second
// request wait behind the first no matter what the router decided.
Parallel int `koanf:"parallel"`
}
// LlamaCpp builds llama.cpp launch commands.
//
// It is the engine that makes a cluster without GPUs useful: llama.cpp runs a
// small model on CPU at a speed that is fine for testing the whole path, and it
// serves the same OpenAI API as the others, so nothing downstream can tell the
// difference.
type LlamaCpp struct{ cfg LlamaCppConfig }
// Name identifies the engine in the catalog.
func (*LlamaCpp) Name() string { return "llamacpp" }
// HealthPath is llama.cpp's readiness endpoint. It returns 503 while the model
// is still loading, which is exactly what the supervisor needs.
func (*LlamaCpp) HealthPath() string { return "/health" }
// MetricsPath is llama.cpp's Prometheus endpoint, which only exists when the
// server was started with --metrics. Commands always passes it.
func (*LlamaCpp) MetricsPath() string { return "/metrics" }
// ServesHTTP reports whether this rank answers the API. Every llama.cpp shard
// does, because a llama.cpp replica is always exactly one shard.
func (*LlamaCpp) ServesHTTP(ShardSpec) bool { return true }
// Validate checks the profile against the GPUs it would run on.
//
// The two refusals here are about not pretending: llama.cpp has no equivalent
// of a tensor-parallel group spanning machines, and its multi-GPU support is a
// different mechanism (--split-mode, --tensor-split) that this adapter does not
// drive. Accepting either would produce a replica that starts and then serves
// from one machine while the catalog believes it has several.
func (*LlamaCpp) Validate(p Profile, gpus []GPUInfo) error {
if err := validateCommon(p, gpus); err != nil {
return err
}
if p.Nodes > 1 {
return fmt.Errorf("profile %q: llamacpp does not span nodes; use vllm or sglang for a %d-node replica",
p.Name, p.Nodes)
}
if p.TP > 1 {
return fmt.Errorf("profile %q: llamacpp does not do tensor parallelism; "+
"it splits a model across local GPUs with --tensor-split, which this adapter does not drive",
p.Name)
}
return nil
}
// Commands returns the processes for one shard, which is always one process.
func (l *LlamaCpp) Commands(s ShardSpec) ([]Command, error) {
bin := l.cfg.Bin
if bin == "" {
bin = "llama-server"
}
p := s.Profile
args := []string{
"--model", p.ModelPath,
"--host", "0.0.0.0",
"--port", strconv.Itoa(s.HTTPPort),
// Without this there is no /metrics, and an endpoint the router cannot
// measure is one it has to route to blind.
"--metrics",
}
if p.ServedModelName != "" {
args = append(args, "--alias", p.ServedModelName)
}
if !hasFlag(p.Args, "--parallel", "-np") {
n := l.cfg.Parallel
if n <= 0 {
n = 4
}
args = append(args, "--parallel", strconv.Itoa(n))
}
// With GPUs assigned, offload everything; llama.cpp keeps the model on the
// CPU otherwise, which is not what a GPU placement asked for. Without them
// the flag is omitted entirely rather than set to zero, so a build with no
// GPU support does not see an argument it rejects.
if len(s.GPUIndices) > 0 && !hasFlag(p.Args, "--n-gpu-layers", "-ngl") {
args = append(args, "--n-gpu-layers", "999")
}
args = append(args, p.Args...)
return []Command{{
Name: "llama-server", Path: bin, Args: args, Env: materializeEnv(baseEnv(s)),
}}, nil
}
// hasFlag reports whether the profile already sets one of these flags, in
// either "--flag value" or "--flag=value" form.
func hasFlag(args []string, names ...string) bool {
for _, a := range args {
for _, n := range names {
if a == n || strings.HasPrefix(a, n+"=") {
return true
}
}
}
return false
}