Stop naming a metric llama.cpp does not have, and say so when it happens

Deployed against real llama.cpp, every scrape failed. The mapping named
llamacpp:kv_cache_usage_ratio; build b10991 serves fifteen metric families
and none is a cache ratio. Because KV usage was mandatory, that one absent
name threw away the two metrics the engine does publish, and the router
scheduled to all three replicas blind while the cluster looked healthy.

A mapping that leaves a metric unnamed now means the engine does not publish
it, and the field stays zero. Naming one the engine does not serve is still a
violation, so this does not become a way to tolerate a typo. Running and
waiting can never be omitted: an engine that reports neither what it is doing
nor what is queued cannot be scheduled against at all. For llama.cpp this
costs the KV half of the saturation signal, which is a real loss softened by
its fixed slot count — queue depth already says when it is full.

The second half is why it took a deployment to notice. A contract violation
was logged per scrape at debug level, so at the default level nothing said
anything. It is now a warning, once, with a matching line when it clears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 08:18:27 +02:00
1 parent 501db3d3e8
commit f19db242ae
4 files changed
+227 -29

No files matched your search

+27 -11
View File
@@ -50,15 +50,23 @@ func MappingFor(engine string) Mapping {
// exposes when started with --metrics.
//
// "Deferred" is llama.cpp's word for a request that arrived with every slot
// busy, which is the same thing vLLM calls waiting. The fields left empty are
// metrics llama.cpp does not have: it counts decodes rather than requests, so
// there is no success counter and no cache configuration to read block sizes
// from.
// busy, which is the same thing vLLM calls waiting.
//
// KV cache usage is deliberately absent: current llama.cpp does not publish it
// (b10991 serves fifteen families and none of them is a cache ratio), and
// naming a metric that is not there would fail the contract on every scrape and
// leave the router with no numbers at all rather than two out of three. The
// loss is smaller here than it would be for vLLM — llama.cpp serves a fixed
// number of slots, so its queue depth already says when it is full — but it is
// a loss: saturation falls back to queue depth alone.
//
// The rest are metrics llama.cpp does not have either. It counts decodes rather
// than requests, so there is no success counter and no cache configuration to
// read block sizes from.
func LlamaCppMapping() Mapping {
return Mapping{
Running: "llamacpp:requests_processing",
Waiting: "llamacpp:requests_deferred",
KVCacheUsage: "llamacpp:kv_cache_usage_ratio",
PromptTokens: "llamacpp:prompt_tokens_total",
GenerationTokens: "llamacpp:tokens_predicted_total",
}
@@ -122,8 +130,11 @@ func (e *MissingError) Error() string {
// Parse converts Prometheus text-format metrics into Stats.
//
// If any of the mandatory three (running, waiting, kv usage) is absent it
// returns *MissingError: routing to that endpoint would be blind.
// A metric the mapping names and the engine does not serve is a contract
// violation and returns *MissingError: routing to that endpoint would be blind.
// A metric the mapping leaves unnamed is one the engine is known not to
// publish, and stays zero — the router loses that signal for those endpoints
// rather than losing the endpoints.
func Parse(r io.Reader, m Mapping) (endpoint.Stats, error) {
m.normalize()
@@ -138,6 +149,9 @@ func Parse(r io.Reader, m Mapping) (endpoint.Stats, error) {
var s endpoint.Stats
var missing []string
// Running and waiting have no such escape: an engine that reports neither
// how much it is doing nor how much is queued cannot be scheduled against
// at all.
if v, ok := firstValue(fams, m.Running); ok {
s.Running = int(v)
} else {
@@ -148,10 +162,12 @@ func Parse(r io.Reader, m Mapping) (endpoint.Stats, error) {
} else {
missing = append(missing, m.Waiting)
}
if v, ok := firstValue(fams, m.KVCacheUsage); ok {
s.KVCacheUsage = v
} else {
missing = append(missing, m.KVCacheUsage)
if m.KVCacheUsage != "" {
if v, ok := firstValue(fams, m.KVCacheUsage); ok {
s.KVCacheUsage = v
} else {
missing = append(missing, m.KVCacheUsage)
}
}
if len(missing) > 0 {
return endpoint.Stats{}, &MissingError{Missing: missing}