Stop naming a metric llama.cpp does not have, and say so when it happens
Deployed against real llama.cpp, every scrape failed. The mapping named llamacpp:kv_cache_usage_ratio; build b10991 serves fifteen metric families and none is a cache ratio. Because KV usage was mandatory, that one absent name threw away the two metrics the engine does publish, and the router scheduled to all three replicas blind while the cluster looked healthy. A mapping that leaves a metric unnamed now means the engine does not publish it, and the field stays zero. Naming one the engine does not serve is still a violation, so this does not become a way to tolerate a typo. Running and waiting can never be omitted: an engine that reports neither what it is doing nor what is queued cannot be scheduled against at all. For llama.cpp this costs the KV half of the saturation signal, which is a real loss softened by its fixed slot count — queue depth already says when it is full. The second half is why it took a deployment to notice. A contract violation was logged per scrape at debug level, so at the default level nothing said anything. It is now a warning, once, with a matching line when it clears. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
501db3d3e8
commit
f19db242ae
4 files changed
+227
-29
No files matched your search
@@ -50,15 +50,23 @@ func MappingFor(engine string) Mapping {
|
||||
// exposes when started with --metrics.
|
||||
//
|
||||
// "Deferred" is llama.cpp's word for a request that arrived with every slot
|
||||
// busy, which is the same thing vLLM calls waiting. The fields left empty are
|
||||
// metrics llama.cpp does not have: it counts decodes rather than requests, so
|
||||
// there is no success counter and no cache configuration to read block sizes
|
||||
// from.
|
||||
// busy, which is the same thing vLLM calls waiting.
|
||||
//
|
||||
// KV cache usage is deliberately absent: current llama.cpp does not publish it
|
||||
// (b10991 serves fifteen families and none of them is a cache ratio), and
|
||||
// naming a metric that is not there would fail the contract on every scrape and
|
||||
// leave the router with no numbers at all rather than two out of three. The
|
||||
// loss is smaller here than it would be for vLLM — llama.cpp serves a fixed
|
||||
// number of slots, so its queue depth already says when it is full — but it is
|
||||
// a loss: saturation falls back to queue depth alone.
|
||||
//
|
||||
// The rest are metrics llama.cpp does not have either. It counts decodes rather
|
||||
// than requests, so there is no success counter and no cache configuration to
|
||||
// read block sizes from.
|
||||
func LlamaCppMapping() Mapping {
|
||||
return Mapping{
|
||||
Running: "llamacpp:requests_processing",
|
||||
Waiting: "llamacpp:requests_deferred",
|
||||
KVCacheUsage: "llamacpp:kv_cache_usage_ratio",
|
||||
PromptTokens: "llamacpp:prompt_tokens_total",
|
||||
GenerationTokens: "llamacpp:tokens_predicted_total",
|
||||
}
|
||||
@@ -122,8 +130,11 @@ func (e *MissingError) Error() string {
|
||||
|
||||
// Parse converts Prometheus text-format metrics into Stats.
|
||||
//
|
||||
// If any of the mandatory three (running, waiting, kv usage) is absent it
|
||||
// returns *MissingError: routing to that endpoint would be blind.
|
||||
// A metric the mapping names and the engine does not serve is a contract
|
||||
// violation and returns *MissingError: routing to that endpoint would be blind.
|
||||
// A metric the mapping leaves unnamed is one the engine is known not to
|
||||
// publish, and stays zero — the router loses that signal for those endpoints
|
||||
// rather than losing the endpoints.
|
||||
func Parse(r io.Reader, m Mapping) (endpoint.Stats, error) {
|
||||
m.normalize()
|
||||
|
||||
@@ -138,6 +149,9 @@ func Parse(r io.Reader, m Mapping) (endpoint.Stats, error) {
|
||||
var s endpoint.Stats
|
||||
var missing []string
|
||||
|
||||
// Running and waiting have no such escape: an engine that reports neither
|
||||
// how much it is doing nor how much is queued cannot be scheduled against
|
||||
// at all.
|
||||
if v, ok := firstValue(fams, m.Running); ok {
|
||||
s.Running = int(v)
|
||||
} else {
|
||||
@@ -148,10 +162,12 @@ func Parse(r io.Reader, m Mapping) (endpoint.Stats, error) {
|
||||
} else {
|
||||
missing = append(missing, m.Waiting)
|
||||
}
|
||||
if v, ok := firstValue(fams, m.KVCacheUsage); ok {
|
||||
s.KVCacheUsage = v
|
||||
} else {
|
||||
missing = append(missing, m.KVCacheUsage)
|
||||
if m.KVCacheUsage != "" {
|
||||
if v, ok := firstValue(fams, m.KVCacheUsage); ok {
|
||||
s.KVCacheUsage = v
|
||||
} else {
|
||||
missing = append(missing, m.KVCacheUsage)
|
||||
}
|
||||
}
|
||||
if len(missing) > 0 {
|
||||
return endpoint.Stats{}, &MissingError{Missing: missing}
|
||||
|
||||
Reference in new issue
Block a user