Add the llama.cpp engine, and let a profile ask for no GPU
llama.cpp is what makes a cluster with no cards useful: it serves the same OpenAI API on CPU, so the whole path — placement, supervision, scraping, scheduling, admission — can be exercised end to end on ordinary machines. Three things had to give way for it. A profile may now omit its GPU class, and that is how it says it runs on the CPU. Only engines that can actually serve without a card may omit it; for vLLM an omitted class stays an error, because otherwise a typo becomes a replica placed on a machine with no hardware and diagnosed minutes later during model loading. Placement spreads CPU replicas instead of bin packing them, since unlike GPUs there is no allocation to stop two of them sharing a machine's cores. Metric mappings are now per engine rather than per cluster. llama.cpp calls the same three numbers requests_processing, requests_deferred and kv_cache_usage_ratio, and a single global mapping would leave half a mixed fleet unmeasurable. An empty mapping still defaults to vLLM's names; one that names its own Running metric is taken as a complete statement, so the fields it leaves blank stay blank rather than being filled with names that engine cannot have. The adapter refuses multi-node and tensor-parallel profiles. llama.cpp splits across local GPUs by a different mechanism than this adapter drives, and accepting either would produce a replica that starts and then serves from one machine while the catalog believes it has several. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
ec1b6b04df
commit
501db3d3e8
15 files changed
+697
-22
No files matched your search
@@ -30,6 +30,40 @@ type Mapping struct {
|
||||
SuccessTotal string
|
||||
}
|
||||
|
||||
// MappingFor returns the metric names for an engine.
|
||||
//
|
||||
// Per engine rather than per cluster: a cluster running llama.cpp on the CPU
|
||||
// nodes and vLLM on the GPU nodes has two vocabularies for the same three
|
||||
// numbers, and one global mapping would leave half the fleet unmeasurable.
|
||||
// An unknown engine falls back to vLLM's names, which is also what an endpoint
|
||||
// that names no engine gets.
|
||||
func MappingFor(engine string) Mapping {
|
||||
switch engine {
|
||||
case "llamacpp":
|
||||
return LlamaCppMapping()
|
||||
default:
|
||||
return VLLMMapping()
|
||||
}
|
||||
}
|
||||
|
||||
// LlamaCppMapping returns llama.cpp's metric names, which llama-server only
|
||||
// exposes when started with --metrics.
|
||||
//
|
||||
// "Deferred" is llama.cpp's word for a request that arrived with every slot
|
||||
// busy, which is the same thing vLLM calls waiting. The fields left empty are
|
||||
// metrics llama.cpp does not have: it counts decodes rather than requests, so
|
||||
// there is no success counter and no cache configuration to read block sizes
|
||||
// from.
|
||||
func LlamaCppMapping() Mapping {
|
||||
return Mapping{
|
||||
Running: "llamacpp:requests_processing",
|
||||
Waiting: "llamacpp:requests_deferred",
|
||||
KVCacheUsage: "llamacpp:kv_cache_usage_ratio",
|
||||
PromptTokens: "llamacpp:prompt_tokens_total",
|
||||
GenerationTokens: "llamacpp:tokens_predicted_total",
|
||||
}
|
||||
}
|
||||
|
||||
// VLLMMapping returns vLLM's current metric names.
|
||||
func VLLMMapping() Mapping {
|
||||
return Mapping{
|
||||
@@ -43,7 +77,16 @@ func VLLMMapping() Mapping {
|
||||
}
|
||||
}
|
||||
|
||||
// normalize fills in vLLM's names for an empty mapping.
|
||||
//
|
||||
// Only for an empty one. A mapping that names its own Running metric is a
|
||||
// complete statement about some other engine, and the fields it leaves blank
|
||||
// are metrics that engine does not expose — filling those with vLLM's names
|
||||
// would have the router hunting for metrics that cannot be there.
|
||||
func (m *Mapping) normalize() {
|
||||
if m.Running != "" {
|
||||
return
|
||||
}
|
||||
d := VLLMMapping()
|
||||
if m.Running == "" {
|
||||
m.Running = d.Running
|
||||
|
||||
Reference in new issue
Block a user