A replica says what its engine computes on, and llama.cpp's word on the CPU refuses a GPU placement
llama.cpp's start-up output is read as a shard becomes ready: the devices it used, its backend, and the layers it offloaded. That goes up with READY in a new ShardStatus.compute, is kept on the shard, and is shown by the admin API, ctl replicas (ON) and the console (COMPUTE). A shard given GPUs whose engine says it found no usable GPU or offloaded no layer is stopped and failed, even where nvidia-smi cannot be asked. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
f285aa0548
commit
0bd0d881c5
15 files changed
+517
-126
No files matched your search
@@ -3714,6 +3714,29 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"ComputeView": {
|
||||
"type": "object",
|
||||
"description": "ComputeView is where an engine runs its model, by its own account.",
|
||||
"properties": {
|
||||
"backend": {
|
||||
"type": "string",
|
||||
"description": "Backend is CUDA, Vulkan, Metal, ROCm, GPU or CPU."
|
||||
},
|
||||
"devices": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"layers_offloaded": {
|
||||
"type": "integer",
|
||||
"description": "LayersOffloaded of LayersTotal went to the devices; absent when unsaid."
|
||||
},
|
||||
"layers_total": {
|
||||
"type": "integer"
|
||||
}
|
||||
}
|
||||
},
|
||||
"ControlPlane": {
|
||||
"type": "object",
|
||||
"description": "ControlPlane is one control plane.",
|
||||
@@ -5779,6 +5802,10 @@
|
||||
"memory_bytes": {
|
||||
"type": "integer",
|
||||
"format": "int64"
|
||||
},
|
||||
"compute": {
|
||||
"$ref": "#/components/schemas/ComputeView",
|
||||
"description": "Compute is where the engine said it put the model as it became ready: absent when it did not say."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
Reference in new issue
Block a user