A replica says what its engine computes on, and llama.cpp's word on the CPU refuses a GPU placement

llama.cpp's start-up output is read as a shard becomes ready: the devices it
used, its backend, and the layers it offloaded. That goes up with READY in a
new ShardStatus.compute, is kept on the shard, and is shown by the admin
API, ctl replicas (ON) and the console (COMPUTE). A shard given GPUs whose
engine says it found no usable GPU or offloaded no layer is stopped and
failed, even where nvidia-smi cannot be asked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-07 13:02:17 +02:00
1 parent f285aa0548
commit 0bd0d881c5
15 files changed
+517 -126

No files matched your search

+16
View File
@@ -605,6 +605,18 @@ message ProvisionStatus {
string detail = 5;
}
// Compute is where an engine runs its model, as it said when it loaded it.
message Compute {
// backend is "CUDA", "Vulkan", "Metal" or "CPU".
string backend = 1;
// devices are the devices it named: "CUDA0: Quadro RTX 6000 (7.5)".
repeated string devices = 2;
// layers_offloaded of layers_total went to the devices; both 0 when the
// engine did not say.
int32 layers_offloaded = 3;
int32 layers_total = 4;
}
// ShardStatus reports a shard's lifecycle transition.
message ShardStatus {
string node_id = 1;
@@ -626,6 +638,10 @@ message ShardStatus {
// going to look.
FetchProgress fetch = 8;
// compute is what the engine says it computes on, read from its own output
// as it became ready: sent with STATE_READY when the engine says.
Compute compute = 9;
enum State {
STATE_UNSPECIFIED = 0;
STATE_PENDING = 1;