What a replica computes on comes from nvidia-smi; llama.cpp says its layers only at -lv 4

Checked against the installed llama.cpp (b10991): at its default verbosity
it prints neither the devices it uses nor the layers it offloads, so the
start-up output said nothing and every replica's ON column stayed empty.
The agent now takes compute from nvidia-smi - the GPUs the shard's processes
hold memory on, and how much - or CPU, for every engine, falling back to it
when llama.cpp's output says nothing; adopted engines get it too, and each
session reports it once per shard as a compute-only status. engine-check
runs its GPU token at -lv 4, where llama.cpp does say where its layers went.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-07 14:04:08 +02:00
1 parent 25b3e129fe
commit b5d650af16
7 files changed
+147 -13

No files matched your search

+11 -9
View File
@@ -28,15 +28,16 @@ What was left of the field report from v0.10.6.
### Added
- **A replica says what its engine computes on**: llama.cpp's start-up output
is read as a shard becomes ready — the devices, the backend, the layers
offloaded — and shown by `ctl replicas` (ON: `CUDA 49/49`, `CPU`), the
console and the admin API.
- **A replica says what it computes on**: `ctl replicas` (ON), the console
and the admin API show `CUDA` with the GPUs its processes hold memory on
and how much, as nvidia-smi sees them, or `CPU` — for every engine, kept by
the agent across its own and the control plane's restarts. llama.cpp's
layer count is added when it says it, which it does only at `-lv 4`.
- **`ctl engine-check llamacpp` computes a token on the GPU**: a small GGUF
(`--model`, or the smallest in the weights cache) is loaded with every layer
on the GPU and asked for one token; a build whose CUDA backend does not load
fails, by llama.cpp's own account. With no `--bin` it checks the build the
agent installed.
on the GPU at `-lv 4` and asked for one token; a build whose CUDA backend
does not load fails, by llama.cpp's own account of its layers. With no
`--bin` it checks the build the agent installed.
- **Engine versions and compute capability on nodes**: `ctl nodes` shows
`llamacpp b10991` and `6000-24g (7.5)`; the admin API gives
`engine_versions` and `compute_capabilities`. A profile's `engine_version`
@@ -64,8 +65,9 @@ What was left of the field report from v0.10.6.
### Changed
- **A GPU placement llama.cpp says it computes on the CPU is refused**, from
its own output, even where nvidia-smi cannot be asked.
- **A GPU placement llama.cpp says it cannot compute on is refused** when it
warns that it found no usable GPU, even where nvidia-smi cannot be asked;
where it can, holding no memory on its GPUs refuses it as before.
## v0.12.2 — 2026-10-06