What a replica computes on comes from nvidia-smi; llama.cpp says its layers only at -lv 4
Checked against the installed llama.cpp (b10991): at its default verbosity it prints neither the devices it uses nor the layers it offloads, so the start-up output said nothing and every replica's ON column stayed empty. The agent now takes compute from nvidia-smi - the GPUs the shard's processes hold memory on, and how much - or CPU, for every engine, falling back to it when llama.cpp's output says nothing; adopted engines get it too, and each session reports it once per shard as a compute-only status. engine-check runs its GPU token at -lv 4, where llama.cpp does say where its layers went. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
25b3e129fe
commit
b5d650af16
7 files changed
+147
-13
No files matched your search
+11
-9
@@ -28,15 +28,16 @@ What was left of the field report from v0.10.6.
|
||||
|
||||
### Added
|
||||
|
||||
- **A replica says what its engine computes on**: llama.cpp's start-up output
|
||||
is read as a shard becomes ready — the devices, the backend, the layers
|
||||
offloaded — and shown by `ctl replicas` (ON: `CUDA 49/49`, `CPU`), the
|
||||
console and the admin API.
|
||||
- **A replica says what it computes on**: `ctl replicas` (ON), the console
|
||||
and the admin API show `CUDA` with the GPUs its processes hold memory on
|
||||
and how much, as nvidia-smi sees them, or `CPU` — for every engine, kept by
|
||||
the agent across its own and the control plane's restarts. llama.cpp's
|
||||
layer count is added when it says it, which it does only at `-lv 4`.
|
||||
- **`ctl engine-check llamacpp` computes a token on the GPU**: a small GGUF
|
||||
(`--model`, or the smallest in the weights cache) is loaded with every layer
|
||||
on the GPU and asked for one token; a build whose CUDA backend does not load
|
||||
fails, by llama.cpp's own account. With no `--bin` it checks the build the
|
||||
agent installed.
|
||||
on the GPU at `-lv 4` and asked for one token; a build whose CUDA backend
|
||||
does not load fails, by llama.cpp's own account of its layers. With no
|
||||
`--bin` it checks the build the agent installed.
|
||||
- **Engine versions and compute capability on nodes**: `ctl nodes` shows
|
||||
`llamacpp b10991` and `6000-24g (7.5)`; the admin API gives
|
||||
`engine_versions` and `compute_capabilities`. A profile's `engine_version`
|
||||
@@ -64,8 +65,9 @@ What was left of the field report from v0.10.6.
|
||||
|
||||
### Changed
|
||||
|
||||
- **A GPU placement llama.cpp says it computes on the CPU is refused**, from
|
||||
its own output, even where nvidia-smi cannot be asked.
|
||||
- **A GPU placement llama.cpp says it cannot compute on is refused** when it
|
||||
warns that it found no usable GPU, even where nvidia-smi cannot be asked;
|
||||
where it can, holding no memory on its GPUs refuses it as before.
|
||||
|
||||
## v0.12.2 — 2026-10-06
|
||||
|
||||
|
||||
Reference in new issue
Block a user