What a replica computes on comes from nvidia-smi; llama.cpp says its layers only at -lv 4
Checked against the installed llama.cpp (b10991): at its default verbosity it prints neither the devices it uses nor the layers it offloads, so the start-up output said nothing and every replica's ON column stayed empty. The agent now takes compute from nvidia-smi - the GPUs the shard's processes hold memory on, and how much - or CPU, for every engine, falling back to it when llama.cpp's output says nothing; adopted engines get it too, and each session reports it once per shard as a compute-only status. engine-check runs its GPU token at -lv 4, where llama.cpp does say where its layers went. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
25b3e129fe
commit
b5d650af16
7 files changed
+147
-13
No files matched your search
@@ -363,7 +363,9 @@ func computeOnGPU(ctx context.Context, bin, model string, out io.Writer) error {
|
||||
defer cancel()
|
||||
var logs syncBuffer
|
||||
c := exec.CommandContext(ctx, bin, "-m", model, "--host", "127.0.0.1", "--port", strconv.Itoa(port), //nolint:gosec // the engine's own binary
|
||||
"--n-gpu-layers", "999", "--ctx-size", "512", "--parallel", "1")
|
||||
"--n-gpu-layers", "999", "--ctx-size", "512", "--parallel", "1",
|
||||
// Where the layers went is said only at this verbosity.
|
||||
"-lv", "4")
|
||||
c.Stdout, c.Stderr = &logs, &logs
|
||||
if err := c.Start(); err != nil {
|
||||
return fmt.Errorf("starting %s: %w", bin, err)
|
||||
|
||||
Reference in new issue
Block a user