engine-check computes a token on the GPU, and finds the installed llama.cpp
Seeing a device is not computing on it. For llama.cpp on a machine with NVIDIA GPUs, engine-check now loads a small GGUF (--model, or the smallest in the weights cache) with every layer on the GPU, asks for one token, and reads from llama.cpp's own output where the layers went: a build that computes on the CPU fails, naming the fix. With no --bin it checks the build the agent installed under /opt/llmgw/engines. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
0bd0d881c5
commit
e4e15200d6
3 files changed
+217
-1
No files matched your search
@@ -277,6 +277,9 @@ func (l *LlamaCpp) bin() string {
|
||||
return "llama-server"
|
||||
}
|
||||
|
||||
// Binary is the llama-server this engine runs.
|
||||
func (l *LlamaCpp) Binary() string { return l.bin() }
|
||||
|
||||
// FallsBackToCPU says that llama.cpp keeps running when its GPU backend does
|
||||
// not load: --n-gpu-layers asks for layers on a GPU it then has none of, and
|
||||
// the server comes up and answers on the CPU, at a tenth of the speed, with no
|
||||
|
||||
Reference in new issue
Block a user