Do not call a GPU shard ready while it computes on the CPU

llama.cpp whose CUDA backend cannot load runs on the CPU without an
error. The CUDA build is now asked for its devices when installed, a
shard on GPUs must hold memory on each before it is ready, and
engine-check reports the devices. Shards get CUDA_DEVICE_ORDER=PCI_BUS_ID
so CUDA numbers GPUs as placement does.

Refs #2 (item 2).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-01 11:54:39 +02:00
1 parent 0c7934ba96
commit 057be16a92
13 files changed
+510 -2

No files matched your search

+15
View File
@@ -9,6 +9,7 @@
package engines
import (
"context"
"fmt"
"os"
"slices"
@@ -146,6 +147,15 @@ type Engine interface {
MetricsPath() string
}
// CPUFallback is an engine that can come up on the CPU when it was placed on
// GPUs, and keep serving there without an error. The agent checks such a shard
// is on its GPUs before calling it ready.
type CPUFallback interface {
FallsBackToCPU() bool
// Devices lists the GPUs the engine's build can see, for saying why.
Devices(ctx context.Context) ([]string, error)
}
// GPUInfo is the subset of GPU facts validation needs.
type GPUInfo struct {
Index int
@@ -232,6 +242,11 @@ func NCCLEnv(s ShardSpec) map[string]string { return baseEnv(s) }
func baseEnv(s ShardSpec) map[string]string {
env := map[string]string{
"CUDA_VISIBLE_DEVICES": joinInts(s.GPUIndices),
// The indices above are nvidia-smi's, which counts by PCI bus. CUDA
// counts fastest first unless told otherwise, and on a node with two
// kinds of GPU the shard would get a different card than placement
// gave it — one another shard may hold.
"CUDA_DEVICE_ORDER": "PCI_BUS_ID",
}
if len(s.HCAs) > 0 {