A replica missing a shard is not ready

Restarting the control plane over a running two-node replica adopted half of
it. Every shard it still had was healthy, so it was reported Ready — with one
shard of two and no endpoint at all, because the missing one was rank 0. The
router was handed a replica nothing could route to.

The cause is that a replica had no idea how many shards it should have. It
could tell "every shard I have is ready" and nothing else, which is the same
sentence whether or not half of it is gone. It now carries the count from its
profile, set at placement and again at adoption, and a replica short of shards
fails rather than pretending.

Onboarding also wrote no engine paths, so a node onboarded by the script could
not find llama-server and failed every shard placed on it with "executable
file not found". The script now takes them, and says so when none are given —
a node with no engine paths can run the sim engine and nothing else.

The cluster now runs both at once: two-node simulated replicas on each fabric,
and real llama.cpp inference on the CPU, which is a better test bed than
either alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 12:43:51 +02:00
1 parent c97b671ef9
commit 7b4acd9284
5 files changed
+133 -1

No files matched your search

+24
View File
@@ -31,6 +31,9 @@ usage: $0 --node ID --addr HOST --server URL [options]
a multi-node one.
--mgmt-addr ADDR address engines are reachable on (default: --addr)
--mgmt-if NAME interface NCCL bootstraps over (default eth0)
--llamacpp-bin P path to llama-server
--vllm-bin P path to the vllm entry point
--sglang-python P python interpreter of the SGLang virtualenv
--nvidia-smi PATH where nvidia-smi lives, when it is not on PATH. Containerized
driver installs routinely put it somewhere else.
--ib-root DIR the sysfs InfiniBand directory (default /sys/class/infiniband)
@@ -43,6 +46,7 @@ USAGE
NODE= ADDR= SERVER= MGMT= MGMT_IF=eth0
NVIDIA_SMI= IB_ROOT=/sys/class/infiniband
LLAMACPP_BIN= VLLM_BIN= SGLANG_PYTHON=
PKI=/etc/llmgw/pki BINARY= SSH_USER=ilker
FABRIC_NAME= FABRIC_HCAS="mlx5_0,mlx5_1"
@@ -54,6 +58,9 @@ while [ $# -gt 0 ]; do
--fabric) FABRIC_NAME="${2%%:*}"; [ "${2#*:}" != "$2" ] && FABRIC_HCAS="${2#*:}"; shift 2;;
--mgmt-addr) MGMT="$2"; shift 2;;
--mgmt-if) MGMT_IF="$2"; shift 2;;
--llamacpp-bin) LLAMACPP_BIN="$2"; shift 2;;
--vllm-bin) VLLM_BIN="$2"; shift 2;;
--sglang-python) SGLANG_PYTHON="$2"; shift 2;;
--nvidia-smi) NVIDIA_SMI="$2"; shift 2;;
--ib-root) IB_ROOT="$2"; shift 2;;
--pki) PKI="$2"; shift 2;;
@@ -140,6 +147,19 @@ discovery:
prefer_nvml: false
skip_gpus: false
# Where the engines live. Empty means "on PATH", which is rarely true: a node
# that cannot find its engine registers healthy and then fails every shard
# placed on it with "executable file not found".
engines:
llamacpp:
bin: "$LLAMACPP_BIN"
parallel: 4
vllm:
bin: "$VLLM_BIN"
launch_mode: ray
sglang:
python: "$SGLANG_PYTHON"
supervisor:
work_dir: /var/lib/llmgw-agent
log_lines: 512
@@ -179,6 +199,10 @@ $SSH 'set -e
sudo systemctl restart llmgw-agent'
echo
if [ -z "$LLAMACPP_BIN$VLLM_BIN$SGLANG_PYTHON" ]; then
say "no engine paths given; this node can only run the 'sim' engine until"
say "one is installed and the configuration points at it"
fi
echo "$NODE onboarded. It should appear in 'llmgw ctl nodes' within a few seconds."
echo "If it does not, the control plane's log says why — a certificate whose"
echo "common name does not match the claimed node id is refused by design."