Ilker ManapandClaude Opus 5.5
7d97c6b74f
A node ran whatever llama-server somebody had once unpacked on it. Onboarding checked an engine only when told its path, installed none, and a node with nothing on it registered as ready. The one time that went wrong it went wrong invisibly: the binary was there, executable, and could not start for want of libgomp1, while the node reported healthy. The control plane now says which engines every node has, at which version — `engines:` in server.yaml, defaulting to the llama.cpp build this release was checked against — and each agent converges on it. What only the node can know, the node works out: the build for its processor and driver (CUDA with its runtime when an NVIDIA driver is loaded, CPU otherwise), the OS packages that build needs from its own package sources, and whether the result runs. Every archive is checked against the sha256 GitHub publishes for it and unpacked refusing any path or link that leaves the install directory. ldd names a missing library outright; running `--version` proves the rest. Only then is a directory marked installed, and it is checked again every ten minutes, because an unrelated upgrade can remove a library. An engine is reported only once installed and seen to run, or when the agent's configuration names it on purpose — never because something happened to be on PATH. The agent now says when it has probed, so an empty list means "has none" rather than "too old to say": a node rebuilt from nothing is no longer a candidate for every model at the moment it can run none of them. Placement's refusal carries the node's own words, "installing: downloading llama-b10991-bin-ubuntu-x64.tar.gz (16 MB)", and so do `ctl nodes` and the cluster page, along with a warning for a CPU without AVX2 — the VM CPU type that makes everything five times slower and nothing look wrong. Agents also report their OS, architecture, vector extensions and NVIDIA driver. Onboarding installs llmgw itself from the CI release for the node's architecture, checked against the release's checksums, rather than copying whatever binary the operator's machine had. vLLM and SGLang are not installed yet: they are Python environments of several gigabytes, not an archive, and still have to be named in the agent's configuration. Also fixes a race on the weights fingerprint, which two heartbeat loops touch during a failover. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
115 lines
4.8 KiB
YAML
115 lines
4.8 KiB
YAML
# llmgw agent configuration, one per GPU node.
|
|
#
|
|
# The agent dials out to the control plane and keeps one stream open. Engines
|
|
# keep serving if the agent restarts: the router talks to them directly, and
|
|
# orphan adoption picks them up again on the next start.
|
|
|
|
# Everything a node can find out about itself, it does: its id is its hostname,
|
|
# its address is the one on the route to the control plane, llama-server is
|
|
# looked for on PATH and in /opt/llama.cpp, GPUs are there when nvidia-smi is,
|
|
# weights are fetched into /var/lib/llmgw-weights and looked for there, and
|
|
# memory, disk and installed engines are re-read while the agent runs. So a
|
|
# plain CPU node needs only the two blocks below, and the same file on every
|
|
# node. What follows them are overrides, for when the defaults are wrong.
|
|
|
|
server: "http://llmgw-server.internal:7443"
|
|
|
|
# The agent's half of mutual TLS. The certificate's common name must be this
|
|
# node's id — its hostname, unless node_id says otherwise — because the control
|
|
# plane compares the two and refuses the stream when they disagree:
|
|
#
|
|
# llmgw ctl cert issue-client gpu-a-01 --dir /etc/llmgw/pki
|
|
#
|
|
# With this set, `server` above must be an https:// address — an agent holding
|
|
# certificates and dialling http:// is refused at startup rather than connecting
|
|
# in the clear without noticing.
|
|
#
|
|
# tls:
|
|
# ca: /etc/llmgw/ca.pem
|
|
# cert: /etc/llmgw/agent.pem
|
|
# key: /etc/llmgw/agent-key.pem
|
|
|
|
# ---- overrides ---------------------------------------------------------------
|
|
|
|
# node_id: gpu-a-01 # when it is not the hostname
|
|
|
|
# The address engines are reachable on, and the address NCCL bootstraps over.
|
|
# Never a fabric address: NCCL rendezvouses over TCP before moving to RDMA.
|
|
# mgmt_addr: "10.0.0.11"
|
|
# mgmt_interface: eth0
|
|
|
|
# Free-form, for a profile's node_selector. Nothing needs them; do not restate
|
|
# what is measured (memory, GPUs, engines), because a label goes stale.
|
|
# labels:
|
|
# rack: r12
|
|
|
|
heartbeat_interval: 5s
|
|
reconnect_base: 1s
|
|
reconnect_max: 30s
|
|
|
|
# Fabric labelling is explicit and there is no way around it. OpenSM's default
|
|
# subnet prefix is fe80:: on every fabric, so a matching prefix proves nothing
|
|
# and a wrong guess produces a replica that starts, passes health checks and
|
|
# then hangs on its first all-reduce (plan §5.2).
|
|
#
|
|
# A node with no fabrics is its own single-node island: it can host single-node
|
|
# replicas and will never be given a multi-node one.
|
|
fabrics:
|
|
- name: ib-a
|
|
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
|
|
rate_gbps: 200
|
|
|
|
discovery:
|
|
# Worth setting where the driver lives outside the default paths, which
|
|
# containerized driver installs often do.
|
|
nvidia_smi: ""
|
|
ib_root: /sys/class/infiniband
|
|
# NVML is the only source of NVLink topology, which is what keeps a
|
|
# tensor-parallel group off the PCIe path inside a node. It needs a binary
|
|
# built with `-tags nvml`; otherwise this silently falls back to nvidia-smi.
|
|
prefer_nvml: true
|
|
|
|
# Where this node keeps model weights besides /var/lib/llmgw-weights, which is
|
|
# where fetched weights land and is always looked in. Storage differs at
|
|
# every installation and llmgw deliberately does not own it: a Lustre mount, an
|
|
# image exported over the fabric with SRP or NVMe-oF, local NVMe, or a copy
|
|
# made by hand all look identical from here.
|
|
#
|
|
# What it buys is placement preferring a node that already holds the model over
|
|
# one that must first move tens of gigabytes. That matters most where fabrics
|
|
# are isolated from one another, because weights in one fabric are not weights
|
|
# in another and the only path between them is the management network.
|
|
#
|
|
# model_dirs:
|
|
# - /var/lib/llmgw-models
|
|
# - /mnt/lustre/models
|
|
|
|
# The agent installs the engines the control plane names (see `engines:` in
|
|
# server.yaml) under /opt/llmgw/engines: the build for this machine — CUDA when
|
|
# an NVIDIA driver is loaded — checked against its publisher's checksum, with
|
|
# the OS packages it needs, and run before the node takes any model. llama.cpp
|
|
# is installed this way today. Naming a binary below opts that engine out and
|
|
# uses yours instead; vLLM and SGLang still have to be named until the agent
|
|
# installs them too.
|
|
# engines_dir: /opt/llmgw/engines
|
|
|
|
engines:
|
|
vllm:
|
|
bin: /opt/venvs/vllm/bin/vllm
|
|
# Which multi-node mechanism works depends on the vLLM version, and the
|
|
# flags have changed more than once. Check the version you deploy.
|
|
launch_mode: ray # "ray" or "mp"
|
|
ray_bin: /opt/venvs/vllm/bin/ray
|
|
sglang:
|
|
# SGLang is launched as a module, which is its documented entry point.
|
|
python: /opt/venvs/sglang/bin/python
|
|
|
|
supervisor:
|
|
work_dir: /var/lib/llmgw-agent
|
|
# Engine output is the only diagnosis when a model fails to load, so keep a
|
|
# useful tail of it.
|
|
log_lines: 512
|
|
# Engines need time to free GPU memory cleanly before being killed.
|
|
stop_timeout: 60s
|
|
health_interval: 2s
|