Files
llmgw/.gitea/workflows/release.yml
T
Ilker ManapandClaude Opus 5 b619546516
CI / test (push) Successful in 1m32s
Stop trusting hardware facts that were never checked
Three places where the system took the infrastructure's word for something
and would have failed silently on real hardware.

A declared fabric adapter that is not on the node. Fabric membership has to
be configuration — OpenSM's default subnet prefix is identical on every
fabric, so a matching prefix proves nothing — but taking it entirely on
trust was a different mistake. A typo, or a card that enumerated
differently after a reboot, produced NCCL_IB_HCA naming an adapter that
does not exist.

An adapter with no ACTIVE port. Physically linked with no subnet manager
sits in INIT: it looks healthy to anything counting adapters, carries no
data, survives a reboot, and looks like a software problem. Discovery had
recorded port state all along and placement only ever asked whether a
fabric had any adapters listed.

Both end the same way — the replica starts, passes its health checks, and
hangs on its first all-reduce, which is the failure the fabric invariant
exists to prevent arriving through a door nobody was watching. The agent
now compares declared against discovered, where both are visible for the
only time, and does not advertise a fabric that cannot carry traffic. The
reason is logged on the machine that knows it, because the operator who
typed the adapter name is the only person who can fix it. The fabric's rate
is the slowest active link, since a collective runs at the speed of its
slowest.

And NVLink topology, which no published binary could see. NVML is its only
source, the bindings need cgo, and CGO_ENABLED=0 is deliberate so the
portable binary runs on an older libc. The cost was silent: tensor groups
fall back to the lowest free indices, which is right on an SXM board with
NVSwitch and wrong on a PCIe box where bridges pair specific cards — a
group straddling two of them runs several times slower and nothing says so.
Releases now also publish an nvml build for those nodes, and
llmgw_gpu_nvlink_peers makes the difference visible: a flat zero across a
fleet means the topology is unknown rather than absent.

A test that declared mlx5_0 and mlx5_1 on a node whose sysfs held neither
had been passing, because nothing compared the two.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:29:15 +02:00

114 lines
4.8 KiB
YAML

# Attaches binaries to a release when a version tag is pushed.
#
# Without this a release is a git tag, and installing llmgw on a node means
# having a Go toolchain and the source. The onboarding script copies a binary
# from wherever it is run, which works and quietly makes whoever runs it the
# build server.
name: Release
on:
push:
tags: ["v*"]
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
# Tags are what the version string is built from, and a shallow clone
# has none of them.
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
- name: Build
run: |
set -euo pipefail
version="${GITHUB_REF_NAME:-$(git describe --tags --always)}"
mkdir -p dist
# CGO_ENABLED=0 is the whole point: the binary has to run on a node
# whose libc is older than the builder's, and llmgw has no C
# dependency — modernc.org/sqlite is pure Go precisely so this stays
# true.
for target in linux/amd64 linux/arm64 darwin/amd64 darwin/arm64; do
os="${target%/*}"; arch="${target#*/}"
name="llmgw-${version}-${os}-${arch}"
echo "building $name"
CGO_ENABLED=0 GOOS="$os" GOARCH="$arch" go build \
-trimpath \
-ldflags "-s -w -X gitea.mynodes.xyz/ilker/llmgw/internal/version.Version=${version}" \
-o "dist/${name}" ./cmd/llmgw
(cd dist && sha256sum "${name}" >> checksums.txt)
done
# An NVML build for nodes with NVLink, which the portable binary
# cannot serve.
#
# NVML is the only source of NVLink topology, its bindings need cgo,
# and cgo is exactly what CGO_ENABLED=0 above exists to avoid. So the
# portable binary silently falls back to taking the lowest free GPU
# indices — right on an SXM box with NVSwitch, and wrong on a PCIe
# box where NVLink bridges pair specific cards: a tensor group
# straddling two bridges runs several times slower and nothing says
# so. Shipping only the portable build meant nobody could have that
# topology without building it themselves.
#
# Dynamically linked against the host's libc by necessity, so it is
# published beside the portable one rather than instead of it.
if command -v gcc >/dev/null; then
nvml_name="llmgw-${version}-linux-amd64-nvml"
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build \
-tags nvml -trimpath \
-ldflags "-s -w -X gitea.mynodes.xyz/ilker/llmgw/internal/version.Version=${version}" \
-o "dist/${nvml_name}" ./cmd/llmgw
(cd dist && sha256sum "${nvml_name}" >> checksums.txt)
else
echo "no C toolchain; skipping the nvml build" >&2
fi
# Proof the artifact runs at all before it is published. It cannot
# test the cross-compiled ones, which is why the native build is
# checked rather than assumed.
./dist/llmgw-${version}-linux-amd64 --version
cat dist/checksums.txt
- name: Attach to the release
env:
TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
if [ -z "${TOKEN:-}" ]; then
echo "No GITEA_TOKEN secret, so nothing can be uploaded."
echo "Add one in the repository's Actions settings with package or"
echo "repository write scope; the binaries are in dist/ meanwhile."
exit 1
fi
tag="${GITHUB_REF_NAME}"
api="${GITHUB_SERVER_URL}/api/v1/repos/${GITHUB_REPOSITORY}"
# The release usually exists already, created by hand with its notes.
id=$(curl -sS -H "Authorization: token $TOKEN" "$api/releases/tags/$tag" \
| sed -n 's/.*"id":\([0-9]*\).*/\1/p' | head -1)
if [ -z "$id" ]; then
echo "creating release $tag"
id=$(curl -sS -X POST -H "Authorization: token $TOKEN" \
-H "Content-Type: application/json" \
-d "{\"tag_name\":\"$tag\",\"name\":\"$tag\"}" \
"$api/releases" | sed -n 's/.*"id":\([0-9]*\).*/\1/p' | head -1)
fi
[ -n "$id" ] || { echo "could not find or create the release" >&2; exit 1; }
for f in dist/*; do
echo "uploading $(basename "$f")"
curl -sS -X POST -H "Authorization: token $TOKEN" \
-F "attachment=@$f" \
"$api/releases/$id/assets?name=$(basename "$f")" >/dev/null
done
echo "attached $(ls dist | wc -l) files to $tag"