Everything here works without GPUs and is covered by tests. The control plane connection is not built yet, so llmgw agent itself is still a stub and the phase exit criterion is not met. - discovery: IB from /sys/class/infiniband, parsing the formats sysfs actually uses and telling native InfiniBand from RoCE; GPUs via nvidia-smi, the cgo-free path, with health flags for uncorrected ECC and memory already held by another process. A node without IB is not an error, it is a single-node island. - engines: vLLM and SGLang command and environment construction, kept free of process handling so it stays testable. The NCCL environment is the part the plan marks critical: NCCL_IB_HCA lists only the chosen fabric's adapters, because leaving it open lets NCCL pick an adapter on a fabric where the peer is unreachable, and the job then hangs instead of failing. - validation before launch: no mixed GPU types in a tensor-parallel group, the node/GPU/TP arithmetic has to add up, and fp8 is refused below compute capability 8.9 rather than silently falling back to a slower path. - supervisor: process groups, because engines fork workers that hold GPU memory and outlive a signal sent only to the leader; a bounded log tail, since engine output is the only diagnosis when loading fails; Starting separate from Loading, because a 70B model legitimately takes minutes; and orphan adoption through pid files fingerprinted with the command line, since pids get reused and adopting an unrelated process is worse than leaking the original. - readiness is two questions: /health says the process is up, /v1/models says the weights loaded. Routing on /health alone sends traffic into a loading model. - llmgw agent inventory reports what the scheduler would see on a node. One plan correction: GPU class sizes come from the driver in GiB, which does not always match the marketing name. An L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g, not l40s-48g. Consistency matters more than matching the box, because that number decides whether a model fits. Plan §2 and §5.1 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.
Status
Releases track the roadmap phases; see CHANGELOG.md.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.3.0 the router serves traffic and protects it: a batch workload no longer starves interactive requests. Still missing before v1.0.0: authentication, rate limits and budgets, and the control plane that replaces the static endpoint list.
Building
make build # bin/llmgw
make test # unit tests
make race # with the race detector
Running the router
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
--endpoint e1=http://127.0.0.1:8001=llama \
--endpoint e2=http://127.0.0.1:8002=llama
curl -N localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
For anything real use a config file; deploy/examples/server.yaml
is commented throughout. Metrics, health and pprof are on the admin port (:9090
by default), deliberately separate from the client-facing one.
Requests carry a priority band (x-llmgw-band). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.
Routing profiles select how endpoints are scored: chat favours session
affinity, batch favours the shortest queue, long-context favours free KV
cache. Pick one per model in the config, or per request with the
x-llmgw-profile header.
sim-engine
Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
/metricsusing vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks,
--prefill-tps, --decode-tps, --batch-contention and
--prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for
deterministic runs.
The engine metric contract
For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.
Module path
gitea.mynodes.xyz/ilker/llmgw. To change it:
make rename NEW=git.example.com/team/llmgw
Language
Code, comments, tests, commit messages and documentation are in English.