llmgw/README.md

llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.

It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.

Status

Phase	Scope	State
0	Skeleton and sim-engine	done
1	Static router (filter/score/pick, streaming proxy)	in progress
1.5	Admission control (priority bands)	todo
2	Agent: GPU/IB discovery, single-node engines	todo
3	Control plane: catalog, scheduler, reconciler	todo
4	Multi-node placement and fabric awareness	todo
5	Tenancy, prefix affinity, benchmarking	todo

Building

make build      # bin/llmgw
make test       # unit tests
make race       # with the race detector

sim-engine

Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:

• continuous batching (bounded number of concurrent sequences)
• KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
• automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
• per-token latency growing with batch size
• dropping a request and freeing its blocks when the client disconnects
• /metrics using vLLM's metric names

./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'

Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks, --prefill-tps, --decode-tps, --batch-contention and --prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for deterministic runs.

The engine metric contract

For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc

Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.

Module path

gitea.mynodes.xyz/ilker/llmgw. To change it:

make rename NEW=git.example.com/team/llmgw

Language

Code, comments, tests, commit messages and documentation are in English.