# llmgw A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs. It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md). ## Status Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). | Version | Phase | Scope | State | |---|---|---|---| | v0.1.0 | 0 | Skeleton and sim-engine | released | | v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | in progress | | v0.3.0 | 1.5 | Admission control (priority bands) | planned | | v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned | | v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned | | v0.6.0 | 4 | Multi-node placement and fabric awareness | planned | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | **v0.1.0 does not serve traffic**: `llmgw server` is still a stub. What works today is `sim-engine` plus the routing libraries it exists to test. ## Building ```sh make build # bin/llmgw make test # unit tests make race # with the race detector ``` ## sim-engine Simulates the inference-engine behaviours **the router cares about**, so the router can be developed and tested without GPUs. What it models: - continuous batching (bounded number of concurrent sequences) - KV cache block accounting, queueing once blocks run out (head-of-line blocking included) - automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token - per-token latency growing with batch size - dropping a request and freeing its blocks when the client disconnects - `/metrics` using vLLM's metric names ```sh ./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b curl -N localhost:8000/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}' ``` Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`, `--prefill-tps`, `--decode-tps`, `--batch-contention` and `--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for deterministic runs. ### The engine metric contract For the router to make routing decisions an engine **must** expose these three metrics (plan ยง7.6). sim-engine publishes them under vLLM's real names: ``` vllm:num_requests_running vllm:num_requests_waiting vllm:kv_cache_usage_perc ``` Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable. ## Module path `gitea.mynodes.xyz/ilker/llmgw`. To change it: ```sh make rename NEW=git.example.com/team/llmgw ``` ## Language Code, comments, tests, commit messages and documentation are in English.