It had one verdict for a failed attempt — "This is the engine, not the model. Choosing a bigger model will not help" — and that verdict was wrong, in a way that sends someone off rebuilding llama.cpp when the fix is a one-line catalog change. Measured on the cluster. The engine renders the tool prompt correctly: the <tools> block and the instruction to wrap the call in <tool_call> are both in the prompt, which I confirmed through the engine's own /apply-template. The coder models simply do not use that wrapper — qwen2.5-coder-3b puts the call in a markdown fence, qwen2.5-coder-7b invents <function-call> — so the parser, which looks for <tool_call>, has nothing to find and leaves it in the content. qwen2.5-3b-instruct, same engine, same build, same flags, returns a populated tool_calls array. So it was the model all along, and the advice against trying another one was the most expensive sentence in the output. A failed attempt is now two verdicts, told apart by whether the reply used the wrapper its template asked for. Wrapped and unparsed is genuinely the engine's parser. Unwrapped is the model, and the fix is one trained for the format rather than a bigger one from the same family. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.
Getting started
Starting from nothing — no packages installed, no drivers, no nodes — follow docs/install.md: it builds a control plane, adds a node and serves a model, and says plainly which parts llmgw does not install for you.
| Guide | For |
|---|---|
| docs/install.md | Standing up a cluster from scratch |
| docs/node-onboarding.md | Adding nodes to one that exists |
| docs/onboarding.md | Giving people accounts and keys |
Status
Releases track the roadmap phases; see CHANGELOG.md.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the control plane places replicas — including ones larger than a single machine — models load when someone asks for them and stop when nobody does, and keys, budgets and rate limits decide who may use any of it.
Before v1.0.0: routing profiles selectable per request, spillover to a fallback pool, tracing, and reference documentation. One thing is worth saying plainly: multi-node replicas are proven on a simulated cluster and on real hardware without a fabric. The InfiniBand, NVML and NCCL paths are written against the formats and failure modes those tools actually have, but they have not yet run on a machine that has them.
Building
make build # bin/llmgw
make test # unit tests
make race # with the race detector
Operating a cluster
llmgw ctl nodes # capacity, health, what is running where
llmgw ctl models # catalog and replica counts
llmgw ctl replicas --model llama # replicas and their shards
llmgw ctl explain-placement llama # why a model has not been placed
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
ctl talks to the admin port (:9090 by default), not the client-facing one.
Add --json to any command for machine-readable output.
explain-placement is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.
Running the router
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
--endpoint e1=http://127.0.0.1:8001=llama \
--endpoint e2=http://127.0.0.1:8002=llama
curl -N localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
For anything real use a config file; deploy/examples/server.yaml
is commented throughout. Metrics, health and pprof are on the admin port (:9090
by default), deliberately separate from the client-facing one.
Requests carry a priority band (x-llmgw-band). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.
Routing profiles select how endpoints are scored: chat favours session
affinity, batch favours the shortest queue, long-context favours free KV
cache. Pick one per model in the config, or per request with the
x-llmgw-profile header.
sim-engine
Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
/metricsusing vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks,
--prefill-tps, --decode-tps, --batch-contention and
--prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for
deterministic runs.
The engine metric contract
For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.
Module path
gitea.mynodes.xyz/ilker/llmgw. To change it:
make rename NEW=git.example.com/team/llmgw
Language
Code, comments, tests, commit messages and documentation are in English.