The second half of phase 5. A key now has a cost, a quota and a bill. Rate limits use a sliding window rather than a fixed one. A fixed window lets a caller send its whole minute's allowance in the last second of one window and again in the first second of the next, so the observed rate is twice the configured limit exactly when the system is least able to take it. Checking and charging are one operation, or two concurrent requests both see room and both take it. Token limits are checked against an estimate, because the real count only exists once the request has run; the difference is charged afterwards, so a caller who consistently underestimates is still bounded, one window late. A request with no max_tokens gets a modest assumption rather than zero — assuming zero would let an unbounded request through every token limit. Limits are checked before admission, not after: a request over its quota should be refused rather than queued, where it would occupy a band's capacity to no purpose. All refusals are 429, but a spent budget carries no Retry-After because waiting does not help until the period rolls over, and saying otherwise turns one client into a retry loop. Usage goes to a buffered channel and is written in batches. Record never blocks: if the buffer fills — a database stalled long enough to lose eight thousand records — records are dropped and counted, because losing accounting is bad and turning a stalled database into an outage is worse. A clean shutdown drains what is queued; those are requests that were already served. Failures are logged at error, since silently losing billing data is the kind of thing discovered at the end of a month. Requests are accounted whatever their outcome. A request that used a GPU and then failed still used it, and a key that could spend nothing by failing would be a way around a budget. Budgets are counted in memory and restored from the store at startup, so a restart does not hand every key a fresh allowance. The budget period travels with the reservation rather than being re-derived, because a monthly budget credited to a daily counter is a budget that never binds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.
Status
Releases track the roadmap phases; see CHANGELOG.md.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.3.0 the router serves traffic and protects it: a batch workload no longer starves interactive requests. Still missing before v1.0.0: authentication, rate limits and budgets, and the control plane that replaces the static endpoint list.
Building
make build # bin/llmgw
make test # unit tests
make race # with the race detector
Operating a cluster
llmgw ctl nodes # capacity, health, what is running where
llmgw ctl models # catalog and replica counts
llmgw ctl replicas --model llama # replicas and their shards
llmgw ctl explain-placement llama # why a model has not been placed
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
ctl talks to the admin port (:9090 by default), not the client-facing one.
Add --json to any command for machine-readable output.
explain-placement is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.
Running the router
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
--endpoint e1=http://127.0.0.1:8001=llama \
--endpoint e2=http://127.0.0.1:8002=llama
curl -N localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
For anything real use a config file; deploy/examples/server.yaml
is commented throughout. Metrics, health and pprof are on the admin port (:9090
by default), deliberately separate from the client-facing one.
Requests carry a priority band (x-llmgw-band). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.
Routing profiles select how endpoints are scored: chat favours session
affinity, batch favours the shortest queue, long-context favours free KV
cache. Pick one per model in the config, or per request with the
x-llmgw-profile header.
sim-engine
Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
/metricsusing vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks,
--prefill-tps, --decode-tps, --batch-contention and
--prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for
deterministic runs.
The engine metric contract
For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.
Module path
gitea.mynodes.xyz/ilker/llmgw. To change it:
make rename NEW=git.example.com/team/llmgw
Language
Code, comments, tests, commit messages and documentation are in English.