Files
llmgw/README.md
T
ilkerandClaude Opus 5 28029eb43b
CI / test (push) Has been cancelled
Switch the project language to English
Comments, doc comments, test names, error and log messages, CLI help, README,
Makefile, CI step names and the design document are now all in English.
No behaviour changes; the full suite passes under the race detector.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:15:21 +02:00

81 lines
2.4 KiB
Markdown

# llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
## Status
| Phase | Scope | State |
|---|---|---|
| 0 | Skeleton and sim-engine | done |
| 1 | Static router (filter/score/pick, streaming proxy) | in progress |
| 1.5 | Admission control (priority bands) | todo |
| 2 | Agent: GPU/IB discovery, single-node engines | todo |
| 3 | Control plane: catalog, scheduler, reconciler | todo |
| 4 | Multi-node placement and fabric awareness | todo |
| 5 | Tenancy, prefix affinity, benchmarking | todo |
## Building
```sh
make build # bin/llmgw
make test # unit tests
make race # with the race detector
```
## sim-engine
Simulates the inference-engine behaviours **the router cares about**, so the
router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line
blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect
on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` using vLLM's metric names
```sh
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
```
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
deterministic runs.
### The engine metric contract
For the router to make routing decisions an engine **must** expose these three
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
```
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
```
Adding support for a new engine means satisfying that contract. Metric names
drift between versions, so the name mapping is configurable.
## Module path
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
```sh
make rename NEW=git.example.com/team/llmgw
```
## Language
Code, comments, tests, commit messages and documentation are in English.