Comments, doc comments, test names, error and log messages, CLI help, README, Makefile, CI step names and the design document are now all in English. No behaviour changes; the full suite passes under the race detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
81 lines
2.4 KiB
Markdown
81 lines
2.4 KiB
Markdown
# llmgw
|
|
|
|
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
|
heterogeneous GPUs.
|
|
|
|
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
|
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
|
|
|
## Status
|
|
|
|
| Phase | Scope | State |
|
|
|---|---|---|
|
|
| 0 | Skeleton and sim-engine | done |
|
|
| 1 | Static router (filter/score/pick, streaming proxy) | in progress |
|
|
| 1.5 | Admission control (priority bands) | todo |
|
|
| 2 | Agent: GPU/IB discovery, single-node engines | todo |
|
|
| 3 | Control plane: catalog, scheduler, reconciler | todo |
|
|
| 4 | Multi-node placement and fabric awareness | todo |
|
|
| 5 | Tenancy, prefix affinity, benchmarking | todo |
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make build # bin/llmgw
|
|
make test # unit tests
|
|
make race # with the race detector
|
|
```
|
|
|
|
## sim-engine
|
|
|
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
|
router can be developed and tested without GPUs. What it models:
|
|
|
|
- continuous batching (bounded number of concurrent sequences)
|
|
- KV cache block accounting, queueing once blocks run out (head-of-line
|
|
blocking included)
|
|
- automatic prefix caching (chained block hashes plus an LRU) and its effect
|
|
on time to first token
|
|
- per-token latency growing with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` using vLLM's metric names
|
|
|
|
```sh
|
|
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
|
|
|
|
curl -N localhost:8000/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
|
|
```
|
|
|
|
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
|
|
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
|
|
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
|
|
deterministic runs.
|
|
|
|
### The engine metric contract
|
|
|
|
For the router to make routing decisions an engine **must** expose these three
|
|
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
|
|
|
|
```
|
|
vllm:num_requests_running
|
|
vllm:num_requests_waiting
|
|
vllm:kv_cache_usage_perc
|
|
```
|
|
|
|
Adding support for a new engine means satisfying that contract. Metric names
|
|
drift between versions, so the name mapping is configurable.
|
|
|
|
## Module path
|
|
|
|
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
|
|
|
|
```sh
|
|
make rename NEW=git.example.com/team/llmgw
|
|
```
|
|
|
|
## Language
|
|
|
|
Code, comments, tests, commit messages and documentation are in English.
|