ilkerandClaude Opus 5
40c884a50a
CI / test (push) Successful in 18s
Versions track the phases in the plan: a phase is released once its exit criterion is met, not when its code lands. That keeps "released" meaning something — v0.1.0 ships sim-engine and the routing libraries, and says plainly that the gateway does not serve traffic yet. scripts/gitea/milestones.py creates the matching Gitea milestones. Idempotent and never destructive, so it can be re-run after the roadmap shifts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
86 lines
2.7 KiB
Markdown
86 lines
2.7 KiB
Markdown
# llmgw
|
|
|
|
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
|
heterogeneous GPUs.
|
|
|
|
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
|
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
|
|
|
## Status
|
|
|
|
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | in progress |
|
|
| v0.3.0 | 1.5 | Admission control (priority bands) | planned |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
|
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
|
|
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
**v0.1.0 does not serve traffic**: `llmgw server` is still a stub. What works
|
|
today is `sim-engine` plus the routing libraries it exists to test.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make build # bin/llmgw
|
|
make test # unit tests
|
|
make race # with the race detector
|
|
```
|
|
|
|
## sim-engine
|
|
|
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
|
router can be developed and tested without GPUs. What it models:
|
|
|
|
- continuous batching (bounded number of concurrent sequences)
|
|
- KV cache block accounting, queueing once blocks run out (head-of-line
|
|
blocking included)
|
|
- automatic prefix caching (chained block hashes plus an LRU) and its effect
|
|
on time to first token
|
|
- per-token latency growing with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` using vLLM's metric names
|
|
|
|
```sh
|
|
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
|
|
|
|
curl -N localhost:8000/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
|
|
```
|
|
|
|
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
|
|
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
|
|
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
|
|
deterministic runs.
|
|
|
|
### The engine metric contract
|
|
|
|
For the router to make routing decisions an engine **must** expose these three
|
|
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
|
|
|
|
```
|
|
vllm:num_requests_running
|
|
vllm:num_requests_waiting
|
|
vllm:kv_cache_usage_perc
|
|
```
|
|
|
|
Adding support for a new engine means satisfying that contract. Metric names
|
|
drift between versions, so the name mapping is configurable.
|
|
|
|
## Module path
|
|
|
|
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
|
|
|
|
```sh
|
|
make rename NEW=git.example.com/team/llmgw
|
|
```
|
|
|
|
## Language
|
|
|
|
Code, comments, tests, commit messages and documentation are in English.
|