Files
llmgw/README.md
T
ilkerandClaude Opus 5 a3f71aa5ee Phase 1: the router serves traffic
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.

- proxy: SSE relayed event by event with a flush after each, cancellation
  propagated upstream, and separate deadlines for dial, headers, first token,
  inter-token idle and total. The idle watchdog is what distinguishes a wedged
  engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
  nothing has reached the client; after that, splicing two responses together
  would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
  Validation rejects duplicate endpoint ids and aliases that shadow a model,
  because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
  per-filter drops on a separate admin listener with health and pprof.

Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.

The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:41:13 +02:00

111 lines
3.7 KiB
Markdown

# llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
## Status
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | planned |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.2.0 the router serves traffic. There is still no admission control, so
a batch workload can starve interactive traffic — that is what v0.3.0 is for.
## Building
```sh
make build # bin/llmgw
make test # unit tests
make race # with the race detector
```
## Running the router
```sh
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
--endpoint e1=http://127.0.0.1:8001=llama \
--endpoint e2=http://127.0.0.1:8002=llama
curl -N localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
```
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
by default), deliberately separate from the client-facing one.
Routing profiles select how endpoints are scored: `chat` favours session
affinity, `batch` favours the shortest queue, `long-context` favours free KV
cache. Pick one per model in the config, or per request with the
`x-llmgw-profile` header.
## sim-engine
Simulates the inference-engine behaviours **the router cares about**, so the
router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line
blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect
on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` using vLLM's metric names
```sh
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
```
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
deterministic runs.
### The engine metric contract
For the router to make routing decisions an engine **must** expose these three
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
```
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
```
Adding support for a new engine means satisfying that contract. Metric names
drift between versions, so the name mapping is configurable.
## Module path
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
```sh
make rename NEW=git.example.com/team/llmgw
```
## Language
Code, comments, tests, commit messages and documentation are in English.