CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
119 lines
4.1 KiB
Markdown
119 lines
4.1 KiB
Markdown
# llmgw
|
|
|
|
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
|
heterogeneous GPUs.
|
|
|
|
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
|
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
|
|
|
## Status
|
|
|
|
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
|
|
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
|
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
|
|
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
As of v0.3.0 the router serves traffic and protects it: a batch workload no
|
|
longer starves interactive requests. Still missing before v1.0.0: authentication,
|
|
rate limits and budgets, and the control plane that replaces the static endpoint
|
|
list.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make build # bin/llmgw
|
|
make test # unit tests
|
|
make race # with the race detector
|
|
```
|
|
|
|
## Running the router
|
|
|
|
```sh
|
|
# Two fake engines and a router in front of them.
|
|
./bin/llmgw sim-engine --listen :8001 --model llama &
|
|
./bin/llmgw sim-engine --listen :8002 --model llama &
|
|
./bin/llmgw server \
|
|
--endpoint e1=http://127.0.0.1:8001=llama \
|
|
--endpoint e2=http://127.0.0.1:8002=llama
|
|
|
|
curl -N localhost:8080/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-H 'x-session-id: demo' \
|
|
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
|
```
|
|
|
|
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
|
|
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
|
|
by default), deliberately separate from the client-facing one.
|
|
|
|
Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
|
|
the gateway while high bands keep flowing, so the cluster degrades from the
|
|
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
|
|
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
|
admission off.
|
|
|
|
Routing profiles select how endpoints are scored: `chat` favours session
|
|
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
|
cache. Pick one per model in the config, or per request with the
|
|
`x-llmgw-profile` header.
|
|
|
|
## sim-engine
|
|
|
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
|
router can be developed and tested without GPUs. What it models:
|
|
|
|
- continuous batching (bounded number of concurrent sequences)
|
|
- KV cache block accounting, queueing once blocks run out (head-of-line
|
|
blocking included)
|
|
- automatic prefix caching (chained block hashes plus an LRU) and its effect
|
|
on time to first token
|
|
- per-token latency growing with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` using vLLM's metric names
|
|
|
|
```sh
|
|
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
|
|
|
|
curl -N localhost:8000/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
|
|
```
|
|
|
|
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
|
|
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
|
|
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
|
|
deterministic runs.
|
|
|
|
### The engine metric contract
|
|
|
|
For the router to make routing decisions an engine **must** expose these three
|
|
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
|
|
|
|
```
|
|
vllm:num_requests_running
|
|
vllm:num_requests_waiting
|
|
vllm:kv_cache_usage_perc
|
|
```
|
|
|
|
Adding support for a new engine means satisfying that contract. Metric names
|
|
drift between versions, so the name mapping is configurable.
|
|
|
|
## Module path
|
|
|
|
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
|
|
|
|
```sh
|
|
make rename NEW=git.example.com/team/llmgw
|
|
```
|
|
|
|
## Language
|
|
|
|
Code, comments, tests, commit messages and documentation are in English.
|