ilkerandClaude Opus 5
e1eb1d4149
CI / test (push) Failing after 50s
ctl talks to a management API on the admin listener, not the client-facing one. The two have different audiences: one serves inference to whoever holds an API key, the other lets a caller cordon nodes and read the cluster's shape. - nodes surfaces unhealthy GPUs and why they are excluded. That is usually the answer when a node looks idle but nothing will schedule onto it, and it was invisible before. - explain-placement prints every candidate considered and why it was rejected. - replicas shows the shards a replica is made of, with per-shard errors. - cordon, uncordon and drain are one endpoint with different targets, because the difference between cordon and drain is exactly what an operator needs to be sure of before running either. Draining had to be fixed first to make ctl drain worth having. A drained replica was stopped on the next pass, which cuts in-flight requests — the thing draining exists to avoid. Now a node marked Draining has its replicas drained, the model falls below its desired count, a replacement is placed elsewhere, and the original is stopped only once the replacement is serving. A grace period bounds it, because draining the only node that can host a model would otherwise never finish and the node could never be taken out of service. Verified against a two-node cluster of real binaries: draining moved a replica to the surviving node and traffic kept returning 200 throughout. ctl logs is deliberately absent. It needs a request/response pair the agent protocol does not have, and the failure path already ships an engine log tail with every status report — so the case that matters most is covered, and reading a healthy shard's output on demand is not worth a protocol change yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
137 lines
4.9 KiB
Markdown
137 lines
4.9 KiB
Markdown
# llmgw
|
|
|
|
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
|
heterogeneous GPUs.
|
|
|
|
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
|
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
|
|
|
## Status
|
|
|
|
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
|
|
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
|
|
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
|
|
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
As of v0.3.0 the router serves traffic and protects it: a batch workload no
|
|
longer starves interactive requests. Still missing before v1.0.0: authentication,
|
|
rate limits and budgets, and the control plane that replaces the static endpoint
|
|
list.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make build # bin/llmgw
|
|
make test # unit tests
|
|
make race # with the race detector
|
|
```
|
|
|
|
## Operating a cluster
|
|
|
|
```sh
|
|
llmgw ctl nodes # capacity, health, what is running where
|
|
llmgw ctl models # catalog and replica counts
|
|
llmgw ctl replicas --model llama # replicas and their shards
|
|
llmgw ctl explain-placement llama # why a model has not been placed
|
|
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
|
|
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
|
|
```
|
|
|
|
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
|
|
Add `--json` to any command for machine-readable output.
|
|
|
|
`explain-placement` is the one worth knowing about: "unschedulable" on its own
|
|
says nothing about which of six constraints bit, so it prints every candidate
|
|
considered and the reason it was rejected.
|
|
|
|
## Running the router
|
|
|
|
```sh
|
|
# Two fake engines and a router in front of them.
|
|
./bin/llmgw sim-engine --listen :8001 --model llama &
|
|
./bin/llmgw sim-engine --listen :8002 --model llama &
|
|
./bin/llmgw server \
|
|
--endpoint e1=http://127.0.0.1:8001=llama \
|
|
--endpoint e2=http://127.0.0.1:8002=llama
|
|
|
|
curl -N localhost:8080/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-H 'x-session-id: demo' \
|
|
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
|
```
|
|
|
|
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
|
|
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
|
|
by default), deliberately separate from the client-facing one.
|
|
|
|
Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
|
|
the gateway while high bands keep flowing, so the cluster degrades from the
|
|
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
|
|
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
|
admission off.
|
|
|
|
Routing profiles select how endpoints are scored: `chat` favours session
|
|
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
|
cache. Pick one per model in the config, or per request with the
|
|
`x-llmgw-profile` header.
|
|
|
|
## sim-engine
|
|
|
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
|
router can be developed and tested without GPUs. What it models:
|
|
|
|
- continuous batching (bounded number of concurrent sequences)
|
|
- KV cache block accounting, queueing once blocks run out (head-of-line
|
|
blocking included)
|
|
- automatic prefix caching (chained block hashes plus an LRU) and its effect
|
|
on time to first token
|
|
- per-token latency growing with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` using vLLM's metric names
|
|
|
|
```sh
|
|
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
|
|
|
|
curl -N localhost:8000/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
|
|
```
|
|
|
|
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
|
|
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
|
|
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
|
|
deterministic runs.
|
|
|
|
### The engine metric contract
|
|
|
|
For the router to make routing decisions an engine **must** expose these three
|
|
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
|
|
|
|
```
|
|
vllm:num_requests_running
|
|
vllm:num_requests_waiting
|
|
vllm:kv_cache_usage_perc
|
|
```
|
|
|
|
Adding support for a new engine means satisfying that contract. Metric names
|
|
drift between versions, so the name mapping is configurable.
|
|
|
|
## Module path
|
|
|
|
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
|
|
|
|
```sh
|
|
make rename NEW=git.example.com/team/llmgw
|
|
```
|
|
|
|
## Language
|
|
|
|
Code, comments, tests, commit messages and documentation are in English.
|