Ilker ManapandClaude Opus 5
a2b3f63a0d
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and was the detail behind it — phases 2 and 3, written while they were in progress and never folded in when the release was cut — so it claimed as unreleased work that had been shipped for two days. It is now a section of v0.5.0. The status table said v0.5.0 was planned, which its own release notes disprove. The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the headline and the rest is what a cluster needs before anyone but its operator is let near it. Two things would have stopped the release workflow publishing anything: Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on the instance's default mode, and under the restricted default it is read-only on releases. The job now asks for `releases: write`, which is correct under either default. An upload that is refused now fails the job — without `-f`, curl treats a 403 body as a successful transfer, so the release would end up with some of its files, or none, and a green tick either way. `workflow_dispatch` is offered but the upload assumed a tag. Fired from a branch it would have created a release named "main". README's status paragraph still said authentication, budgets and the control plane were missing. It now says what is actually left, including the part worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on a machine that has any of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
156 lines
5.9 KiB
Markdown
156 lines
5.9 KiB
Markdown
# llmgw
|
|
|
|
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
|
heterogeneous GPUs.
|
|
|
|
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
|
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
|
|
|
## Getting started
|
|
|
|
Starting from nothing — no packages installed, no drivers, no nodes — follow
|
|
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
|
|
serves a model, and says plainly which parts llmgw does not install for you.
|
|
|
|
| Guide | For |
|
|
|---|---|
|
|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
|
|
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
|
|
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
|
|
|
|
## Status
|
|
|
|
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
|
|
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
|
|
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
|
|
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
|
control plane places replicas — including ones larger than a single machine —
|
|
models load when someone asks for them and stop when nobody does, and keys,
|
|
budgets and rate limits decide who may use any of it.
|
|
|
|
Before v1.0.0: routing profiles selectable per request, spillover to a fallback
|
|
pool, tracing, and reference documentation. One thing is worth saying plainly:
|
|
multi-node replicas are proven on a simulated cluster and on real hardware
|
|
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
|
|
formats and failure modes those tools actually have, but they have not yet run
|
|
on a machine that has them.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make build # bin/llmgw
|
|
make test # unit tests
|
|
make race # with the race detector
|
|
```
|
|
|
|
## Operating a cluster
|
|
|
|
```sh
|
|
llmgw ctl nodes # capacity, health, what is running where
|
|
llmgw ctl models # catalog and replica counts
|
|
llmgw ctl replicas --model llama # replicas and their shards
|
|
llmgw ctl explain-placement llama # why a model has not been placed
|
|
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
|
|
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
|
|
```
|
|
|
|
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
|
|
Add `--json` to any command for machine-readable output.
|
|
|
|
`explain-placement` is the one worth knowing about: "unschedulable" on its own
|
|
says nothing about which of six constraints bit, so it prints every candidate
|
|
considered and the reason it was rejected.
|
|
|
|
## Running the router
|
|
|
|
```sh
|
|
# Two fake engines and a router in front of them.
|
|
./bin/llmgw sim-engine --listen :8001 --model llama &
|
|
./bin/llmgw sim-engine --listen :8002 --model llama &
|
|
./bin/llmgw server \
|
|
--endpoint e1=http://127.0.0.1:8001=llama \
|
|
--endpoint e2=http://127.0.0.1:8002=llama
|
|
|
|
curl -N localhost:8080/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-H 'x-session-id: demo' \
|
|
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
|
```
|
|
|
|
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
|
|
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
|
|
by default), deliberately separate from the client-facing one.
|
|
|
|
Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
|
|
the gateway while high bands keep flowing, so the cluster degrades from the
|
|
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
|
|
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
|
admission off.
|
|
|
|
Routing profiles select how endpoints are scored: `chat` favours session
|
|
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
|
cache. Pick one per model in the config, or per request with the
|
|
`x-llmgw-profile` header.
|
|
|
|
## sim-engine
|
|
|
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
|
router can be developed and tested without GPUs. What it models:
|
|
|
|
- continuous batching (bounded number of concurrent sequences)
|
|
- KV cache block accounting, queueing once blocks run out (head-of-line
|
|
blocking included)
|
|
- automatic prefix caching (chained block hashes plus an LRU) and its effect
|
|
on time to first token
|
|
- per-token latency growing with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` using vLLM's metric names
|
|
|
|
```sh
|
|
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
|
|
|
|
curl -N localhost:8000/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
|
|
```
|
|
|
|
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
|
|
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
|
|
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
|
|
deterministic runs.
|
|
|
|
### The engine metric contract
|
|
|
|
For the router to make routing decisions an engine **must** expose these three
|
|
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
|
|
|
|
```
|
|
vllm:num_requests_running
|
|
vllm:num_requests_waiting
|
|
vllm:kv_cache_usage_perc
|
|
```
|
|
|
|
Adding support for a new engine means satisfying that contract. Metric names
|
|
drift between versions, so the name mapping is configurable.
|
|
|
|
## Module path
|
|
|
|
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
|
|
|
|
```sh
|
|
make rename NEW=git.example.com/team/llmgw
|
|
```
|
|
|
|
## Language
|
|
|
|
Code, comments, tests, commit messages and documentation are in English.
|