Ilker ManapandClaude Opus 5
ea4dd37f80
CI / test (push) Successful in 1m24s
Written for whoever is standing up the cluster with root on some servers and nothing installed. Control plane, certificate authority, first node, first model, first request. It leads with what llmgw does not install — drivers, CUDA, engines, weights — because a node missing any of them registers, reports ready, and fails every replica placed on it, and the reason is three logs away. The useful order is to make a node able to serve before onboarding it, which is the opposite of what the tooling's convenience suggests. The failures it warns about are ones this cluster actually hit: llama-server present and unable to start for want of libgomp1 while the node reported healthy; a Proxmox VM on the default kvm64 CPU, where inference is six to nine times slower and nothing looks broken because every number is simply small; requests dying at exactly sixty seconds on a liveness timeout that is a generation budget in disguise for non-streaming calls. Verified rather than recalled. Every command it prints was checked against the binary, every config key against the struct tags, and its own verification and troubleshooting commands were run against the live cluster. That caught two errors in my draft: the health endpoint is /health and returns an empty body, not /healthz, and the Go version needed is 1.26. The known-gaps section is there so a reader finds them on a Tuesday afternoon rather than at 2am: weights and engines are theirs to distribute, there is one control plane and no HA, and multi-node replicas have never run on real InfiniBand — only against simulated hardware. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
149 lines
5.4 KiB
Markdown
149 lines
5.4 KiB
Markdown
# llmgw
|
|
|
|
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
|
|
heterogeneous GPUs.
|
|
|
|
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
|
|
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
|
|
|
|
## Getting started
|
|
|
|
Starting from nothing — no packages installed, no drivers, no nodes — follow
|
|
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
|
|
serves a model, and says plainly which parts llmgw does not install for you.
|
|
|
|
| Guide | For |
|
|
|---|---|
|
|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
|
|
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
|
|
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
|
|
|
|
## Status
|
|
|
|
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
| Version | Phase | Scope | State |
|
|
|---|---|---|---|
|
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
|
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
|
|
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
|
|
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
|
|
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
|
|
|
As of v0.3.0 the router serves traffic and protects it: a batch workload no
|
|
longer starves interactive requests. Still missing before v1.0.0: authentication,
|
|
rate limits and budgets, and the control plane that replaces the static endpoint
|
|
list.
|
|
|
|
## Building
|
|
|
|
```sh
|
|
make build # bin/llmgw
|
|
make test # unit tests
|
|
make race # with the race detector
|
|
```
|
|
|
|
## Operating a cluster
|
|
|
|
```sh
|
|
llmgw ctl nodes # capacity, health, what is running where
|
|
llmgw ctl models # catalog and replica counts
|
|
llmgw ctl replicas --model llama # replicas and their shards
|
|
llmgw ctl explain-placement llama # why a model has not been placed
|
|
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
|
|
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
|
|
```
|
|
|
|
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
|
|
Add `--json` to any command for machine-readable output.
|
|
|
|
`explain-placement` is the one worth knowing about: "unschedulable" on its own
|
|
says nothing about which of six constraints bit, so it prints every candidate
|
|
considered and the reason it was rejected.
|
|
|
|
## Running the router
|
|
|
|
```sh
|
|
# Two fake engines and a router in front of them.
|
|
./bin/llmgw sim-engine --listen :8001 --model llama &
|
|
./bin/llmgw sim-engine --listen :8002 --model llama &
|
|
./bin/llmgw server \
|
|
--endpoint e1=http://127.0.0.1:8001=llama \
|
|
--endpoint e2=http://127.0.0.1:8002=llama
|
|
|
|
curl -N localhost:8080/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-H 'x-session-id: demo' \
|
|
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
|
```
|
|
|
|
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
|
|
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
|
|
by default), deliberately separate from the client-facing one.
|
|
|
|
Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
|
|
the gateway while high bands keep flowing, so the cluster degrades from the
|
|
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
|
|
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
|
admission off.
|
|
|
|
Routing profiles select how endpoints are scored: `chat` favours session
|
|
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
|
cache. Pick one per model in the config, or per request with the
|
|
`x-llmgw-profile` header.
|
|
|
|
## sim-engine
|
|
|
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
|
router can be developed and tested without GPUs. What it models:
|
|
|
|
- continuous batching (bounded number of concurrent sequences)
|
|
- KV cache block accounting, queueing once blocks run out (head-of-line
|
|
blocking included)
|
|
- automatic prefix caching (chained block hashes plus an LRU) and its effect
|
|
on time to first token
|
|
- per-token latency growing with batch size
|
|
- dropping a request and freeing its blocks when the client disconnects
|
|
- `/metrics` using vLLM's metric names
|
|
|
|
```sh
|
|
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
|
|
|
|
curl -N localhost:8000/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
|
|
```
|
|
|
|
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
|
|
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
|
|
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
|
|
deterministic runs.
|
|
|
|
### The engine metric contract
|
|
|
|
For the router to make routing decisions an engine **must** expose these three
|
|
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
|
|
|
|
```
|
|
vllm:num_requests_running
|
|
vllm:num_requests_waiting
|
|
vllm:kv_cache_usage_perc
|
|
```
|
|
|
|
Adding support for a new engine means satisfying that contract. Metric names
|
|
drift between versions, so the name mapping is configurable.
|
|
|
|
## Module path
|
|
|
|
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
|
|
|
|
```sh
|
|
make rename NEW=git.example.com/team/llmgw
|
|
```
|
|
|
|
## Language
|
|
|
|
Code, comments, tests, commit messages and documentation are in English.
|