Files
llmgw/README.md
T
Ilker ManapandClaude Opus 5 100f10ac17
CI / test (push) Successful in 1m41s
Generate the configuration reference, and fail when it goes stale
There was no reference. Four narrative documents explain how to install and
operate llmgw, and the only way to learn that `lease_ttl` exists was to read
the one that happened to mention it — or the struct.

Written by hand it would have been wrong within a fortnight. Six settings
arrived in the last two weeks alone: store, servers, lease_ttl, the two
retention windows, min_disk_gb. A reference nobody maintains is worse than
none, because it is the same list with the setting you came to look up quietly
missing from it.

So it is derived from the structs, with the field comments as the descriptions
— which means the only way for it to be wrong is for the code to be wrong, and
it makes the comments worth writing twice over. 1163 lines, 24 blocks, 87 keys.

Two tests keep that true after everyone has forgotten this exists. One
regenerates and compares, so adding a setting without regenerating is a
failure rather than an omission — checked by adding one and watching it fail.
The other walks the tree for koanf tags and fails on a configuration file
that is in neither the source list nor the short list of structs an operator
never writes, because a list is a thing to forget.

The generator finds the repository from its own path rather than the working
directory, so `go generate ./...` produces the same document from anywhere.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 07:43:30 +02:00

175 lines
6.9 KiB
Markdown

# llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
## Getting started
Starting from nothing — no packages installed, no drivers, no nodes — follow
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
serves a model, and says plainly which parts llmgw does not install for you.
| Guide | For |
|---|---|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |
## Status
Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
control plane places replicas — including ones larger than a single machine —
models load when someone asks for them and stop when nobody does, and keys,
budgets and rate limits decide who may use any of it.
Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes`
and `ctl replicas` for fleets too large to read a whole table of. Two items
that were on this list are not gaps: a model falls through to its fallbacks
when there is no capacity as well as when a budget is spent, and routing
profiles are chosen by the key rather than by a request header on purpose —
a header would let any client promote itself into the top priority band.
One thing is worth saying plainly:
multi-node replicas are proven on a simulated cluster and on real hardware
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
formats and failure modes those tools actually have, but they have not yet run
on a machine that has them.
## Documentation
- [docs/install.md](docs/install.md) — installing from nothing: control plane,
nodes, engines, weights, first request
- [docs/quickstart.md](docs/quickstart.md) — the short path
- [docs/reference.md](docs/reference.md) — every configuration key, generated
from the structs and checked by a test, so it cannot describe a setting that
does not exist or miss one that does
- [docs/node-onboarding.md](docs/node-onboarding.md),
[docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people
## Building
```sh
make build # bin/llmgw
make test # unit tests
make race # with the race detector
```
## Operating a cluster
```sh
llmgw ctl nodes # capacity, health, what is running where
llmgw ctl models # catalog and replica counts
llmgw ctl replicas --model llama # replicas and their shards
llmgw ctl explain-placement llama # why a model has not been placed
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
```
`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
Add `--json` to any command for machine-readable output.
`explain-placement` is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.
## Running the router
```sh
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
--endpoint e1=http://127.0.0.1:8001=llama \
--endpoint e2=http://127.0.0.1:8002=llama
curl -N localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
```
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
by default), deliberately separate from the client-facing one.
Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.
Routing profiles select how endpoints are scored: `chat` favours session
affinity, `batch` favours the shortest queue, `long-context` favours free KV
cache. Pick one per model in the config, or per request with the
`x-llmgw-profile` header.
## sim-engine
Simulates the inference-engine behaviours **the router cares about**, so the
router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line
blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect
on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` using vLLM's metric names
```sh
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
```
Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
deterministic runs.
### The engine metric contract
For the router to make routing decisions an engine **must** expose these three
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
```
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
```
Adding support for a new engine means satisfying that contract. Metric names
drift between versions, so the name mapping is configurable.
## Module path
`gitea.mynodes.xyz/ilker/llmgw`. To change it:
```sh
make rename NEW=git.example.com/team/llmgw
```
## Language
Code, comments, tests, commit messages and documentation are in English.