llmgw/README.md

# llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and
heterogeneous GPUs.

It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).

## Getting started

Starting from nothing — no packages installed, no drivers, no nodes — follow
[docs/install.md](docs/install.md): it builds a control plane, adds a node and
serves a model, and says plainly which parts llmgw does not install for you.

| Guide | For |
|---|---|
| [docs/install.md](docs/install.md) | Standing up a cluster from scratch |
| [docs/node-onboarding.md](docs/node-onboarding.md) | Adding nodes to one that exists |
| [docs/onboarding.md](docs/onboarding.md) | Giving people accounts and keys |

## Status

Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).

| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |

As of v0.6.0 a cluster runs unattended: agents report their hardware, the
control plane places replicas — including ones larger than a single machine —
models load when someone asks for them and stop when nobody does, and keys,
budgets and rate limits decide who may use any of it.

Before v1.0.0: tracing, reference documentation, and filtering on `ctl nodes`
and `ctl replicas` for fleets too large to read a whole table of. Two items
that were on this list are not gaps: a model falls through to its fallbacks
when there is no capacity as well as when a budget is spent, and routing
profiles are chosen by the key rather than by a request header on purpose —
a header would let any client promote itself into the top priority band.

One thing is worth saying plainly:
multi-node replicas are proven on a simulated cluster and on real hardware
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
formats and failure modes those tools actually have, but they have not yet run
on a machine that has them.

## Documentation

- [docs/install.md](docs/install.md) — installing from nothing: control plane,
  nodes, engines, weights, first request
- [docs/quickstart.md](docs/quickstart.md) — the short path
- [docs/reference.md](docs/reference.md) — every configuration key, generated
  from the structs and checked by a test, so it cannot describe a setting that
  does not exist or miss one that does
- [deploy/ha/](deploy/ha/) — running two control planes, and the one health
  check that has to be right
- [docs/node-onboarding.md](docs/node-onboarding.md),
  [docs/onboarding.md](docs/onboarding.md) — adding machines, and adding people

## Building

```sh
make build      # bin/llmgw
make test       # unit tests
make race       # with the race detector
```

## Operating a cluster

```sh
llmgw ctl nodes                      # capacity, health, what is running where
llmgw ctl models                     # catalog and replica counts
llmgw ctl replicas --model llama     # replicas and their shards
llmgw ctl explain-placement llama    # why a model has not been placed
llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
```

`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
Add `--json` to any command for machine-readable output.

`explain-placement` is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.

## Running the router

```sh
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8001=llama \
  --endpoint e2=http://127.0.0.1:8002=llama

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
```

For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
by default), deliberately separate from the client-facing one.

Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.

Routing profiles select how endpoints are scored: `chat` favours session
affinity, `batch` favours the shortest queue, `long-context` favours free KV
cache. Pick one per model in the config, or per request with the
`x-llmgw-profile` header.

## sim-engine

Simulates the inference-engine behaviours **the router cares about**, so the
router can be developed and tested without GPUs. What it models:

- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line
  blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect
  on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` using vLLM's metric names

```sh
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
```

Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
deterministic runs.

### The engine metric contract

For the router to make routing decisions an engine **must** expose these three
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

```
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
```

Adding support for a new engine means satisfying that contract. Metric names
drift between versions, so the name mapping is configurable.

## Module path

`gitea.mynodes.xyz/ilker/llmgw`. To change it:

```sh
make rename NEW=git.example.com/team/llmgw
```

## Language

Code, comments, tests, commit messages and documentation are in English.