llmgw/README.md

# llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
NVLink domains and heterogeneous GPUs.

It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang,
llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible
endpoint, and answers from your own documents. Design document: [llmgw-plan.md](llmgw-plan.md).

## Getting started

On the machine that will be the control plane, with the release binary
installed:

```sh
sudo llmgw
```

The setup screens install one machine or three (high availability), check
them first, and leave a model answering. After that the same command opens
the console. [docs/quick-install.md](docs/quick-install.md) has the six steps
from empty servers to a first request; everything else is below.

## Status

Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).

| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released as part of v0.5.0 |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v0.9.0 | — | Managing people and the tenancy tree | released |
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released (v0.14.1: adapters, the tree over the API, the firewall on every machine; v0.14.2: pictures from ComfyUI; v0.14.3: 3D models as jobs; v0.14.4: any ComfyUI workflow as a job, presets, licences; v0.14.5: picture and 3D models that add themselves; v0.14.6: setup that explains every part and lets it be chosen, the gateway on all three HA machines; v0.14.7: the judge sized for its machine) |
| v1.0.0 | 5 | What phase 5 has left: precise KV routing proven on vLLM, the block cost picker, defragmentation across NVLink domains, prefill/decode disaggregation | planned |

As of v0.6.0 a cluster runs unattended: agents report their hardware, the
control plane places replicas — including ones larger than a single machine —
models load when someone asks for them and stop when nobody does, and keys,
budgets and rate limits decide who may use any of it.

Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):

- **Setup that explains, and high availability that is whole** (v0.14.6).
  `sudo llmgw` lists the parts of an install, each with what it is for and
  what is missing without it, and every one can be turned off or set: the
  starter model and its context, the mirror's directory, the judge's
  categories and what it does when it does not answer, limits, the
  portal's language, pictures and 3D. With three machines each runs the
  control plane, with a floating address on the leader, and any one can
  be lost: on three VMs, cutting the store primary's power lost no
  request. One ComfyUI serves an environment's models, pictures, 3D and
  workflows are MCP tools, and `/v1/messages` gives a model's reasoning as
  thinking blocks.
- **Picture and 3D models that add themselves** (v0.14.5). Where a node's
  card fits flux-schnell or triposr and the catalog makes nothing of that
  kind, the leader adds it on demand, so a new cluster's portal has its
  Images and 3D pages; the cluster page and the console show each preset
  and add it by hand, and `presets.auto` turns it off. Stable Fast 3D's
  colours can be brought to the picture's (`albedo_match`), and every mesh
  carries its whole licence.
- **Any ComfyUI workflow, and models by name** (v0.14.4). `POST
  /v1/workflows/runs` runs a ComfyUI model's workflow as a job, its inputs
  (text, numbers, choices, images, masks, audio, video, files) declared in
  the catalog and listed in `/v1/models`. `llmgw ctl model add triposr` or
  `flux-schnell` adds a tested model with no YAML; a profile may name the
  card it needs instead of a class, and ComfyUI now installs on V100s. A
  licence carries what a machine can match, and a person can leave
  licences out on the profile page so no key of theirs reaches such a
  model. A 3D job waiting for the one card says what holds it and for
  about how long.
- **3D models as jobs** (v0.14.3). `POST /v1/3d/generations` answers with a
  job, asked after until its mesh is ready to download. ComfyUI makes it on
  the cluster's own cards, TripoSR being the tested model. The job says how
  far along it is, the model's licence is written into the GLB, and
  `per_mesh` prices it. People make them at `/3d` on the portal; `llmgw ctl
  3d` shows every owner's jobs. A model profile may also load weights of
  its own, and an admission band may wait longer than the rest, for bulk
  work.
- **Pictures from ComfyUI** (v0.14.2). `POST /v1/images/generations` is made
  by ComfyUI, behind a gate that demands the replica's key, as an engine the
  agents install from its source checked against a digest. `per_image`
  prices it, and people make pictures at `/images` on the portal.
- **Adapters, and the tenancy tree over the API** (v0.14.1). `llmgw ctl
  kb dataset` and `llmgw ctl train` make a LoRA adapter from a collection
  on a node's spare GPU memory, kept on the mirror and seen only by its
  owner. Organizations, teams, projects, keys and memberships are in the
  admin API and the Terraform provider. The agents' settings are edited
  with the cluster's, on the portal too, and the firewall setting reaches
  every machine.
- **Knowledge bases** (v0.14.0). People put a company's documents in from
  their own machines with `llmgw-kb`, a separate download for Windows,
  macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
  formats and sends text and vectors, never the files. Collections are kept
  apart per tenant, down to row-level security in PostgreSQL. A chat
  request, or a part of the tree by default, is answered from them with
  its sources: [docs/knowledge.md](docs/knowledge.md).
- **One command for the whole system** (v0.13.0): `llmgw stop`, `start` and
  `restart`. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs
  together.
- **Settings that apply when changed** (v0.12.0). The cluster's
  configuration lives in the store, the same on every control plane and in
  force within seconds (`llmgw ctl config`, the console's Settings). The
  firewall can be switched on and off for the whole cluster.
- **Security hardening** (v0.11.2): signed releases only, a token on the
  admin API, engines not run as root, TLS for the agent port, etcd and
  Patroni, a firewall for llmgw's internal ports, encrypted backups,
  bounded connections, sign-in throttling, a Content-Security-Policy.
- **Guardrails** (v0.11.2). Policies check requests and answers for PII,
  secrets, prompt injection, deny patterns, and with a judge model; they
  observe or enforce, and attach to any part of the tenancy tree:
  [docs/guardrails.md](docs/guardrails.md).
- **Administration** (v0.11.2):
  - an audit log of every change (`ctl audit`);
  - staff roles (viewer, operator, billing, admin);
  - single sign-on through OpenID Connect and provisioning through SCIM
    2.0;
  - key lifetime limits, and rotation with overlap (`keys ttl`, `keys
    rotate`);
  - usage export as CSV (`ctl usage export`).
- **Notifications** (v0.11.2): webhooks, signed or Slack-shaped, for a
  model parked, a node lost, a budget spent and new weights published, and
  Alertmanager receiver templates.
- **Traffic and models** (v0.11.2):
  - request tags, canaries of new weights with automatic rollback, and
    quality suites gating updates;
  - a response cache, kept and replayed requests, and batches;
  - Anthropic's Messages API, and more of OpenAI's (structured output,
    images, rerank, audio, moderations);
  - scaling by the clock, and SLO-aware admission;
  - TensorRT-LLM, MIG, and models imported from Ollama;
  - routing by what each vLLM replica's cache holds, and LoRA adapters
    loaded on demand;
  - other llmgw clusters as providers, and placement that knows racks,
    PDUs and power caps.
- **Operations** (v0.11.2):
  - `server restore`, off-machine backup copies, and secrets in Vault;
  - forgetting a person, and keeping a project's requests in the cluster;
  - offline installs from a bundle, and a Helm chart;
  - tracing from the control plane to the engines;
  - the admin API in OpenAPI, with a Terraform provider;
  - the portal in English and Turkish, with its docs at /docs.

Every one of these is in the console as well as on the command line:
[docs/console.md](docs/console.md).

What is still ahead is in [BACKLOG.md](BACKLOG.md). Two things that look like
gaps are not: a model falls through to its fallbacks when there is no
capacity as well as when a budget is spent, and routing profiles are chosen
by the key rather than by a request header on purpose — a header would let
any client promote itself into the top priority band.

One thing is worth saying plainly:
multi-node replicas are proven on a simulated cluster and on real hardware
without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart
by their subnet managers, mapping them, finding NVLink domains, launching across
nodes — are written against the formats and failure modes those tools actually
have and tested against them, but they have not yet run on a machine that has
them. On such a machine, run `llmgw doctor --json` first and keep the report;
[docs/fabrics.md](docs/fabrics.md) says what to look at after that.

## Documentation

Every page is also in the binary: `llmgw docs`, or **Manual** in the console.

**Installing**

- [docs/quick-install.md](docs/quick-install.md) — from empty servers to a
  first request, in six steps: start here
- [docs/install.md](docs/install.md) — the same with every option explained,
  and running the cluster after: monitoring, notifications, single sign-on,
  the firewall, upgrades, backups
- [docs/node-onboarding.md](docs/node-onboarding.md) — adding nodes to a
  cluster that exists
- [docs/kubernetes.md](docs/kubernetes.md) — the control plane on
  Kubernetes with the Helm chart, and why GPU nodes stay outside it
- [docs/fabrics.md](docs/fabrics.md) — InfiniBand fabrics and NVLink: what
  llmgw needs of them, checking a node with `llmgw doctor`, and what to look at
  when a replica spanning nodes will not run

**Running it**

- [docs/console.md](docs/console.md) — the console that bare `llmgw` opens:
  every view, and the command each of its actions runs
- [docs/onboarding.md](docs/onboarding.md) — accounts, keys, limits, staff
  roles and single sign-on
- [docs/clients.md](docs/clients.md) — using the gateway from code with the
  official OpenAI and Anthropic SDKs, and llmgw's extensions
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
  policies, detectors, a judge model, and where each applies
- [docs/knowledge.md](docs/knowledge.md) — knowledge bases: collections,
  who may read them, `llmgw-kb` on people's machines, and answers from
  documents
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
  how many of each, and where the weights should live

**Reference**

- [docs/reference.md](docs/reference.md) — every configuration key, generated
  from the structs and checked by a test, so it cannot describe a setting that
  does not exist or miss one that does
- [docs/data.md](docs/data.md) — what llmgw stores, what it does not, and
  what leaves the machine; written for a security review
- [docs/admin-api.md](docs/admin-api.md) and its OpenAPI document — the
  admin API, for scripts; [terraform-provider-llmgw/](terraform-provider-llmgw/)
  manages models, quotas and guardrails with Terraform or OpenTofu
- [deploy/](deploy/) — [a mirror](deploy/mirror/) on its own machine,
  [Caddy and nginx](deploy/proxy/) in front of the client port, [two control
  planes](deploy/ha/) behind a balancer, and [Prometheus, Alertmanager and
  Grafana](deploy/monitoring/)

**Working on llmgw**

- [docs/development.md](docs/development.md) — built from source on one
  workstation, no systemd and no `sudo`
- [llmgw-plan.md](llmgw-plan.md) — the design; [BACKLOG.md](BACKLOG.md) —
  what is next

## Building

```sh
make build      # bin/llmgw and bin/llmgw-kb
make test       # unit tests
make race       # with the race detector
```

## Operating a cluster

```sh
llmgw ctl nodes                      # capacity, health, what is running where
llmgw ctl models                     # catalog and replica counts
llmgw ctl replicas --model llama     # replicas and their shards
llmgw ctl explain-placement llama    # why a model has not been placed
llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
llmgw ctl audit --since 24h          # who changed what
llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
llmgw ctl kb list                    # knowledge-base collections, grants, defaults
```

Bare `llmgw` opens the console, which does all of this from one screen.

`ctl` talks to the admin port (`:9090` by default), not the client-facing one.
Add `--json` to any command for machine-readable output.

`explain-placement` is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.

## What llmgw adds to a request

Measured with `llmgw ctl bench overhead`. It starts an engine that answers
at once and a gateway in front of it, in one process on loopback. It sends
the same load straight to the engine and through the gateway, and reports
what the gateway added. Run on an Apple M5 (10 CPUs, load about 2) on
2026-10-16, 10 s a step, 72,000 requests with none failed:

| Through the gateway | added at p50 | added at p99 |
|---|---|---|
| bare | 0.06–0.24 ms | 0.10–0.33 ms |
| with authentication and usage accounting | 0.05–0.32 ms | 0.09–0.37 ms |
| with the guardrails' pattern checks as well | 0.10–0.42 ms | 0.15–0.49 ms |

The range is across 100 to 2,000 requests a second. The overhead was
*lowest* at 2,000, most likely because a busier processor keeps its cores
awake. The guardrails' judge model is not in these figures: a judgment
takes seconds on CPUs (see [install.md](docs/install.md)) and is a model's
time, not the gateway's.

The command also prints the gateway's CPU per thousand requests. Gateway,
engine and load share one process there, so that figure is rough (0.04 to
0.65 s per thousand in the run above); the latency is the one to quote.
Figures from a VM on a busy host move with the host: run it where you will
deploy, and keep the load it prints beside the result. CI runs a one-second
version on every change (`TestTheGatewaysOverheadIsMeasured`).

## Running the router

```sh
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8001=llama \
  --endpoint e2=http://127.0.0.1:8002=llama

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
```

For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
by default), deliberately separate from the client-facing one.

Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.

Routing profiles select how endpoints are scored: `chat` favours session
affinity, `batch` favours the shortest queue, `long-context` favours free KV
cache. Pick one per model in the config, or per request with the
`x-llmgw-profile` header.

## sim-engine

Simulates the inference-engine behaviours **the router cares about**, so the
router can be developed and tested without GPUs. What it models:

- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line
  blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect
  on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
- `/metrics` using vLLM's metric names

```sh
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
```

Tune load and latency behaviour with `--max-num-seqs`, `--num-gpu-blocks`,
`--prefill-tps`, `--decode-tps`, `--batch-contention` and
`--prefix-cache-blocks`. For chaos testing use `--fail-rate`, and `--seed` for
deterministic runs.

### The engine metric contract

For the router to make routing decisions an engine **must** expose these three
metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

```
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
```

Adding support for a new engine means satisfying that contract. Metric names
drift between versions, so the name mapping is configurable.

## Module path

`gitea.mynodes.xyz/ilker/llmgw`. To change it:

```sh
make rename NEW=git.example.com/team/llmgw
```

## Language

Code, comments, tests, commit messages and documentation are in English.