llmgw/README.md

llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics, NVLink domains and heterogeneous GPUs.

It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang, llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible endpoint, and answers from your own documents. Design document: llmgw-plan.md.

Getting started

On the machine that will be the control plane, with the release binary installed:

sudo llmgw

The setup screens install one machine or three (high availability), check them first, and leave a model answering. After that the same command opens the console. docs/quick-install.md has the six steps from empty servers to a first request; everything else is below.

Status

Releases track the roadmap phases; see CHANGELOG.md.

Version	Phase	Scope	State
v0.1.0	0	Skeleton and sim-engine	released
v0.2.0	1	Static router (filter/score/pick, streaming proxy)	released
v0.3.0	1.5	Admission control (priority bands)	released
v0.4.0	2	Agent: GPU/IB discovery, engine supervision	released
v0.5.0	3	Control plane: catalog, placement, reconciler, ctl	released
v0.6.0	4	Multi-node placement and fabric awareness	released
v0.7.0	—	Conversations, agent memory, disk management	released
v0.8.0	—	PostgreSQL, leader election, agent failover	released
v0.9.0	—	Managing people and the tenancy tree	released
v0.10.0	—	InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes	released
v0.11.0	—	High availability from setup, the setup and console screens, usage sums and per-tenant alerts	released
v0.12.0	—	The cluster's settings in the store, in force when changed; the firewall switch; wipe	released
v0.13.0	—	The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together	released
v0.14.0	—	Knowledge bases: documents from people's own machines, kept apart per tenant, answered from	released
v1.0.0	5	Tenancy, prefix affinity, benchmarking	planned

As of v0.6.0 a cluster runs unattended: agents report their hardware, the control plane places replicas — including ones larger than a single machine — models load when someone asks for them and stop when nobody does, and keys, budgets and rate limits decide who may use any of it.

Since v0.11.0 (each release's detail is in CHANGELOG.md):

• Knowledge bases (v0.14.0). People put a company's documents in from their own machines with llmgw-kb, a separate download for Windows, macOS and Linux that reads PDF, Office, OpenDocument, mail and many more formats and sends text and vectors, never the files. Collections are kept apart per tenant, down to row-level security in PostgreSQL. A chat request, or a part of the tree by default, is answered from them with its sources: docs/knowledge.md.
• One command for the whole system (v0.13.0): llmgw stop, start and restart. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs together.
• Settings that apply when changed (v0.12.0). The cluster's configuration lives in the store, the same on every control plane and in force within seconds (llmgw ctl config, the console's Settings). The firewall can be switched on and off for the whole cluster.
• Security hardening (v0.11.2): signed releases only, a token on the admin API, engines not run as root, TLS for the agent port, etcd and Patroni, a firewall for llmgw's internal ports, encrypted backups, bounded connections, sign-in throttling, a Content-Security-Policy.
• Guardrails (v0.11.2). Policies check requests and answers for PII, secrets, prompt injection, deny patterns, and with a judge model; they observe or enforce, and attach to any part of the tenancy tree: docs/guardrails.md.
• Administration (v0.11.2):
  • an audit log of every change (ctl audit);
  • staff roles (viewer, operator, billing, admin);
  • single sign-on through OpenID Connect and provisioning through SCIM 2.0;
  • key lifetime limits, and rotation with overlap (keys ttl, keys rotate);
  • usage export as CSV (ctl usage export).
• Notifications (v0.11.2): webhooks, signed or Slack-shaped, for a model parked, a node lost, a budget spent and new weights published, and Alertmanager receiver templates.
• Traffic and models (v0.11.2):
  • request tags, canaries of new weights with automatic rollback, and quality suites gating updates;
  • a response cache, kept and replayed requests, and batches;
  • Anthropic's Messages API, and more of OpenAI's (structured output, images, rerank, audio, moderations);
  • scaling by the clock, and SLO-aware admission;
  • TensorRT-LLM, MIG, and models imported from Ollama;
  • routing by what each vLLM replica's cache holds, and LoRA adapters loaded on demand;
  • other llmgw clusters as providers, and placement that knows racks, PDUs and power caps.
• Operations (v0.11.2):
  • server restore, off-machine backup copies, and secrets in Vault;
  • forgetting a person, and keeping a project's requests in the cluster;
  • offline installs from a bundle, and a Helm chart;
  • tracing from the control plane to the engines;
  • the admin API in OpenAPI, with a Terraform provider;
  • the portal in English and Turkish, with its docs at /docs.

Every one of these is in the console as well as on the command line: docs/console.md.

What is still ahead is in BACKLOG.md. Two things that look like gaps are not: a model falls through to its fallbacks when there is no capacity as well as when a budget is spent, and routing profiles are chosen by the key rather than by a request header on purpose — a header would let any client promote itself into the top priority band.

One thing is worth saying plainly: multi-node replicas are proven on a simulated cluster and on real hardware without a fabric. The InfiniBand, NVLink and NCCL paths — telling fabrics apart by their subnet managers, mapping them, finding NVLink domains, launching across nodes — are written against the formats and failure modes those tools actually have and tested against them, but they have not yet run on a machine that has them. On such a machine, run llmgw doctor --json first and keep the report; docs/fabrics.md says what to look at after that.

Documentation

Every page is also in the binary: llmgw docs, or Manual in the console.

Installing

• docs/quick-install.md — from empty servers to a first request, in six steps: start here
• docs/install.md — the same with every option explained, and running the cluster after: monitoring, notifications, single sign-on, the firewall, upgrades, backups
• docs/node-onboarding.md — adding nodes to a cluster that exists
• docs/kubernetes.md — the control plane on Kubernetes with the Helm chart, and why GPU nodes stay outside it
• docs/fabrics.md — InfiniBand fabrics and NVLink: what llmgw needs of them, checking a node with llmgw doctor, and what to look at when a replica spanning nodes will not run

Running it

• docs/console.md — the console that bare llmgw opens: every view, and the command each of its actions runs
• docs/onboarding.md — accounts, keys, limits, staff roles and single sign-on
• docs/clients.md — using the gateway from code with the official OpenAI and Anthropic SDKs, and llmgw's extensions
• docs/guardrails.md — checking requests and answers: policies, detectors, a judge model, and where each applies
• docs/knowledge.md — knowledge bases: collections, who may read them, llmgw-kb on people's machines, and answers from documents
• docs/choosing-models.md — which models to run, how many of each, and where the weights should live

Reference

• docs/reference.md — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does
• docs/data.md — what llmgw stores, what it does not, and what leaves the machine; written for a security review
• docs/admin-api.md and its OpenAPI document — the admin API, for scripts; terraform-provider-llmgw/ manages models, quotas and guardrails with Terraform or OpenTofu
• deploy/ — a mirror on its own machine, Caddy and nginx in front of the client port, two control planes behind a balancer, and Prometheus, Alertmanager and Grafana

Working on llmgw

• docs/development.md — built from source on one workstation, no systemd and no sudo
• llmgw-plan.md — the design; BACKLOG.md — what is next

Building

make build      # bin/llmgw and bin/llmgw-kb
make test       # unit tests
make race       # with the race detector

Operating a cluster

llmgw ctl nodes                      # capacity, health, what is running where
llmgw ctl models                     # catalog and replica counts
llmgw ctl replicas --model llama     # replicas and their shards
llmgw ctl explain-placement llama    # why a model has not been placed
llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs
sudo llmgw doctor                    # on a node: what its GPUs, NVLink and fabric need
llmgw ctl audit --since 24h          # who changed what
llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
llmgw ctl kb list                    # knowledge-base collections, grants, defaults

Bare llmgw opens the console, which does all of this from one screen.

ctl talks to the admin port (:9090 by default), not the client-facing one. Add --json to any command for machine-readable output.

explain-placement is the one worth knowing about: "unschedulable" on its own says nothing about which of six constraints bit, so it prints every candidate considered and the reason it was rejected.

Running the router

# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8001=llama \
  --endpoint e2=http://127.0.0.1:8002=llama

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'

For anything real use a config file; deploy/examples/server.yaml is commented throughout. Metrics, health and pprof are on the admin port (:9090 by default), deliberately separate from the client-facing one.

Requests carry a priority band (x-llmgw-band). Under load, low bands wait at the gateway while high bands keep flowing, so the cluster degrades from the bottom up instead of everywhere at once. Measured effect: interactive p99 stays within 1.24x of idle while 500 concurrent batch requests run, against 58x with admission off.

Routing profiles select how endpoints are scored: chat favours session affinity, batch favours the shortest queue, long-context favours free KV cache. Pick one per model in the config, or per request with the x-llmgw-profile header.

sim-engine

Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:

• continuous batching (bounded number of concurrent sequences)
• KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
• automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
• per-token latency growing with batch size
• dropping a request and freeing its blocks when the client disconnects
• /metrics using vLLM's metric names

./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'

Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks, --prefill-tps, --decode-tps, --batch-contention and --prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for deterministic runs.

The engine metric contract

For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc

Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.

Module path

gitea.mynodes.xyz/ilker/llmgw. To change it:

make rename NEW=git.example.com/team/llmgw

Language

Code, comments, tests, commit messages and documentation are in English.