README: the releases since v0.11.0 as released, and knowledge bases

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-08 17:17:40 +02:00
1 parent 54570066e3
commit 6bc6a67fb0
1 file changed
+63 -33
+63 -33
View File
@@ -3,8 +3,9 @@
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics,
NVLink domains and heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one
OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md).
It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang,
llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible
endpoint, and answers from your own documents. Design document: [llmgw-plan.md](llmgw-plan.md).
## Getting started
@@ -37,6 +38,9 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.9.0 | — | Managing people and the tenancy tree | released |
| v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released |
| v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released |
| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released |
| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released |
| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
@@ -44,38 +48,60 @@ control plane places replicas — including ones larger than a single machine
models load when someone asks for them and stop when nobody does, and keys,
budgets and rate limits decide who may use any of it.
Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md),
"Unreleased"):
Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)):
- **Security hardening** — signed releases only, a token on the admin API,
engines not run as root, TLS for the agent port, etcd and Patroni, a
firewall for llmgw's internal ports, encrypted backups, bounded
connections, sign-in throttling, a Content-Security-Policy.
- **Guardrails** — policies that check requests and answers (PII, secrets,
prompt injection, deny patterns, a judge model such as Llama Guard), in
observe or enforce mode, attached to any part of the tenancy tree:
- **Knowledge bases** (v0.14.0). People put a company's documents in from
their own machines with `llmgw-kb`, a separate download for Windows,
macOS and Linux that reads PDF, Office, OpenDocument, mail and many more
formats and sends text and vectors, never the files. Collections are kept
apart per tenant, down to row-level security in PostgreSQL. A chat
request, or a part of the tree by default, is answered from them with
its sources: [docs/knowledge.md](docs/knowledge.md).
- **One command for the whole system** (v0.13.0): `llmgw stop`, `start` and
`restart`. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs
together.
- **Settings that apply when changed** (v0.12.0). The cluster's
configuration lives in the store, the same on every control plane and in
force within seconds (`llmgw ctl config`, the console's Settings). The
firewall can be switched on and off for the whole cluster.
- **Security hardening** (v0.11.2): signed releases only, a token on the
admin API, engines not run as root, TLS for the agent port, etcd and
Patroni, a firewall for llmgw's internal ports, encrypted backups,
bounded connections, sign-in throttling, a Content-Security-Policy.
- **Guardrails** (v0.11.2). Policies check requests and answers for PII,
secrets, prompt injection, deny patterns, and with a judge model; they
observe or enforce, and attach to any part of the tenancy tree:
[docs/guardrails.md](docs/guardrails.md).
- **Administration** — an audit log of every change (`ctl audit`), staff
roles (viewer, operator, billing, admin), single sign-on through OpenID
Connect and provisioning through SCIM 2.0, key lifetime limits and
rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV
(`ctl usage export`).
- **Notifications** — webhooks, signed or Slack-shaped, for a model parked,
a node lost, a budget spent and new weights published (`notify.webhooks`,
`ctl notify test`), and Alertmanager receiver templates.
- **Traffic and models** — request tags, canaries of new weights with
automatic rollback, quality suites gating updates, a response cache,
kept and replayed requests, batches, Anthropic's Messages API, more of
OpenAI's (structured output, images, rerank, audio, moderations),
scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models
imported from Ollama, routing by what each vLLM replica's cache holds,
LoRA adapters loaded on demand, other llmgw clusters as providers, and
placement that knows racks, PDUs and power caps.
- **Operations** — `server restore` and off-machine backup copies, secrets
in Vault, forgetting a person and keeping a project's requests in the
cluster, offline installs from a bundle, a Helm chart, tracing from the
control plane to the engines, the admin API in OpenAPI with a Terraform
provider, and the portal in English and Turkish with its docs at /docs.
- **Administration** (v0.11.2):
- an audit log of every change (`ctl audit`);
- staff roles (viewer, operator, billing, admin);
- single sign-on through OpenID Connect and provisioning through SCIM
2.0;
- key lifetime limits, and rotation with overlap (`keys ttl`, `keys
rotate`);
- usage export as CSV (`ctl usage export`).
- **Notifications** (v0.11.2): webhooks, signed or Slack-shaped, for a
model parked, a node lost, a budget spent and new weights published, and
Alertmanager receiver templates.
- **Traffic and models** (v0.11.2):
- request tags, canaries of new weights with automatic rollback, and
quality suites gating updates;
- a response cache, kept and replayed requests, and batches;
- Anthropic's Messages API, and more of OpenAI's (structured output,
images, rerank, audio, moderations);
- scaling by the clock, and SLO-aware admission;
- TensorRT-LLM, MIG, and models imported from Ollama;
- routing by what each vLLM replica's cache holds, and LoRA adapters
loaded on demand;
- other llmgw clusters as providers, and placement that knows racks,
PDUs and power caps.
- **Operations** (v0.11.2):
- `server restore`, off-machine backup copies, and secrets in Vault;
- forgetting a person, and keeping a project's requests in the cluster;
- offline installs from a bundle, and a Helm chart;
- tracing from the control plane to the engines;
- the admin API in OpenAPI, with a Terraform provider;
- the portal in English and Turkish, with its docs at /docs.
Every one of these is in the console as well as on the command line:
[docs/console.md](docs/console.md).
@@ -124,6 +150,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
official OpenAI and Anthropic SDKs, and llmgw's extensions
- [docs/guardrails.md](docs/guardrails.md) — checking requests and answers:
policies, detectors, a judge model, and where each applies
- [docs/knowledge.md](docs/knowledge.md) — knowledge bases: collections,
who may read them, `llmgw-kb` on people's machines, and answers from
documents
- [docs/choosing-models.md](docs/choosing-models.md) — which models to run,
how many of each, and where the weights should live
@@ -152,7 +181,7 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console.
## Building
```sh
make build # bin/llmgw
make build # bin/llmgw and bin/llmgw-kb
make test # unit tests
make race # with the race detector
```
@@ -169,6 +198,7 @@ llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need
llmgw ctl audit --since 24h # who changed what
llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model
llmgw ctl kb list # knowledge-base collections, grants, defaults
```
Bare `llmgw` opens the console, which does all of this from one screen.