diff --git a/README.md b/README.md index d1be6041..17213e49 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,9 @@ A single-binary LLM gateway for clusters with multiple InfiniBand fabrics, NVLink domains and heterogeneous GPUs. -It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one -OpenAI-compatible endpoint. Design document: [llmgw-plan.md](llmgw-plan.md). +It serves LLMs running on your own GPU and CPU servers (vLLM, SGLang, +llama.cpp, TensorRT-LLM) behind one OpenAI- and Anthropic-compatible +endpoint, and answers from your own documents. Design document: [llmgw-plan.md](llmgw-plan.md). ## Getting started @@ -37,6 +38,9 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). | v0.9.0 | — | Managing people and the tenancy tree | released | | v0.10.0 | — | InfiniBand fabrics and NVLink, doctor, node add, engines installed by the nodes | released | | v0.11.0 | — | High availability from setup, the setup and console screens, usage sums and per-tenant alerts | released | +| v0.12.0 | — | The cluster's settings in the store, in force when changed; the firewall switch; wipe | released | +| v0.13.0 | — | The whole system stopped and started from one command; Qwen3Guard judges, on a GPU and CPUs together | released | +| v0.14.0 | — | Knowledge bases: documents from people's own machines, kept apart per tenant, answered from | released | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | As of v0.6.0 a cluster runs unattended: agents report their hardware, the @@ -44,38 +48,60 @@ control plane places replicas — including ones larger than a single machine models load when someone asks for them and stop when nobody does, and keys, budgets and rate limits decide who may use any of it. -Since v0.11.0, not yet released (see [CHANGELOG.md](CHANGELOG.md), -"Unreleased"): +Since v0.11.0 (each release's detail is in [CHANGELOG.md](CHANGELOG.md)): -- **Security hardening** — signed releases only, a token on the admin API, - engines not run as root, TLS for the agent port, etcd and Patroni, a - firewall for llmgw's internal ports, encrypted backups, bounded - connections, sign-in throttling, a Content-Security-Policy. -- **Guardrails** — policies that check requests and answers (PII, secrets, - prompt injection, deny patterns, a judge model such as Llama Guard), in - observe or enforce mode, attached to any part of the tenancy tree: +- **Knowledge bases** (v0.14.0). People put a company's documents in from + their own machines with `llmgw-kb`, a separate download for Windows, + macOS and Linux that reads PDF, Office, OpenDocument, mail and many more + formats and sends text and vectors, never the files. Collections are kept + apart per tenant, down to row-level security in PostgreSQL. A chat + request, or a part of the tree by default, is answered from them with + its sources: [docs/knowledge.md](docs/knowledge.md). +- **One command for the whole system** (v0.13.0): `llmgw stop`, `start` and + `restart`. Qwen3Guard is the guardrails' judge, on a GPU and on CPUs + together. +- **Settings that apply when changed** (v0.12.0). The cluster's + configuration lives in the store, the same on every control plane and in + force within seconds (`llmgw ctl config`, the console's Settings). The + firewall can be switched on and off for the whole cluster. +- **Security hardening** (v0.11.2): signed releases only, a token on the + admin API, engines not run as root, TLS for the agent port, etcd and + Patroni, a firewall for llmgw's internal ports, encrypted backups, + bounded connections, sign-in throttling, a Content-Security-Policy. +- **Guardrails** (v0.11.2). Policies check requests and answers for PII, + secrets, prompt injection, deny patterns, and with a judge model; they + observe or enforce, and attach to any part of the tenancy tree: [docs/guardrails.md](docs/guardrails.md). -- **Administration** — an audit log of every change (`ctl audit`), staff - roles (viewer, operator, billing, admin), single sign-on through OpenID - Connect and provisioning through SCIM 2.0, key lifetime limits and - rotation with overlap (`keys ttl`, `keys rotate`), usage export as CSV - (`ctl usage export`). -- **Notifications** — webhooks, signed or Slack-shaped, for a model parked, - a node lost, a budget spent and new weights published (`notify.webhooks`, - `ctl notify test`), and Alertmanager receiver templates. -- **Traffic and models** — request tags, canaries of new weights with - automatic rollback, quality suites gating updates, a response cache, - kept and replayed requests, batches, Anthropic's Messages API, more of - OpenAI's (structured output, images, rerank, audio, moderations), - scaling by the clock, SLO-aware admission, TensorRT-LLM, MIG, models - imported from Ollama, routing by what each vLLM replica's cache holds, - LoRA adapters loaded on demand, other llmgw clusters as providers, and - placement that knows racks, PDUs and power caps. -- **Operations** — `server restore` and off-machine backup copies, secrets - in Vault, forgetting a person and keeping a project's requests in the - cluster, offline installs from a bundle, a Helm chart, tracing from the - control plane to the engines, the admin API in OpenAPI with a Terraform - provider, and the portal in English and Turkish with its docs at /docs. +- **Administration** (v0.11.2): + - an audit log of every change (`ctl audit`); + - staff roles (viewer, operator, billing, admin); + - single sign-on through OpenID Connect and provisioning through SCIM + 2.0; + - key lifetime limits, and rotation with overlap (`keys ttl`, `keys + rotate`); + - usage export as CSV (`ctl usage export`). +- **Notifications** (v0.11.2): webhooks, signed or Slack-shaped, for a + model parked, a node lost, a budget spent and new weights published, and + Alertmanager receiver templates. +- **Traffic and models** (v0.11.2): + - request tags, canaries of new weights with automatic rollback, and + quality suites gating updates; + - a response cache, kept and replayed requests, and batches; + - Anthropic's Messages API, and more of OpenAI's (structured output, + images, rerank, audio, moderations); + - scaling by the clock, and SLO-aware admission; + - TensorRT-LLM, MIG, and models imported from Ollama; + - routing by what each vLLM replica's cache holds, and LoRA adapters + loaded on demand; + - other llmgw clusters as providers, and placement that knows racks, + PDUs and power caps. +- **Operations** (v0.11.2): + - `server restore`, off-machine backup copies, and secrets in Vault; + - forgetting a person, and keeping a project's requests in the cluster; + - offline installs from a bundle, and a Helm chart; + - tracing from the control plane to the engines; + - the admin API in OpenAPI, with a Terraform provider; + - the portal in English and Turkish, with its docs at /docs. Every one of these is in the console as well as on the command line: [docs/console.md](docs/console.md). @@ -124,6 +150,9 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console. official OpenAI and Anthropic SDKs, and llmgw's extensions - [docs/guardrails.md](docs/guardrails.md) — checking requests and answers: policies, detectors, a judge model, and where each applies +- [docs/knowledge.md](docs/knowledge.md) — knowledge bases: collections, + who may read them, `llmgw-kb` on people's machines, and answers from + documents - [docs/choosing-models.md](docs/choosing-models.md) — which models to run, how many of each, and where the weights should live @@ -152,7 +181,7 @@ Every page is also in the binary: `llmgw docs`, or **Manual** in the console. ## Building ```sh -make build # bin/llmgw +make build # bin/llmgw and bin/llmgw-kb make test # unit tests make race # with the race detector ``` @@ -169,6 +198,7 @@ llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs sudo llmgw doctor # on a node: what its GPUs, NVLink and fabric need llmgw ctl audit --since 24h # who changed what llmgw ctl usage export --from 2026-09-01 --to 2026-10-01 --by project,model +llmgw ctl kb list # knowledge-base collections, grants, defaults ``` Bare `llmgw` opens the console, which does all of this from one screen.