Merge features-c2: the docs portal and its links in English and Turkish

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:15:25 +02:00
commit e8a62b23b7
201 files changed
+26276 -630

No files matched your search

+160
View File
@@ -0,0 +1,160 @@
# The admin API
Written for whoever automates a cluster rather than typing at it: how to reach
the control plane's management API, what it asks of a request, and where its
description is.
`llmgw ctl` and the console are clients of this API and nothing more, so
anything they do can be done with `curl`, a generated client or the
[Terraform provider](../terraform-provider-llmgw/README.md). It adds and
removes models, scales them, cordons and upgrades nodes, sets capacity
quotas, attaches guardrail policies, and reads the fleet, usage and the audit
log.
## Reaching it
The API is under `/api/v1` on the admin listener, `admin_listen` in
`/etc/llmgw/server.yaml`: `127.0.0.1:9090` unless changed. It is on loopback
on purpose. The same listener serves pprof when it is turned on, and the
server refuses to start with it on any other address unless
`admin_remote: true` — the token would cross the network in clear.
From another machine, tunnel to it rather than moving it:
```sh
ssh -N -L 9090:127.0.0.1:9090 admin@control-plane &
curl -s -H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" http://127.0.0.1:9090/api/v1/summary
```
With `admin_remote: true` and `admin_listen: 10.0.0.5:9090`, call that address
directly — on a network you trust, or behind a TLS proxy that keeps the
`Host` header an address.
## The token
Every path under `/api` needs the admin token as a bearer token:
```
Authorization: Bearer 6f1c…
```
The server writes it the first time it starts, 0600, to
`/var/lib/llmgw/admin-token` under its systemd unit (`admin_token_file`
moves it; a server run by hand keeps it in `secrets/admin-token` beside its
configuration). `llmgw ctl` reads it from there, which is why it runs with
sudo, or from `LLMGW_ADMIN_TOKEN`. To give it to automation:
```sh
sudo cat /var/lib/llmgw/admin-token
```
The token is the whole of the API's authorisation: whoever holds it can do
everything the API does. Keep it where you keep other root credentials.
## What a request must look like
Besides the token, the listener refuses three things before a handler sees
them, as defences against a browser being turned against it:
- **A name in `Host`.** The listener answers to an address or `localhost`,
not a DNS name, so a page on some other site cannot reach it by rebinding
a name to 127.0.0.1. Call it as `http://127.0.0.1:9090`, not
`http://cp.example.com:9090`; behind a proxy, have the proxy send an
address.
- **A cross-site request.** One whose `Origin` is another host, or whose
`Sec-Fetch-Site` says another site sent it.
- **A change that is not JSON.** Every method but `GET` and `HEAD` must carry
`Content-Type: application/json`, even a `DELETE` or a `POST` with no body.
A form post from a page cannot set that header.
A change is recorded in the audit log with who made it, as the request says
in `X-Llmgw-Actor` (letters, digits and `._-@`, at most 64; `unknown` when
absent). `llmgw ctl` sends the name of whoever ran it; automation should send
its own, `terraform` or `ci@deploy`, so `llmgw ctl audit` can tell them apart.
A refusal is a 4xx or 5xx with `{"error": "…"}` saying why. 501 means this
control plane lacks what the endpoint needs — a store, a mirror, guard
policies — and says which.
## Some calls
```sh
H=(-H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" -H "Content-Type: application/json" -H "X-Llmgw-Actor: $USER")
API=http://127.0.0.1:9090/api/v1
# The fleet in one answer, and every model.
curl -s "${H[@]}" $API/summary
curl -s "${H[@]}" $API/models
# Add a model, read back its definition with the defaults filled in, scale it.
curl -s "${H[@]}" -X POST $API/models -d '{
"name": "qwen-small",
"source": "/srv/models/qwen2.5-1.5b-instruct-q4_k_m.gguf",
"profiles": [{"name": "cpu", "engine": "llamacpp", "min_memory_gb": 4, "cpu_cores": 4}],
"scaling": {"min": 1, "max": 2}
}'
curl -s "${H[@]}" $API/models/qwen-small
curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2, "idle_timeout": 1800000000000}'
# A team's capacity quota.
curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'
# Remove the model; a DELETE carries the content type too.
curl -s "${H[@]}" -X DELETE $API/models/qwen-small
```
Durations in bodies are integer nanoseconds, as Go encodes them: half an hour
is `1800000000000`.
## The description
The API is described in OpenAPI 3.1:
- served by every control plane at `GET /api/v1/openapi.json` (with the
token, like the rest), as built into that binary, so it describes the
server you are talking to;
- committed as [admin-api.openapi.json](admin-api.openapi.json) for reading
without a cluster.
```sh
curl -s "${H[@]}" $API/openapi.json > llmgw-admin.json
```
It lists every endpoint, its path and query parameters, and the schema of
each request and answer, with the meaning of each field. Load it into any
OpenAPI tool — Swagger UI, Redoc, an editor — or generate a client from it:
```sh
docker run --rm -v "$PWD:/w" openapitools/openapi-generator-cli generate \
-i /w/llmgw-admin.json -g python -o /w/llmgw-admin-client
```
### How it is kept true
The document is generated, not written. `internal/control/adminapi/routes.go`
holds a table of every endpoint: its method and path, a summary, its query
parameters, and Go values of the types it reads and answers with. The
generator in `internal/docgen/openapi` reads those types by reflection, the
way `encoding/json` encodes them, and takes each field's description from its
comment in the source.
Two tests keep it from drifting. One registers the API on a recording mux and
fails when an endpoint is served and not in the table, or in the table and not
served. The other regenerates the document and fails when it differs from the
committed one. After changing an endpoint or a type it carries:
```sh
go generate ./internal/docgen
```
## What it does not cover
- **Organizations, teams, projects, people and API keys.** These live in the
store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
control plane, or in the web console's `/admin` pages on the client port
behind a session. The admin API reads the tree only where a quota or a
policy names a part of it.
- **The client-facing API** (`/v1/chat/completions` and the rest), which is
OpenAI's and Anthropic's, on `listen`, with API keys.
- **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
pages and `/debug/pprof`, which are not under `/api`.
File diff suppressed because it is too large. Load diff
+548
View File
@@ -0,0 +1,548 @@
# Using llmgw from code
Written for a developer whose program talks to a model through llmgw. There
is no llmgw SDK, and there will not be one: the gateway speaks OpenAI's API
and Anthropic's Messages API, so the official SDKs for both work as they are.
Only two things change — the base URL and the key. Everything llmgw adds on
top is an HTTP header, which every one of those SDKs lets you set.
The examples use `https://llm.example.com` for the gateway; use the address
your operator gave you, or the one on the gateway's own front page. Every
example reads the key from `LLMGW_API_KEY`. `<model-name>` is a model this
gateway serves: `GET /v1/models` lists the ones your key may use, and sending
the placeholder unchanged is answered with that list rather than with an
error.
## Getting a key
Keys are made on the gateway's **Account** page after signing in, or issued
by an operator. A key is shown once and never again — only its hash is kept —
so put it in a password manager straight away. `GET /me` with the key says
what it may do and what it has spent.
## The base URL and the key
| SDK | Base URL | Key |
|---|---|---|
| OpenAI (any language) | `https://llm.example.com/v1` | `Authorization: Bearer …`, as the SDK sends it |
| Anthropic (any language) | `https://llm.example.com` — the SDK adds `/v1/messages` | `x-api-key: …`, as the SDK sends it |
Both SDKs also read their usual environment variables, so a program that
already uses them needs no change at all:
```
export OPENAI_BASE_URL=https://llm.example.com/v1
export OPENAI_API_KEY=$LLMGW_API_KEY
export ANTHROPIC_BASE_URL=https://llm.example.com
export ANTHROPIC_API_KEY=$LLMGW_API_KEY
```
## A first request
### curl
```
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "<model-name>",
"messages": [{"role": "user", "content": "Say hello in Turkish."}]}'
```
### Python
```
pip install openai
```
```python
import os
from openai import OpenAI
client = OpenAI(base_url="https://llm.example.com/v1",
api_key=os.environ["LLMGW_API_KEY"])
reply = client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": "Say hello in Turkish."}],
)
print(reply.choices[0].message.content)
```
### TypeScript
```
npm install openai
```
```typescript
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://llm.example.com/v1",
apiKey: process.env.LLMGW_API_KEY,
});
const reply = await client.chat.completions.create({
model: "<model-name>",
messages: [{ role: "user", content: "Say hello in Turkish." }],
});
console.log(reply.choices[0].message.content);
```
### Go
```
go get github.com/openai/openai-go/v3
```
```go
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/openai/openai-go/v3"
"github.com/openai/openai-go/v3/option"
)
func main() {
client := openai.NewClient(
option.WithBaseURL("https://llm.example.com/v1/"),
option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
)
reply, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
Model: "<model-name>",
Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Say hello in Turkish.")},
})
if err != nil {
log.Fatal(err)
}
fmt.Println(reply.Choices[0].Message.Content)
}
```
The import path carries the SDK's major version; use the one your module
already has.
## Streaming
Set `stream` and read the chunks. llmgw may send a last chunk that carries
only the token counts and no choices, so check for a choice before reading
one — the OpenAI API does the same when usage is asked for.
```python
stream = client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": "Count to ten."}],
stream=True,
)
for chunk in stream:
if chunk.choices and chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)
print()
```
```typescript
const stream = await client.chat.completions.create({
model: "<model-name>",
messages: [{ role: "user", content: "Count to ten." }],
stream: true,
});
for await (const chunk of stream) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
```
```go
stream := client.Chat.Completions.NewStreaming(ctx, openai.ChatCompletionNewParams{
Model: "<model-name>",
Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Count to ten.")},
})
for stream.Next() {
chunk := stream.Current()
if len(chunk.Choices) > 0 {
fmt.Print(chunk.Choices[0].Delta.Content)
}
}
if err := stream.Err(); err != nil {
log.Fatal(err)
}
```
With curl, add `"stream": true` to the body and `-N` so curl prints the
events as they come.
## Tools
Function calling is OpenAI's: describe the tools, run the ones the model
asks for, send the results back. Whether a model calls tools well is the
model's business, not the gateway's; ask your operator which of the served
models are set up for it.
```python
import json
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"description": "The current weather in a city.",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}]
messages = [{"role": "user", "content": "Is it raining in Ankara?"}]
reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)
msg = reply.choices[0].message
if msg.tool_calls:
messages.append(msg)
for call in msg.tool_calls:
args = json.loads(call.function.arguments)
result = {"city": args["city"], "raining": False} # your code here
messages.append({"role": "tool", "tool_call_id": call.id,
"content": json.dumps(result)})
reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)
print(reply.choices[0].message.content)
```
## Embeddings
```python
result = client.embeddings.create(
model="<embedding-model>",
input=["llmgw routes requests", "to GPUs it can reach"],
)
vectors = [d.embedding for d in result.data]
```
```typescript
const result = await client.embeddings.create({
model: "<embedding-model>",
input: ["llmgw routes requests", "to GPUs it can reach"],
});
const vectors = result.data.map((d) => d.embedding);
```
An embedding model is a model of its own; a chat model does not answer
`/v1/embeddings`.
## The Responses API
`/v1/responses` is served too, for clients that speak nothing else (Codex
among them). llmgw translates it to chat completions, so it works with every
model the gateway serves; what has no equivalent there — OpenAI's hosted
tools, reasoning summaries — is ignored rather than refused.
```python
response = client.responses.create(
model="<model-name>",
instructions="Answer in one sentence.",
input="Why is the sky blue?",
)
print(response.output_text)
```
Streamed, the events are the Responses API's own:
```python
stream = client.responses.create(model="<model-name>", input="Count to ten.", stream=True)
for event in stream:
if event.type == "response.output_text.delta":
print(event.delta, end="", flush=True)
```
## Batches
For work that need not finish now: OpenAI's Files and Batch APIs. A batch runs
at the lowest priority, so it uses what interactive traffic leaves free, and
each line is counted against the key that submitted it.
```python
import json
with open("requests.jsonl", "w") as f:
for i, text in enumerate(["first document", "second document"]):
f.write(json.dumps({
"custom_id": f"doc-{i}",
"method": "POST",
"url": "/v1/chat/completions",
"body": {"model": "<model-name>",
"messages": [{"role": "user", "content": "Summarise: " + text}]},
}) + "\n")
batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=batch_file.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
# Later:
batch = client.batches.retrieve(batch.id)
if batch.status == "completed":
for line in client.files.content(batch.output_file_id).text.splitlines():
result = json.loads(line)
print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
```
## The Anthropic SDK
`/v1/messages` is Anthropic's Messages API, translated to chat completions and
back, streamed or not, with tools and images. It serves the gateway's own
models — `model` is a model this gateway serves, not a Claude model.
```
pip install anthropic
```
```python
import os
import anthropic
client = anthropic.Anthropic(base_url="https://llm.example.com",
api_key=os.environ["LLMGW_API_KEY"])
message = client.messages.create(
model="<model-name>",
max_tokens=1024,
messages=[{"role": "user", "content": "Say hello in Turkish."}],
)
print(message.content[0].text)
with client.messages.stream(
model="<model-name>",
max_tokens=1024,
messages=[{"role": "user", "content": "Count to ten."}],
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True)
```
```typescript
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
baseURL: "https://llm.example.com",
apiKey: process.env.LLMGW_API_KEY,
});
const message = await client.messages.create({
model: "<model-name>",
max_tokens: 1024,
messages: [{ role: "user", content: "Say hello in Turkish." }],
});
for (const block of message.content) {
if (block.type === "text") console.log(block.text);
}
```
```go
client := anthropic.NewClient(
option.WithBaseURL("https://llm.example.com/"),
option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
)
message, err := client.Messages.New(ctx, anthropic.MessageNewParams{
Model: "<model-name>",
MaxTokens: 1024,
Messages: []anthropic.MessageParam{
anthropic.NewUserMessage(anthropic.NewTextBlock("Say hello in Turkish.")),
},
})
if err != nil {
log.Fatal(err)
}
for _, block := range message.Content {
if text, ok := block.AsAny().(anthropic.TextBlock); ok {
fmt.Println(text.Text)
}
}
```
(`github.com/anthropics/anthropic-sdk-go` and its `option` package.)
With curl:
```
curl https://llm.example.com/v1/messages \
-H "x-api-key: $LLMGW_API_KEY" \
-H "anthropic-version: 2023-06-01" \
-H "Content-Type: application/json" \
-d '{"model": "<model-name>", "max_tokens": 1024,
"messages": [{"role": "user", "content": "Say hello in Turkish."}]}'
```
## What llmgw adds
Each is a header, so a request stays exactly what the SDK would send anywhere
else. Set one for every request on the client, or for one request:
| | Every request | One request |
|---|---|---|
| Python (both SDKs) | `OpenAI(..., default_headers={...})` | `create(..., extra_headers={...})` |
| TypeScript (both SDKs) | `new OpenAI({ ..., defaultHeaders: {...} })` | `create({...}, { headers: {...} })` |
| Go (both SDKs) | `openai.NewClient(..., option.WithHeader(k, v))` | `New(ctx, params, option.WithHeader(k, v))` |
| curl | | `-H "k: v"` |
### Stored conversations: `x-llmgw-thread`
Send the same thread id with each request and only the new turn; llmgw puts
the stored conversation in front of it. The conversation continues on
whichever model serves the next turn, even a different one. It must be
turned on for your project by an operator (`llmgw keys threads`); a request
that names a thread on a gateway that keeps none is refused rather than
answered without its history.
```python
import uuid
thread = str(uuid.uuid4()) # anyone in your project who knows it reads it
for turn in ["My name is Deniz.", "What is my name?"]:
reply = client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": turn}],
extra_headers={"x-llmgw-thread": thread},
)
print(reply.choices[0].message.content)
```
A thread belongs to the project, not to the key, so use an id nobody can
guess: at most 128 letters, digits and `- _ . :`. When the conversation no
longer fits the model, the oldest turns are left out and the reply carries
`x-llmgw-thread-trimmed` with how many. `DELETE /v1/threads/{id}` forgets one:
```
curl -X DELETE https://llm.example.com/v1/threads/$THREAD \
-H "Authorization: Bearer $LLMGW_API_KEY"
```
### Sessions: `x-session-id`
Requests with the same session id go to the same replica while it is
healthy, so a conversation that resends its history finds it already in that
replica's cache and answers sooner. Without the header, the body's `user`
field is used. Nothing is stored; it is only a routing hint.
```python
client = OpenAI(base_url="https://llm.example.com/v1", api_key=os.environ["LLMGW_API_KEY"],
default_headers={"x-session-id": "chat-7f3a"})
```
### Tags and metadata
Requests can be labelled so the usage they record can be told apart — by
feature, by customer, by job. Send OpenAI's own `metadata`, string keys to
string values, or the same as the `x-llmgw-tags` header for a client that
cannot set `metadata`; the `user` field is recorded too. Which tags also
become metric labels is the operator's choice (`router.tag_labels`).
```python
reply = client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": "Classify this ticket."}],
metadata={"feature": "triage", "customer": "acme"},
user="user-1234",
)
```
```
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "x-llmgw-tags: feature=triage,customer=acme" \
-H "Content-Type: application/json" \
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "Classify this ticket."}]}'
```
### Canaries: `x-llmgw-variant`
While an operator tries a model's newer weights on a share of its traffic, a
session stays on one side. `x-llmgw-variant: canary` sends a request to the
canary deliberately — to try it before it is promoted, or to compare the two.
```python
reply = client.chat.completions.create(
model="<model-name>",
messages=[{"role": "user", "content": "hello"}],
extra_headers={"x-llmgw-variant": "canary"},
)
```
### Memory notes: `/v1/memory`
Notes an agent keeps for itself across conversations: a key, some text, and
optional tags, private to your project and to the agent named in
`x-llmgw-agent` (`default` when it is not sent). An operator turns them on for
a project (`llmgw keys memory`). They are plain REST, so any HTTP client
will do:
```
H="Authorization: Bearer $LLMGW_API_KEY"
curl -X PUT https://llm.example.com/v1/memory/favourite-colour -H "$H" \
-H "x-llmgw-agent: helper" -H "Content-Type: application/json" \
-d '{"text": "The user likes dark green.", "tags": ["preference"]}'
curl "https://llm.example.com/v1/memory?q=green" -H "$H" -H "x-llmgw-agent: helper"
curl "https://llm.example.com/v1/memory?tag=preference" -H "$H" -H "x-llmgw-agent: helper"
curl https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
curl -X DELETE https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
```
A key is up to 200 bytes, a note up to 16 KiB, with at most 16 tags of
lowercase letters, digits, `-` and `_`, and 1000 notes per agent.
### MCP: `/mcp`
The same notes as tools an agent can be given — `remember`, `recall` and
`forget` — from an MCP server at `https://llm.example.com/mcp` (the HTTP
transport, answered as plain JSON). Add it to your agent's MCP servers with
the key as a bearer token and, if you want notes kept apart, `x-llmgw-agent`.
llmgw does not put these tools into requests by itself: an agent that is not
given the server never sees its notes.
```
curl https://llm.example.com/mcp \
-H "Authorization: Bearer $LLMGW_API_KEY" \
-H "Content-Type: application/json" \
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
```
### Headers llmgw sends back
| Header | Says |
|---|---|
| `x-request-id` | The request's id in the gateway's logs and usage records — quote it when asking an operator about a request. |
| `x-llmgw-served-model` | The model that answered, when it was not the one asked for: a spent budget can move a request to a cheaper model. |
| `x-llmgw-queue-ms` | How long the request waited for room before it was sent. |
| `x-llmgw-thread-trimmed` | How many stored turns were left out to fit the model. |
## When something is refused
Errors come back in the shape of the API that was called — OpenAI's
`{"error": {"message": …, "type": …}}` or Anthropic's — so each SDK raises
its usual exception with the gateway's message in it. The ones worth knowing:
| Status | Usually means |
|---|---|
| 400 | Besides a malformed request: a prompt too long for the model, or a guardrail policy refused it (the code `content_policy_violation`). |
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
| 403 | The key may not use that model. |
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
## Priority
With keys on, the key decides the request's priority band and routing
profile, set by an operator; the `x-llmgw-band` and `x-llmgw-profile` headers
are ignored, so a client cannot promote itself. On a gateway without keys
they are read from the request.
+17 -2
View File
@@ -117,6 +117,16 @@ and **UPDATE** when the registry has new weights for it. On a row:
canary is removed) and **Take the canary out…**. A canary that errs or
answers more slowly than the model is taken out by itself. The MODEL
column shows `(10% to canary)` and `(canary of …)`.
- **Ask it a quality suite…** — one of the suites the control plane has (in
`control.eval_dir`, or built in: `arithmetic`, `json-format`), asked
through the router; the dialog counts the answers as they come in, then
shows the score, each case and the model's last runs. Esc stops watching;
the run carries on and is kept. **Its quality runs** lists them, as
`llmgw ctl eval --history` does.
- **Hold its updates to a quality suite…** — the suite and score its new
weights must reach before an update or a promotion moves it to them
(`llmgw ctl model gate`); **none** takes it away. With one set, **Update
to …** and **Promote the canary…** say so.
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
- **Try again**, for a model parked after failing every start.
- **Remove** — its replicas stop and it leaves the catalog. A model from
@@ -138,7 +148,10 @@ team or project is guaranteed and may at most hold, and what it holds now.
**Mirror** is what the model mirror holds, one row for each version of each
model — a file, or the set of files a model of several is: shards, a vision
projector, a config and tokenizer — with how far its download has got and
which models use it. **A** downloads a URL into it; on a row, its files with
which models use it. **A** downloads a URL into it; **I** imports a model
from an Ollama on this machine into a mirror directory on this machine — the
mirror beside the control plane — and shows the model definition to add for
it (a mirror elsewhere: `llmgw mirror import-ollama` there); on a row, its files with
their checksums and where each was fetched from, the source a model
definition names for it, or remove it — with a warning when a model still
uses it.
@@ -195,7 +208,9 @@ row: add a team or project inside it, rename, move, limits for the level and
for each of its members, its capacity quota, and for a project how long
conversations and agent notes are kept, and whether its requests are
(**Keep requests…**: for how long, and what share), and whether they stay in
the cluster (**Requests stay in the cluster…**: never an outside provider). **Longest key lifetime…** caps how
the cluster (**Requests stay in the cluster…**: never an outside provider),
and whether the same request is answered again from memory (**Cache
answers…**: for how long, and whether only reproducible answers or any). **Longest key lifetime…** caps how
long a key made there may live, and is the expiry of one made without any
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
**Remove** takes away only something empty.
+10
View File
@@ -67,10 +67,20 @@ until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
them can read them, and a project under a `no_store` policy cannot make
them.
**Reused answers** — only for a project that asked (`llmgw keys cache`):
each answer that may be reused, as the client got it, and a hash of the
request it answers, in the gateway process's memory — never in the store,
never on disk — for the project's TTL or until pushed out by newer ones
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
policy. Forgetting a person does not reach into it; the answers age out by
the TTL. The store keeps only the setting, and a usage record's mark that it
was answered from the cache.
## What is not stored
Prompts. Completions. Anything the model was shown or produced, unless the
project turned on conversations, notes or kept requests, or sent a batch.
(Reused answers are held in memory only, as above.)
The chat page in llmgw's own web interface keeps the conversation in a
JavaScript variable in the browser. Refresh it and the conversation is gone,
+400 -3
View File
@@ -252,7 +252,8 @@ knowing before you edit it:
`LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
keys requests presented, is off unless `debug_pprof: true`, and then behind
the token too. To see the dashboard from elsewhere use `/admin` on the
client port — do not move this listener.
client port — do not move this listener. The API itself is described in
[admin-api.md](admin-api.md).
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
budgets mean anything. Without a price every request costs zero and a budget
never binds; without the defaults, a project created without limits is
@@ -383,6 +384,46 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
or from the admin page, or by copying files into the directory by hand — the
mirror is a directory, and anything that puts files in it works.
Files already on the mirror machine go in with `import`, which copies them the
way a download is kept — under a temporary name until whole, the checksum
beside them for nodes to check against, a version never replaced — and checks
that a `.gguf` is one:
```
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
--version 2026-10 --sha256 <the checksum its publisher gave>
```
#### From Ollama
Models you already run in Ollama are GGUFs llama.cpp serves as they are.
On the mirror machine, with Ollama's models readable there:
```
llmgw mirror import-ollama # lists what Ollama has
llmgw mirror import-ollama llama3.2:3b
```
It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
model's GGUF, and a vision model's projector beside it, each checked against
the sha256 Ollama named its file by; a file that does not match is not
mirrored, and nothing of the model is. The version is the tag and the start of
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
`latest` moves in Ollama and a mirror version must not; `--model` and
`--version` choose otherwise. It ends with a model definition to save and
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
with.
Ollama's prompt template is not carried: llama.cpp renders the chat template
inside the GGUF, which for any recent model is the one it was trained on. Its
system prompt and adapters are not carried either, and the import says so.
Models Ollama keeps as separate tensors rather than one GGUF, or as several
weight layers, are refused with a sentence saying so — mirror that model's GGUF
from its publisher instead. When the mirror runs beside the control plane, the
console's Mirror view does the same with **I**.
## 4. Preparing a node
**Do this before onboarding.** A node missing its driver or engine registers
@@ -428,6 +469,71 @@ sudo llmgw doctor
A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.
### MIG, if you want to cut a card into pieces
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
GPU) instances, each with its own memory, its own share of the card's compute
and its own UUID. llmgw places a model on an instance as it would on a card,
which makes MIG the way to run several small models on one large card without
them competing for it.
**llmgw does not create the instances.** How a card is cut is your decision,
made once with `nvidia-smi` on the node; the agent reads what is there when
it starts:
```
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
nvidia-smi -L # the instances, with their MIG- UUIDs
sudo systemctl restart llmgw-agent # so it registers the new devices
```
`-C` matters: a GPU instance with no compute instance inside it is not a
device yet. Turning MIG on takes effect at the card's next reset
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
do not survive a reboot unless something recreates them — NVIDIA's
`mig-parted` or a systemd unit of your own.
**Classes.** A card in MIG mode is reported as its instances instead of as
the whole card, and each instance's class is the card's class and the
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
```yaml
profiles:
- name: mig
engine: vllm
gpu_class: a100-80g-mig-3g.40gb
```
**One instance a replica.** A process can open only one MIG device, and
instances have no NVLink or peer-to-peer access between them, so a replica on
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
MIG class asking for more is refused when the model is added, with that
reason. A model too large for one instance wants a larger profile or a whole
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
instance is offered for a model it holds, and never several together.
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
device 1002 (1000 + 100 × card + instance), and every view shows it as
`0/mig2`. The engine is given the instance by its UUID —
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
build. llama.cpp's Vulkan build cannot see MIG instances at all.
**What is measured.** Each instance's memory is its own; temperature, power
and wear are its card's, shown on each of its instances. Utilisation is not
reported for an instance, because no driver measures it per instance: the
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
published for it rather than a zero.
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
card in MIG mode with none created (nothing can run on it: CUDA will not open
the whole card while MIG is on), of GPU instances with no compute instance,
and of a mode change waiting for a reset. Such a card is also kept out of
placement, with the reason, in `llmgw ctl nodes`.
### InfiniBand, if you have it
```
@@ -458,6 +564,7 @@ Nothing to do: the agent installs them once the node joins, under
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
@@ -492,8 +599,8 @@ engines:
A changed version reaches every node when it next connects. To use an engine
you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.
#### Choosing a node's vLLM build
@@ -544,6 +651,82 @@ which drift between releases:
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
```
#### TensorRT-LLM, which you bring
llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
container; an installer guessing at all of that would fail on most machines it
met, a long way into a large download. Install it on the node yourself — a
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
interpreter in that node's `agent.yaml`:
```yaml
engines:
trtllm:
python: /opt/trtllm/venv/bin/python # python3 inside NVIDIA's container
```
The node then offers the engine `trtllm` whenever that interpreter has the
`tensorrt_llm` package; without the line it offers nothing, whatever is on
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
`--served_model_name`, and publishes none of the queue figures the router
reads. Check a node's against the flags llmgw builds:
```
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
```
A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
downloads a repository itself, or loads one from the mirror:
```yaml
name: llama-3.1-70b-instruct
source: meta-llama/Llama-3.1-70B-Instruct
profiles:
- name: h100x4
engine: trtllm
gpu_class: h100-80g
tp: 4
args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
```
The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
--served_model_name …`, through a few lines of Python of its own that do what
trtllm-serve has no flag for:
- **the key.** trtllm-serve checks none; every path but `/health` is made to
demand the one the gateway gives the replica, so the port on the node is not
a way around the gateway.
- **the metrics.** Its options file turns on `return_perf_metrics` and
`enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
and its queue figures do not move. The router reads
`trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
`trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
is JSON. They are set as the engine iterates, so an engine gone idle shows
its last batch until the next request.
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
are set, after the profile's own `env`.
What it does not do:
- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
under MPI or Slurm from one of them, which the agents, each starting its own
node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
or SGLang for one.
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
TensorRT-LLM builds kernels for.
- **take an options file from a profile.** `--config` and
`--extra_llm_api_options` are llmgw's; change a setting with `--set
PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
refused, like the other engines' equivalents: `--trust_remote_code`,
`--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
`--num_serve_frontends` (both would serve without the key), and the flags the
agent sets.
This engine was written against TensorRT-LLM's documented command line,
server and metrics, and tested with stand-ins for them; it has not yet run on a
datacenter GPU. Run `engine-check` on the first node before placing a model on it.
### Weights
Nothing to do here if you built the mirror in step 3 — the agent fetches what a
@@ -911,6 +1094,118 @@ llmgw ctl model canary qwen-7b --abort # or not
- **Promote** is the update the canary was a trial of: the model's replicas
are replaced one at a time, and `llmgw ctl model rollback` goes back.
### Quality suites, and holding new weights to one
A canary says whether new weights serve as fast and as reliably; it does not
say whether their answers are as good. A republished quantisation or a chat
template that no longer matches can make a model faster and worse at once. A
quality suite is a handful of questions with known answers, asked of a model
and scored:
```
llmgw ctl eval --suites # what the control plane can run
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
llmgw ctl eval --history qwen-7b # the runs kept
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
```
It prints each case's score and why, then the suite's: the cases' scores
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
`--min-score`); it exits non-zero below it, so a script can stop on it.
- **How it is asked:** on the control plane, through the router as the
gateway's own requests, the way a replay is — past admission, quotas,
accounting and the guardrails, so the score is the weights' and not a
tenant's limits'. It reaches the model exactly as a client's request
would, a canary or a model loading on demand included. Answers are asked
at temperature 0 unless a case says otherwise, and a reasoning model's
`<think>` block is not part of its answer.
- **Where suites come from:** `--suite` is a file on the machine running
`ctl`, sent whole, or the name of one in the control plane's
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
models: `arithmetic` (sums and word problems, each scored by the number in
the answer) and `json-format` (answers asked for as JSON of a given shape);
their files are in `deploy/eval/`, to copy and change. A file in the
directory with the same name replaces the built-in one.
- **What is kept:** every run, in the store, with the model, the repository
and revision the answering weights came from (when known), the suite's name
and sha256, when, the score, and each case's answer (up to 2,000
characters), score and reason. A run on a canary is filed under the model
it is a canary of. Runs are not pruned.
A suite is YAML (or JSON lines: one case on each line, and optionally a line
without a prompt that holds the suite's own settings):
```yaml
name: support # the file's name unless given
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
judge: llama-70b # grades the cases scored by a rubric
system: You are a support agent for Acme. # every case's, unless it has one
max_tokens: 256 # every case's; 512 unless given
cases:
- id: refund-window
prompt: How many days do customers have to ask for a refund?
number: 30 # the last number in the answer, ±tolerance
tolerance: 0
- prompt: Which plan includes SSO?
contains: [Enterprise] # every string must appear
ignore_case: true
- prompt: Reply with only the word OK.
exact: OK # the whole answer, white space around it aside
- prompt: What is today's ISO date format?
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
- prompt: Give the order as JSON with "id" and "total".
json_schema: # the answer, read as JSON, satisfies it
type: object
required: [id, total]
properties: { id: {type: string}, total: {type: number, minimum: 0} }
- messages: # a conversation instead of a prompt
- { role: user, content: My card was charged twice. }
judge: Apologises, and says the duplicate charge is refunded within 5 days.
weight: 3 # counts three times; 1 unless given
```
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
still read. The schema supports `type`, `properties`, `required`,
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
is read rather than passed unchecked. A `judge` case asks another model on the
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
got no answer scores 0 and says why.
**Holding updates to a suite.** `model update` and `model promote` take
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
definition added with `ctl model add` — so every update and promotion of it
is held to it, from `ctl`, the console or anywhere:
```
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
llmgw ctl model update qwen-7b # asked the suite first
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
llmgw ctl model gate qwen-7b --off
```
- **Promote** asks the canary the suite before anything moves; below the
score the job fails with the score, and the canary stays, with its share,
until you take it out.
- **Update** runs the new weights first as `MODEL-canary` given none of the
model's requests (hidden from clients like any canary), asks it the suite,
then either promotes it — the model's replicas are replaced one at a time,
as for any update — or takes it out again, leaving the model on its old
weights. That needs room for one more replica while it runs; a model that
already has a canary is promoted or aborted first. Without a suite, an
update swaps the weights in as it always has.
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
kept with the job's id beside it.
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
control plane has, the score as it comes in), **Its quality runs**, and **Hold
its updates to a quality suite…**.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -1090,6 +1385,46 @@ same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.
### Tracing
Metrics say that a model took four minutes to start; a trace says where they
went. Give the control plane and every agent the same OpenTelemetry collector,
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
```yaml
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
tracing:
endpoint: http://otel.internal:4318
sample_ratio: 1 # of requests that arrive without a decision of their own
```
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
neither, nothing is traced and nothing is sent: every span is a no-op, and a
command to a node carries no trace context.
What is traced:
| Trace | Spans |
|---|---|
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
| An engine install | `install engine` on the node, a trace of its own. |
So "why did qwen take four minutes to start" is one trace: find `replica start`
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
time went to the download, the launch or the engine loading.
A node follows the control plane's decision: a start the control plane kept is
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
out is not; the agent's ratio decides only for what arrives with no trace, as
from a control plane that is not tracing. The agent needs its own `tracing`
block either way — without one its spans go nowhere, and the control plane's
trace simply ends at `send start`.
## PostgreSQL, when SQLite is no longer the right shape
SQLite is the default and stays the right answer for most installations: one
@@ -1208,6 +1543,68 @@ llmgw keys requests --project Acme/ml/train --off
requests past their hours, or of a project turned off, are removed hourly.
The console has it as **Keep requests…** on a project.
## Reusing answers
A project that asks the same thing again and again — an evaluation run over
the same prompts, embeddings of documents that do not change, an agent that
retries — can have the same request answered from memory rather than by the
model:
```
llmgw keys cache --project Acme/ml/train --ttl 1h # reproducible answers
llmgw keys cache --project Acme/ml/eval --ttl 24h --any # every answer
llmgw keys cache --project Acme/ml/train # how it stands
llmgw keys cache --project Acme/ml/train --off
```
- **What is the same request:** the same project, the same endpoint, the same
model (by its catalog name, whichever alias was used) and the same body,
with `stream`, `stream_options`, `user`, `metadata`, `store` and
`service_tier` left out and the keys in any order. Keys of another project
never see the answer, whatever they send.
- **Which answers are reused:** a chat or completion asked at `temperature`
0 or with a `seed`, and every embedding, rerank and score, which sample
nothing. A request sampled above temperature 0 with no seed asks for a new
answer, and is asked anew — unless the project says `--any`. Only a whole
200 from the model that was asked for is kept: not an error, not a
fallback model's or a canary's answer, not one a guardrail changed or
stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
- **Streams:** an answer given whole is streamed to a client that asks for a
stream, as one chunk per choice with the usage chunk if it asked for that;
a streamed answer is kept put back together, and served whole to a client
that does not stream. Through `/v1/messages` and `/v1/responses` too.
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
`x-llmgw-cache: no`, is always asked anew. Every answer of a project that
reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
can be reused), and a hit says its `Age` in seconds.
- **Policies still apply:** the limits are checked and the guardrails judge
the request before the cache is looked in, and the output policy judges a
reused answer as it would a new one. A request continuing a stored
conversation, or under a `no_store` policy, is never answered from the
cache nor kept in it.
- **What a hit costs:** nothing. It is recorded as a request — status 200,
the model, the key, the rate limit's count — with no tokens and no cost,
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
budget and no token limit. The answer itself carries the usage the model
reported when it was first given.
- **Where it lives:** in each gateway process's memory, at most
`router.cache.max_bytes` (128 MB) of answers for all projects together,
those used longest ago dropped first; nothing is written to the store, and
a restart begins it again. With two control planes behind a balancer each
keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
applies to what is already kept.
```yaml
router:
cache:
max_bytes: 268435456 # 256 MB
max_entry_bytes: 2097152 # 2 MB
```
`llmgw_cache_requests_total{model,result}` counts hits, misses and
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
console has it as **Cache answers…** on a project.
## Batches
Work that can wait — a night's classification, embeddings for a document
+22
View File
@@ -309,6 +309,28 @@ shown as text. Other schemes are never linked — configuration is not always
written by whoever runs the gateway, and a `javascript:` href on a page served to
every visitor is a short path to a stored XSS.
## The docs portal
The gateway also serves its manual at `/docs`: these pages, rendered, with
[clients.md](clients.md) — how to use the gateway from the official OpenAI
and Anthropic SDKs — first on its index. The landing page and the account
page link to it, so a user's questions about streaming, tools or the Anthropic
SDK have an answer that is not you.
It is public, like the landing page. The client guide is for the person who
has no account yet, and the operator documents describe llmgw in general —
no address, model, node or key of this installation — which is what the
project's repository publishes anyway. A site that would rather not say even
that turns sign-in on:
```yaml
landing:
docs_sign_in: true
```
A visitor is then sent to sign in first. On a gateway with API keys and no
accounts, the portal asks for a key instead, like the API.
## What the user sees
They open the link, press the button, and get the key once. The page shows them
+135 -6
View File
@@ -47,6 +47,8 @@ rather than repeated.
| `max_lines` | `int` | [Batch](#batch) |
| `concurrency` | `int` | [Batch](#batch) |
| `retain_days` | `int` | [Batch](#batch) |
| `max_bytes` | `int64` | [Cache](#cache) |
| `max_entry_bytes` | `int64` | [Cache](#cache) |
| `model` | `string` | [Canary](#canary) |
| `percent` | `int` | [Canary](#canary) |
| `min_requests` | `int` | [Canary](#canary) |
@@ -63,6 +65,7 @@ rather than repeated.
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `timezone` | `string` | [Control](#control) |
| `eval_dir` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
@@ -77,6 +80,9 @@ rather than repeated.
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
| `suite` | `string` | [EvalGate](#evalgate) |
| `min_score` | `float64` | [EvalGate](#evalgate) |
| `judge` | `string` | [EvalGate](#evalgate) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
@@ -90,6 +96,7 @@ rather than repeated.
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `language` | `string` | [Landing](#landing) |
| `docs_sign_in` | `bool` | [Landing](#landing) |
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
@@ -110,6 +117,7 @@ rather than repeated.
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `canary` | `*Canary` | [Model](#model) |
| `eval` | `*EvalGate` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
| `context_length` | `int` | [Model](#model) |
@@ -205,6 +213,7 @@ rather than repeated.
| `tag_labels` | `[]string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `batch` | `Batch` | [Router](#router) |
| `cache` | `Cache` | [Router](#router) |
| `trusted_proxies` | `[]string` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
| `scoring` | `Scoring` | [Router](#router) |
@@ -247,7 +256,7 @@ rather than repeated.
| `debug_pprof` | `bool` | [Server](#server) |
| `agent_insecure` | `bool` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `tracing` | `obs.Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
@@ -265,6 +274,7 @@ rather than repeated.
| `models` | `[]catalog.Model` | [Server](#server) |
| `endpoints` | `[]Endpoint` | [Server](#server) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
| `store` | `string` | [Threads](#threads) |
| `retain_hours` | `int` | [Threads](#threads) |
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
@@ -323,6 +333,7 @@ rather than repeated.
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
@@ -332,6 +343,7 @@ rather than repeated.
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
@@ -571,6 +583,22 @@ every interactive request.
RetainDays is how long a batch's answer and error files are kept;
30 unless set. Uploaded files are kept until deleted.
## Cache
### `max_bytes`
`int64`
MaxBytes bounds every project's answers together; 128 MB unless set.
Past it, the answers used longest ago go first.
### `max_entry_bytes`
`int64`
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
unless set.
## Canary
### `model`
@@ -666,6 +694,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over.
Timezone is where models' scaling schedules are read, an IANA name
like Europe/Istanbul; empty is the control plane's own.
### `eval_dir`
`string`
EvalDir is where quality suites are read from by name, for `llmgw ctl
eval MODEL --suite NAME`, a model's eval gate and the console. The
suites the binary carries (arithmetic, json-format) are found when it
has none of the name; a suite here of the same name replaces one.
### `enabled`
`bool`
@@ -759,6 +796,28 @@ Labels picks nodes carrying all of these.
Engines are laid over the cluster's: a version, or "" for none.
## EvalGate
### `suite`
`string`
Suite is a suite's name in control.eval_dir, or one the binary
carries (arithmetic, json-format).
### `min_score`
`float64`
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
threshold.
### `judge`
`string`
Judge grades the suite's rubric cases, instead of the judge it names.
## External
### `url`
@@ -855,6 +914,17 @@ Language is what the web portal speaks to someone who has not chosen
a language with the switcher at the foot of every page and whose
browser asks for none it speaks: "en" or "tr". Empty is English.
### `docs_sign_in`
`bool`
DocsSignIn serves the docs portal at /docs only to someone signed in.
Off, the portal is public like this page: its client guide is for the
person who has no account yet, and the operator documents describe
llmgw in general — no address, model or key of this installation —
which the project's repository publishes anyway. On, a visitor is sent
to sign in, and without accounts the portal needs an API key.
## LlamaCppConfig
### `bin`
@@ -1033,6 +1103,16 @@ moving to it (`llmgw ctl model canary`).
Takes a [Canary](#canary) block.
### `eval`
`*EvalGate`
Eval is a quality suite new weights must pass before they replace
this model's: `llmgw ctl model update` and `model promote` run it
against them first, and stop below its score.
Takes a [EvalGate](#evalgate) block.
### `routing_profile`
`string`
@@ -1701,6 +1781,16 @@ another. Needs the store.
Takes a [Batch](#batch) block.
### `cache`
`Cache`
Cache bounds the answers kept for the projects that reuse them
(llmgw keys cache). Which projects do, and for how long, is theirs
to set; this is only how much memory it may take.
Takes a [Cache](#cache) block.
### `trusted_proxies`
`[]string`
@@ -2029,10 +2119,11 @@ Takes a [Monitoring](#monitoring) block.
### `tracing`
`Tracing`
`obs.Tracing`
Tracing sends a trace of each inference request to an OpenTelemetry
collector. Off unless an endpoint is set here or in
Tracing sends traces to an OpenTelemetry collector: each inference
request, and on a control plane each placement, replica start and
model job. Off unless an endpoint is set here or in
OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
@@ -2184,6 +2275,21 @@ against sim-engines possible without a control plane.
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## TRTLLMConfig
### `python`
`string`
Python is the interpreter of the environment TensorRT-LLM is installed
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
container. Naming it is what makes this node offer the engine; empty
offers nothing.
The interpreter rather than trtllm-serve, because llmgw starts the
server through a few lines of its own that make it demand the
gateway's key, which trtllm-serve has no option for.
## Threads
### `store`
@@ -2282,8 +2388,10 @@ Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
`float64`
SampleRatio is the share of requests traced, 0 to 1, when the caller
did not decide it with a sampled traceparent of its own. Zero takes 1.
SampleRatio is the share of traces kept, 0 to 1, of those nobody
upstream already decided. Zero takes 1. A request with a sampled
traceparent of its own is kept, and an agent keeps every start the
control plane kept, whatever their own ratio says.
## VLLMConfig
@@ -2590,6 +2698,18 @@ Takes a [engines.Config](#enginesconfig) block.
Takes a [supervisor.Config](#supervisorconfig) block.
### `tracing`
`obs.Tracing`
Tracing sends this node's part of each replica start — the weights
fetched, the engine launched, the wait for it to answer — to an
OpenTelemetry collector, inside the control plane's trace of that
start. Give it the collector server.yaml's tracing names. Off unless
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
## config.Band
### `name`
@@ -2642,6 +2762,15 @@ Takes a [SGLangConfig](#sglangconfig) block.
Takes a [LlamaCppConfig](#llamacppconfig) block.
### `trtllm`
`TRTLLMConfig`
TRTLLM is TensorRT-LLM, which the agent does not install: a node
offers it only when its interpreter is named here.
Takes a [TRTLLMConfig](#trtllmconfig) block.
### `sim`
`SimConfig`