Merge features-c2: the docs portal and its links in English and Turkish
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
e8a62b23b7
201 files changed
+26276
-630
No files matched your search
@@ -0,0 +1,160 @@
|
||||
# The admin API
|
||||
|
||||
Written for whoever automates a cluster rather than typing at it: how to reach
|
||||
the control plane's management API, what it asks of a request, and where its
|
||||
description is.
|
||||
|
||||
`llmgw ctl` and the console are clients of this API and nothing more, so
|
||||
anything they do can be done with `curl`, a generated client or the
|
||||
[Terraform provider](../terraform-provider-llmgw/README.md). It adds and
|
||||
removes models, scales them, cordons and upgrades nodes, sets capacity
|
||||
quotas, attaches guardrail policies, and reads the fleet, usage and the audit
|
||||
log.
|
||||
|
||||
## Reaching it
|
||||
|
||||
The API is under `/api/v1` on the admin listener, `admin_listen` in
|
||||
`/etc/llmgw/server.yaml`: `127.0.0.1:9090` unless changed. It is on loopback
|
||||
on purpose. The same listener serves pprof when it is turned on, and the
|
||||
server refuses to start with it on any other address unless
|
||||
`admin_remote: true` — the token would cross the network in clear.
|
||||
|
||||
From another machine, tunnel to it rather than moving it:
|
||||
|
||||
```sh
|
||||
ssh -N -L 9090:127.0.0.1:9090 admin@control-plane &
|
||||
curl -s -H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" http://127.0.0.1:9090/api/v1/summary
|
||||
```
|
||||
|
||||
With `admin_remote: true` and `admin_listen: 10.0.0.5:9090`, call that address
|
||||
directly — on a network you trust, or behind a TLS proxy that keeps the
|
||||
`Host` header an address.
|
||||
|
||||
## The token
|
||||
|
||||
Every path under `/api` needs the admin token as a bearer token:
|
||||
|
||||
```
|
||||
Authorization: Bearer 6f1c…
|
||||
```
|
||||
|
||||
The server writes it the first time it starts, 0600, to
|
||||
`/var/lib/llmgw/admin-token` under its systemd unit (`admin_token_file`
|
||||
moves it; a server run by hand keeps it in `secrets/admin-token` beside its
|
||||
configuration). `llmgw ctl` reads it from there, which is why it runs with
|
||||
sudo, or from `LLMGW_ADMIN_TOKEN`. To give it to automation:
|
||||
|
||||
```sh
|
||||
sudo cat /var/lib/llmgw/admin-token
|
||||
```
|
||||
|
||||
The token is the whole of the API's authorisation: whoever holds it can do
|
||||
everything the API does. Keep it where you keep other root credentials.
|
||||
|
||||
## What a request must look like
|
||||
|
||||
Besides the token, the listener refuses three things before a handler sees
|
||||
them, as defences against a browser being turned against it:
|
||||
|
||||
- **A name in `Host`.** The listener answers to an address or `localhost`,
|
||||
not a DNS name, so a page on some other site cannot reach it by rebinding
|
||||
a name to 127.0.0.1. Call it as `http://127.0.0.1:9090`, not
|
||||
`http://cp.example.com:9090`; behind a proxy, have the proxy send an
|
||||
address.
|
||||
- **A cross-site request.** One whose `Origin` is another host, or whose
|
||||
`Sec-Fetch-Site` says another site sent it.
|
||||
- **A change that is not JSON.** Every method but `GET` and `HEAD` must carry
|
||||
`Content-Type: application/json`, even a `DELETE` or a `POST` with no body.
|
||||
A form post from a page cannot set that header.
|
||||
|
||||
A change is recorded in the audit log with who made it, as the request says
|
||||
in `X-Llmgw-Actor` (letters, digits and `._-@`, at most 64; `unknown` when
|
||||
absent). `llmgw ctl` sends the name of whoever ran it; automation should send
|
||||
its own, `terraform` or `ci@deploy`, so `llmgw ctl audit` can tell them apart.
|
||||
|
||||
A refusal is a 4xx or 5xx with `{"error": "…"}` saying why. 501 means this
|
||||
control plane lacks what the endpoint needs — a store, a mirror, guard
|
||||
policies — and says which.
|
||||
|
||||
## Some calls
|
||||
|
||||
```sh
|
||||
H=(-H "Authorization: Bearer $LLMGW_ADMIN_TOKEN" -H "Content-Type: application/json" -H "X-Llmgw-Actor: $USER")
|
||||
API=http://127.0.0.1:9090/api/v1
|
||||
|
||||
# The fleet in one answer, and every model.
|
||||
curl -s "${H[@]}" $API/summary
|
||||
curl -s "${H[@]}" $API/models
|
||||
|
||||
# Add a model, read back its definition with the defaults filled in, scale it.
|
||||
curl -s "${H[@]}" -X POST $API/models -d '{
|
||||
"name": "qwen-small",
|
||||
"source": "/srv/models/qwen2.5-1.5b-instruct-q4_k_m.gguf",
|
||||
"profiles": [{"name": "cpu", "engine": "llamacpp", "min_memory_gb": 4, "cpu_cores": 4}],
|
||||
"scaling": {"min": 1, "max": 2}
|
||||
}'
|
||||
curl -s "${H[@]}" $API/models/qwen-small
|
||||
curl -s "${H[@]}" -X POST $API/models/qwen-small/scale -d '{"min": 0, "max": 2, "idle_timeout": 1800000000000}'
|
||||
|
||||
# A team's capacity quota.
|
||||
curl -s "${H[@]}" -X PUT $API/quotas -d '{"scope": "team:Acme/ml", "limit": {"gpus": 8}}'
|
||||
|
||||
# Remove the model; a DELETE carries the content type too.
|
||||
curl -s "${H[@]}" -X DELETE $API/models/qwen-small
|
||||
```
|
||||
|
||||
Durations in bodies are integer nanoseconds, as Go encodes them: half an hour
|
||||
is `1800000000000`.
|
||||
|
||||
## The description
|
||||
|
||||
The API is described in OpenAPI 3.1:
|
||||
|
||||
- served by every control plane at `GET /api/v1/openapi.json` (with the
|
||||
token, like the rest), as built into that binary, so it describes the
|
||||
server you are talking to;
|
||||
- committed as [admin-api.openapi.json](admin-api.openapi.json) for reading
|
||||
without a cluster.
|
||||
|
||||
```sh
|
||||
curl -s "${H[@]}" $API/openapi.json > llmgw-admin.json
|
||||
```
|
||||
|
||||
It lists every endpoint, its path and query parameters, and the schema of
|
||||
each request and answer, with the meaning of each field. Load it into any
|
||||
OpenAPI tool — Swagger UI, Redoc, an editor — or generate a client from it:
|
||||
|
||||
```sh
|
||||
docker run --rm -v "$PWD:/w" openapitools/openapi-generator-cli generate \
|
||||
-i /w/llmgw-admin.json -g python -o /w/llmgw-admin-client
|
||||
```
|
||||
|
||||
### How it is kept true
|
||||
|
||||
The document is generated, not written. `internal/control/adminapi/routes.go`
|
||||
holds a table of every endpoint: its method and path, a summary, its query
|
||||
parameters, and Go values of the types it reads and answers with. The
|
||||
generator in `internal/docgen/openapi` reads those types by reflection, the
|
||||
way `encoding/json` encodes them, and takes each field's description from its
|
||||
comment in the source.
|
||||
|
||||
Two tests keep it from drifting. One registers the API on a recording mux and
|
||||
fails when an endpoint is served and not in the table, or in the table and not
|
||||
served. The other regenerates the document and fails when it differs from the
|
||||
committed one. After changing an endpoint or a type it carries:
|
||||
|
||||
```sh
|
||||
go generate ./internal/docgen
|
||||
```
|
||||
|
||||
## What it does not cover
|
||||
|
||||
- **Organizations, teams, projects, people and API keys.** These live in the
|
||||
store and are managed with `llmgw org`, `llmgw users` and `llmgw keys` on the
|
||||
control plane, or in the web console's `/admin` pages on the client port
|
||||
behind a session. The admin API reads the tree only where a quota or a
|
||||
policy names a part of it.
|
||||
- **The client-facing API** (`/v1/chat/completions` and the rest), which is
|
||||
OpenAI's and Anthropic's, on `listen`, with API keys.
|
||||
- **The rest of the admin listener**: `/metrics`, `/health`, the dashboard
|
||||
pages and `/debug/pprof`, which are not under `/api`.
|
||||
File diff suppressed because it is too large.
Load diff
+548
@@ -0,0 +1,548 @@
|
||||
# Using llmgw from code
|
||||
|
||||
Written for a developer whose program talks to a model through llmgw. There
|
||||
is no llmgw SDK, and there will not be one: the gateway speaks OpenAI's API
|
||||
and Anthropic's Messages API, so the official SDKs for both work as they are.
|
||||
Only two things change — the base URL and the key. Everything llmgw adds on
|
||||
top is an HTTP header, which every one of those SDKs lets you set.
|
||||
|
||||
The examples use `https://llm.example.com` for the gateway; use the address
|
||||
your operator gave you, or the one on the gateway's own front page. Every
|
||||
example reads the key from `LLMGW_API_KEY`. `<model-name>` is a model this
|
||||
gateway serves: `GET /v1/models` lists the ones your key may use, and sending
|
||||
the placeholder unchanged is answered with that list rather than with an
|
||||
error.
|
||||
|
||||
## Getting a key
|
||||
|
||||
Keys are made on the gateway's **Account** page after signing in, or issued
|
||||
by an operator. A key is shown once and never again — only its hash is kept —
|
||||
so put it in a password manager straight away. `GET /me` with the key says
|
||||
what it may do and what it has spent.
|
||||
|
||||
## The base URL and the key
|
||||
|
||||
| SDK | Base URL | Key |
|
||||
|---|---|---|
|
||||
| OpenAI (any language) | `https://llm.example.com/v1` | `Authorization: Bearer …`, as the SDK sends it |
|
||||
| Anthropic (any language) | `https://llm.example.com` — the SDK adds `/v1/messages` | `x-api-key: …`, as the SDK sends it |
|
||||
|
||||
Both SDKs also read their usual environment variables, so a program that
|
||||
already uses them needs no change at all:
|
||||
|
||||
```
|
||||
export OPENAI_BASE_URL=https://llm.example.com/v1
|
||||
export OPENAI_API_KEY=$LLMGW_API_KEY
|
||||
|
||||
export ANTHROPIC_BASE_URL=https://llm.example.com
|
||||
export ANTHROPIC_API_KEY=$LLMGW_API_KEY
|
||||
```
|
||||
|
||||
## A first request
|
||||
|
||||
### curl
|
||||
|
||||
```
|
||||
curl https://llm.example.com/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "<model-name>",
|
||||
"messages": [{"role": "user", "content": "Say hello in Turkish."}]}'
|
||||
```
|
||||
|
||||
### Python
|
||||
|
||||
```
|
||||
pip install openai
|
||||
```
|
||||
|
||||
```python
|
||||
import os
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="https://llm.example.com/v1",
|
||||
api_key=os.environ["LLMGW_API_KEY"])
|
||||
|
||||
reply = client.chat.completions.create(
|
||||
model="<model-name>",
|
||||
messages=[{"role": "user", "content": "Say hello in Turkish."}],
|
||||
)
|
||||
print(reply.choices[0].message.content)
|
||||
```
|
||||
|
||||
### TypeScript
|
||||
|
||||
```
|
||||
npm install openai
|
||||
```
|
||||
|
||||
```typescript
|
||||
import OpenAI from "openai";
|
||||
|
||||
const client = new OpenAI({
|
||||
baseURL: "https://llm.example.com/v1",
|
||||
apiKey: process.env.LLMGW_API_KEY,
|
||||
});
|
||||
|
||||
const reply = await client.chat.completions.create({
|
||||
model: "<model-name>",
|
||||
messages: [{ role: "user", content: "Say hello in Turkish." }],
|
||||
});
|
||||
console.log(reply.choices[0].message.content);
|
||||
```
|
||||
|
||||
### Go
|
||||
|
||||
```
|
||||
go get github.com/openai/openai-go/v3
|
||||
```
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
|
||||
"github.com/openai/openai-go/v3"
|
||||
"github.com/openai/openai-go/v3/option"
|
||||
)
|
||||
|
||||
func main() {
|
||||
client := openai.NewClient(
|
||||
option.WithBaseURL("https://llm.example.com/v1/"),
|
||||
option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
|
||||
)
|
||||
reply, err := client.Chat.Completions.New(context.Background(), openai.ChatCompletionNewParams{
|
||||
Model: "<model-name>",
|
||||
Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Say hello in Turkish.")},
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
fmt.Println(reply.Choices[0].Message.Content)
|
||||
}
|
||||
```
|
||||
|
||||
The import path carries the SDK's major version; use the one your module
|
||||
already has.
|
||||
|
||||
## Streaming
|
||||
|
||||
Set `stream` and read the chunks. llmgw may send a last chunk that carries
|
||||
only the token counts and no choices, so check for a choice before reading
|
||||
one — the OpenAI API does the same when usage is asked for.
|
||||
|
||||
```python
|
||||
stream = client.chat.completions.create(
|
||||
model="<model-name>",
|
||||
messages=[{"role": "user", "content": "Count to ten."}],
|
||||
stream=True,
|
||||
)
|
||||
for chunk in stream:
|
||||
if chunk.choices and chunk.choices[0].delta.content:
|
||||
print(chunk.choices[0].delta.content, end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
```typescript
|
||||
const stream = await client.chat.completions.create({
|
||||
model: "<model-name>",
|
||||
messages: [{ role: "user", content: "Count to ten." }],
|
||||
stream: true,
|
||||
});
|
||||
for await (const chunk of stream) {
|
||||
process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
|
||||
}
|
||||
```
|
||||
|
||||
```go
|
||||
stream := client.Chat.Completions.NewStreaming(ctx, openai.ChatCompletionNewParams{
|
||||
Model: "<model-name>",
|
||||
Messages: []openai.ChatCompletionMessageParamUnion{openai.UserMessage("Count to ten.")},
|
||||
})
|
||||
for stream.Next() {
|
||||
chunk := stream.Current()
|
||||
if len(chunk.Choices) > 0 {
|
||||
fmt.Print(chunk.Choices[0].Delta.Content)
|
||||
}
|
||||
}
|
||||
if err := stream.Err(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
```
|
||||
|
||||
With curl, add `"stream": true` to the body and `-N` so curl prints the
|
||||
events as they come.
|
||||
|
||||
## Tools
|
||||
|
||||
Function calling is OpenAI's: describe the tools, run the ones the model
|
||||
asks for, send the results back. Whether a model calls tools well is the
|
||||
model's business, not the gateway's; ask your operator which of the served
|
||||
models are set up for it.
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "The current weather in a city.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
|
||||
messages = [{"role": "user", "content": "Is it raining in Ankara?"}]
|
||||
reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)
|
||||
msg = reply.choices[0].message
|
||||
|
||||
if msg.tool_calls:
|
||||
messages.append(msg)
|
||||
for call in msg.tool_calls:
|
||||
args = json.loads(call.function.arguments)
|
||||
result = {"city": args["city"], "raining": False} # your code here
|
||||
messages.append({"role": "tool", "tool_call_id": call.id,
|
||||
"content": json.dumps(result)})
|
||||
reply = client.chat.completions.create(model="<model-name>", messages=messages, tools=tools)
|
||||
|
||||
print(reply.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Embeddings
|
||||
|
||||
```python
|
||||
result = client.embeddings.create(
|
||||
model="<embedding-model>",
|
||||
input=["llmgw routes requests", "to GPUs it can reach"],
|
||||
)
|
||||
vectors = [d.embedding for d in result.data]
|
||||
```
|
||||
|
||||
```typescript
|
||||
const result = await client.embeddings.create({
|
||||
model: "<embedding-model>",
|
||||
input: ["llmgw routes requests", "to GPUs it can reach"],
|
||||
});
|
||||
const vectors = result.data.map((d) => d.embedding);
|
||||
```
|
||||
|
||||
An embedding model is a model of its own; a chat model does not answer
|
||||
`/v1/embeddings`.
|
||||
|
||||
## The Responses API
|
||||
|
||||
`/v1/responses` is served too, for clients that speak nothing else (Codex
|
||||
among them). llmgw translates it to chat completions, so it works with every
|
||||
model the gateway serves; what has no equivalent there — OpenAI's hosted
|
||||
tools, reasoning summaries — is ignored rather than refused.
|
||||
|
||||
```python
|
||||
response = client.responses.create(
|
||||
model="<model-name>",
|
||||
instructions="Answer in one sentence.",
|
||||
input="Why is the sky blue?",
|
||||
)
|
||||
print(response.output_text)
|
||||
```
|
||||
|
||||
Streamed, the events are the Responses API's own:
|
||||
|
||||
```python
|
||||
stream = client.responses.create(model="<model-name>", input="Count to ten.", stream=True)
|
||||
for event in stream:
|
||||
if event.type == "response.output_text.delta":
|
||||
print(event.delta, end="", flush=True)
|
||||
```
|
||||
|
||||
## Batches
|
||||
|
||||
For work that need not finish now: OpenAI's Files and Batch APIs. A batch runs
|
||||
at the lowest priority, so it uses what interactive traffic leaves free, and
|
||||
each line is counted against the key that submitted it.
|
||||
|
||||
```python
|
||||
import json
|
||||
|
||||
with open("requests.jsonl", "w") as f:
|
||||
for i, text in enumerate(["first document", "second document"]):
|
||||
f.write(json.dumps({
|
||||
"custom_id": f"doc-{i}",
|
||||
"method": "POST",
|
||||
"url": "/v1/chat/completions",
|
||||
"body": {"model": "<model-name>",
|
||||
"messages": [{"role": "user", "content": "Summarise: " + text}]},
|
||||
}) + "\n")
|
||||
|
||||
batch_file = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
|
||||
batch = client.batches.create(
|
||||
input_file_id=batch_file.id,
|
||||
endpoint="/v1/chat/completions",
|
||||
completion_window="24h",
|
||||
)
|
||||
|
||||
# Later:
|
||||
batch = client.batches.retrieve(batch.id)
|
||||
if batch.status == "completed":
|
||||
for line in client.files.content(batch.output_file_id).text.splitlines():
|
||||
result = json.loads(line)
|
||||
print(result["custom_id"], result["response"]["body"]["choices"][0]["message"]["content"])
|
||||
```
|
||||
|
||||
## The Anthropic SDK
|
||||
|
||||
`/v1/messages` is Anthropic's Messages API, translated to chat completions and
|
||||
back, streamed or not, with tools and images. It serves the gateway's own
|
||||
models — `model` is a model this gateway serves, not a Claude model.
|
||||
|
||||
```
|
||||
pip install anthropic
|
||||
```
|
||||
|
||||
```python
|
||||
import os
|
||||
import anthropic
|
||||
|
||||
client = anthropic.Anthropic(base_url="https://llm.example.com",
|
||||
api_key=os.environ["LLMGW_API_KEY"])
|
||||
|
||||
message = client.messages.create(
|
||||
model="<model-name>",
|
||||
max_tokens=1024,
|
||||
messages=[{"role": "user", "content": "Say hello in Turkish."}],
|
||||
)
|
||||
print(message.content[0].text)
|
||||
|
||||
with client.messages.stream(
|
||||
model="<model-name>",
|
||||
max_tokens=1024,
|
||||
messages=[{"role": "user", "content": "Count to ten."}],
|
||||
) as stream:
|
||||
for text in stream.text_stream:
|
||||
print(text, end="", flush=True)
|
||||
```
|
||||
|
||||
```typescript
|
||||
import Anthropic from "@anthropic-ai/sdk";
|
||||
|
||||
const client = new Anthropic({
|
||||
baseURL: "https://llm.example.com",
|
||||
apiKey: process.env.LLMGW_API_KEY,
|
||||
});
|
||||
|
||||
const message = await client.messages.create({
|
||||
model: "<model-name>",
|
||||
max_tokens: 1024,
|
||||
messages: [{ role: "user", content: "Say hello in Turkish." }],
|
||||
});
|
||||
for (const block of message.content) {
|
||||
if (block.type === "text") console.log(block.text);
|
||||
}
|
||||
```
|
||||
|
||||
```go
|
||||
client := anthropic.NewClient(
|
||||
option.WithBaseURL("https://llm.example.com/"),
|
||||
option.WithAPIKey(os.Getenv("LLMGW_API_KEY")),
|
||||
)
|
||||
message, err := client.Messages.New(ctx, anthropic.MessageNewParams{
|
||||
Model: "<model-name>",
|
||||
MaxTokens: 1024,
|
||||
Messages: []anthropic.MessageParam{
|
||||
anthropic.NewUserMessage(anthropic.NewTextBlock("Say hello in Turkish.")),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
for _, block := range message.Content {
|
||||
if text, ok := block.AsAny().(anthropic.TextBlock); ok {
|
||||
fmt.Println(text.Text)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`github.com/anthropics/anthropic-sdk-go` and its `option` package.)
|
||||
|
||||
With curl:
|
||||
|
||||
```
|
||||
curl https://llm.example.com/v1/messages \
|
||||
-H "x-api-key: $LLMGW_API_KEY" \
|
||||
-H "anthropic-version: 2023-06-01" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "<model-name>", "max_tokens": 1024,
|
||||
"messages": [{"role": "user", "content": "Say hello in Turkish."}]}'
|
||||
```
|
||||
|
||||
## What llmgw adds
|
||||
|
||||
Each is a header, so a request stays exactly what the SDK would send anywhere
|
||||
else. Set one for every request on the client, or for one request:
|
||||
|
||||
| | Every request | One request |
|
||||
|---|---|---|
|
||||
| Python (both SDKs) | `OpenAI(..., default_headers={...})` | `create(..., extra_headers={...})` |
|
||||
| TypeScript (both SDKs) | `new OpenAI({ ..., defaultHeaders: {...} })` | `create({...}, { headers: {...} })` |
|
||||
| Go (both SDKs) | `openai.NewClient(..., option.WithHeader(k, v))` | `New(ctx, params, option.WithHeader(k, v))` |
|
||||
| curl | | `-H "k: v"` |
|
||||
|
||||
### Stored conversations: `x-llmgw-thread`
|
||||
|
||||
Send the same thread id with each request and only the new turn; llmgw puts
|
||||
the stored conversation in front of it. The conversation continues on
|
||||
whichever model serves the next turn, even a different one. It must be
|
||||
turned on for your project by an operator (`llmgw keys threads`); a request
|
||||
that names a thread on a gateway that keeps none is refused rather than
|
||||
answered without its history.
|
||||
|
||||
```python
|
||||
import uuid
|
||||
|
||||
thread = str(uuid.uuid4()) # anyone in your project who knows it reads it
|
||||
for turn in ["My name is Deniz.", "What is my name?"]:
|
||||
reply = client.chat.completions.create(
|
||||
model="<model-name>",
|
||||
messages=[{"role": "user", "content": turn}],
|
||||
extra_headers={"x-llmgw-thread": thread},
|
||||
)
|
||||
print(reply.choices[0].message.content)
|
||||
```
|
||||
|
||||
A thread belongs to the project, not to the key, so use an id nobody can
|
||||
guess: at most 128 letters, digits and `- _ . :`. When the conversation no
|
||||
longer fits the model, the oldest turns are left out and the reply carries
|
||||
`x-llmgw-thread-trimmed` with how many. `DELETE /v1/threads/{id}` forgets one:
|
||||
|
||||
```
|
||||
curl -X DELETE https://llm.example.com/v1/threads/$THREAD \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY"
|
||||
```
|
||||
|
||||
### Sessions: `x-session-id`
|
||||
|
||||
Requests with the same session id go to the same replica while it is
|
||||
healthy, so a conversation that resends its history finds it already in that
|
||||
replica's cache and answers sooner. Without the header, the body's `user`
|
||||
field is used. Nothing is stored; it is only a routing hint.
|
||||
|
||||
```python
|
||||
client = OpenAI(base_url="https://llm.example.com/v1", api_key=os.environ["LLMGW_API_KEY"],
|
||||
default_headers={"x-session-id": "chat-7f3a"})
|
||||
```
|
||||
|
||||
### Tags and metadata
|
||||
|
||||
Requests can be labelled so the usage they record can be told apart — by
|
||||
feature, by customer, by job. Send OpenAI's own `metadata`, string keys to
|
||||
string values, or the same as the `x-llmgw-tags` header for a client that
|
||||
cannot set `metadata`; the `user` field is recorded too. Which tags also
|
||||
become metric labels is the operator's choice (`router.tag_labels`).
|
||||
|
||||
```python
|
||||
reply = client.chat.completions.create(
|
||||
model="<model-name>",
|
||||
messages=[{"role": "user", "content": "Classify this ticket."}],
|
||||
metadata={"feature": "triage", "customer": "acme"},
|
||||
user="user-1234",
|
||||
)
|
||||
```
|
||||
|
||||
```
|
||||
curl https://llm.example.com/v1/chat/completions \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "x-llmgw-tags: feature=triage,customer=acme" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model": "<model-name>", "messages": [{"role": "user", "content": "Classify this ticket."}]}'
|
||||
```
|
||||
|
||||
### Canaries: `x-llmgw-variant`
|
||||
|
||||
While an operator tries a model's newer weights on a share of its traffic, a
|
||||
session stays on one side. `x-llmgw-variant: canary` sends a request to the
|
||||
canary deliberately — to try it before it is promoted, or to compare the two.
|
||||
|
||||
```python
|
||||
reply = client.chat.completions.create(
|
||||
model="<model-name>",
|
||||
messages=[{"role": "user", "content": "hello"}],
|
||||
extra_headers={"x-llmgw-variant": "canary"},
|
||||
)
|
||||
```
|
||||
|
||||
### Memory notes: `/v1/memory`
|
||||
|
||||
Notes an agent keeps for itself across conversations: a key, some text, and
|
||||
optional tags, private to your project and to the agent named in
|
||||
`x-llmgw-agent` (`default` when it is not sent). An operator turns them on for
|
||||
a project (`llmgw keys memory`). They are plain REST, so any HTTP client
|
||||
will do:
|
||||
|
||||
```
|
||||
H="Authorization: Bearer $LLMGW_API_KEY"
|
||||
|
||||
curl -X PUT https://llm.example.com/v1/memory/favourite-colour -H "$H" \
|
||||
-H "x-llmgw-agent: helper" -H "Content-Type: application/json" \
|
||||
-d '{"text": "The user likes dark green.", "tags": ["preference"]}'
|
||||
|
||||
curl "https://llm.example.com/v1/memory?q=green" -H "$H" -H "x-llmgw-agent: helper"
|
||||
curl "https://llm.example.com/v1/memory?tag=preference" -H "$H" -H "x-llmgw-agent: helper"
|
||||
curl https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
|
||||
curl -X DELETE https://llm.example.com/v1/memory/favourite-colour -H "$H" -H "x-llmgw-agent: helper"
|
||||
```
|
||||
|
||||
A key is up to 200 bytes, a note up to 16 KiB, with at most 16 tags of
|
||||
lowercase letters, digits, `-` and `_`, and 1000 notes per agent.
|
||||
|
||||
### MCP: `/mcp`
|
||||
|
||||
The same notes as tools an agent can be given — `remember`, `recall` and
|
||||
`forget` — from an MCP server at `https://llm.example.com/mcp` (the HTTP
|
||||
transport, answered as plain JSON). Add it to your agent's MCP servers with
|
||||
the key as a bearer token and, if you want notes kept apart, `x-llmgw-agent`.
|
||||
llmgw does not put these tools into requests by itself: an agent that is not
|
||||
given the server never sees its notes.
|
||||
|
||||
```
|
||||
curl https://llm.example.com/mcp \
|
||||
-H "Authorization: Bearer $LLMGW_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
|
||||
```
|
||||
|
||||
### Headers llmgw sends back
|
||||
|
||||
| Header | Says |
|
||||
|---|---|
|
||||
| `x-request-id` | The request's id in the gateway's logs and usage records — quote it when asking an operator about a request. |
|
||||
| `x-llmgw-served-model` | The model that answered, when it was not the one asked for: a spent budget can move a request to a cheaper model. |
|
||||
| `x-llmgw-queue-ms` | How long the request waited for room before it was sent. |
|
||||
| `x-llmgw-thread-trimmed` | How many stored turns were left out to fit the model. |
|
||||
|
||||
## When something is refused
|
||||
|
||||
Errors come back in the shape of the API that was called — OpenAI's
|
||||
`{"error": {"message": …, "type": …}}` or Anthropic's — so each SDK raises
|
||||
its usual exception with the gateway's message in it. The ones worth knowing:
|
||||
|
||||
| Status | Usually means |
|
||||
|---|---|
|
||||
| 400 | Besides a malformed request: a prompt too long for the model, or a guardrail policy refused it (the code `content_policy_violation`). |
|
||||
| 401 | No key, a mistyped one, or one that was revoked or has expired. |
|
||||
| 403 | The key may not use that model. |
|
||||
| 429 | A rate limit or budget of the key, its project or its team was reached. `Retry-After` says when to try again, and is absent when waiting will not help — a spent monthly budget. |
|
||||
| 503 | No model of that name is served (the message lists the ones that are), the model is still loading, or every replica is busy and the request waited as long as it may. The SDKs retry by themselves; a name that is not served will not start to be. |
|
||||
|
||||
## Priority
|
||||
|
||||
With keys on, the key decides the request's priority band and routing
|
||||
profile, set by an operator; the `x-llmgw-band` and `x-llmgw-profile` headers
|
||||
are ignored, so a client cannot promote itself. On a gateway without keys
|
||||
they are read from the request.
|
||||
+17
-2
@@ -117,6 +117,16 @@ and **UPDATE** when the registry has new weights for it. On a row:
|
||||
canary is removed) and **Take the canary out…**. A canary that errs or
|
||||
answers more slowly than the model is taken out by itself. The MODEL
|
||||
column shows `(10% to canary)` and `(canary of …)`.
|
||||
- **Ask it a quality suite…** — one of the suites the control plane has (in
|
||||
`control.eval_dir`, or built in: `arithmetic`, `json-format`), asked
|
||||
through the router; the dialog counts the answers as they come in, then
|
||||
shows the score, each case and the model's last runs. Esc stops watching;
|
||||
the run carries on and is kept. **Its quality runs** lists them, as
|
||||
`llmgw ctl eval --history` does.
|
||||
- **Hold its updates to a quality suite…** — the suite and score its new
|
||||
weights must reach before an update or a promotion moves it to them
|
||||
(`llmgw ctl model gate`); **none** takes it away. With one set, **Update
|
||||
to …** and **Promote the canary…** say so.
|
||||
- **Why is it placed where it is** — `llmgw ctl explain-placement`.
|
||||
- **Try again**, for a model parked after failing every start.
|
||||
- **Remove** — its replicas stop and it leaves the catalog. A model from
|
||||
@@ -138,7 +148,10 @@ team or project is guaranteed and may at most hold, and what it holds now.
|
||||
**Mirror** is what the model mirror holds, one row for each version of each
|
||||
model — a file, or the set of files a model of several is: shards, a vision
|
||||
projector, a config and tokenizer — with how far its download has got and
|
||||
which models use it. **A** downloads a URL into it; on a row, its files with
|
||||
which models use it. **A** downloads a URL into it; **I** imports a model
|
||||
from an Ollama on this machine into a mirror directory on this machine — the
|
||||
mirror beside the control plane — and shows the model definition to add for
|
||||
it (a mirror elsewhere: `llmgw mirror import-ollama` there); on a row, its files with
|
||||
their checksums and where each was fetched from, the source a model
|
||||
definition names for it, or remove it — with a warning when a model still
|
||||
uses it.
|
||||
@@ -195,7 +208,9 @@ row: add a team or project inside it, rename, move, limits for the level and
|
||||
for each of its members, its capacity quota, and for a project how long
|
||||
conversations and agent notes are kept, and whether its requests are
|
||||
(**Keep requests…**: for how long, and what share), and whether they stay in
|
||||
the cluster (**Requests stay in the cluster…**: never an outside provider). **Longest key lifetime…** caps how
|
||||
the cluster (**Requests stay in the cluster…**: never an outside provider),
|
||||
and whether the same request is answered again from memory (**Cache
|
||||
answers…**: for how long, and whether only reproducible answers or any). **Longest key lifetime…** caps how
|
||||
long a key made there may live, and is the expiry of one made without any
|
||||
(`llmgw keys ttl`; `auth.max_key_ttl` in server.yaml caps every key).
|
||||
**Remove** takes away only something empty.
|
||||
|
||||
@@ -67,10 +67,20 @@ until deleted (`DELETE /v1/files/{id}`) or, for what a batch wrote, until
|
||||
them can read them, and a project under a `no_store` policy cannot make
|
||||
them.
|
||||
|
||||
**Reused answers** — only for a project that asked (`llmgw keys cache`):
|
||||
each answer that may be reused, as the client got it, and a hash of the
|
||||
request it answers, in the gateway process's memory — never in the store,
|
||||
never on disk — for the project's TTL or until pushed out by newer ones
|
||||
(`router.cache.max_bytes`), and gone with a restart. Never under a `no_store`
|
||||
policy. Forgetting a person does not reach into it; the answers age out by
|
||||
the TTL. The store keeps only the setting, and a usage record's mark that it
|
||||
was answered from the cache.
|
||||
|
||||
## What is not stored
|
||||
|
||||
Prompts. Completions. Anything the model was shown or produced, unless the
|
||||
project turned on conversations, notes or kept requests, or sent a batch.
|
||||
(Reused answers are held in memory only, as above.)
|
||||
|
||||
The chat page in llmgw's own web interface keeps the conversation in a
|
||||
JavaScript variable in the browser. Refresh it and the conversation is gone,
|
||||
|
||||
+400
-3
@@ -252,7 +252,8 @@ knowing before you edit it:
|
||||
`LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
|
||||
keys requests presented, is off unless `debug_pprof: true`, and then behind
|
||||
the token too. To see the dashboard from elsewhere use `/admin` on the
|
||||
client port — do not move this listener.
|
||||
client port — do not move this listener. The API itself is described in
|
||||
[admin-api.md](admin-api.md).
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
@@ -383,6 +384,46 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
|
||||
or from the admin page, or by copying files into the directory by hand — the
|
||||
mirror is a directory, and anything that puts files in it works.
|
||||
|
||||
Files already on the mirror machine go in with `import`, which copies them the
|
||||
way a download is kept — under a temporary name until whole, the checksum
|
||||
beside them for nodes to check against, a version never replaced — and checks
|
||||
that a `.gguf` is one:
|
||||
|
||||
```
|
||||
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
|
||||
--version 2026-10 --sha256 <the checksum its publisher gave>
|
||||
```
|
||||
|
||||
#### From Ollama
|
||||
|
||||
Models you already run in Ollama are GGUFs llama.cpp serves as they are.
|
||||
On the mirror machine, with Ollama's models readable there:
|
||||
|
||||
```
|
||||
llmgw mirror import-ollama # lists what Ollama has
|
||||
llmgw mirror import-ollama llama3.2:3b
|
||||
```
|
||||
|
||||
It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
|
||||
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
|
||||
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
|
||||
model's GGUF, and a vision model's projector beside it, each checked against
|
||||
the sha256 Ollama named its file by; a file that does not match is not
|
||||
mirrored, and nothing of the model is. The version is the tag and the start of
|
||||
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
|
||||
`latest` moves in Ollama and a mirror version must not; `--model` and
|
||||
`--version` choose otherwise. It ends with a model definition to save and
|
||||
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
|
||||
with.
|
||||
|
||||
Ollama's prompt template is not carried: llama.cpp renders the chat template
|
||||
inside the GGUF, which for any recent model is the one it was trained on. Its
|
||||
system prompt and adapters are not carried either, and the import says so.
|
||||
Models Ollama keeps as separate tensors rather than one GGUF, or as several
|
||||
weight layers, are refused with a sentence saying so — mirror that model's GGUF
|
||||
from its publisher instead. When the mirror runs beside the control plane, the
|
||||
console's Mirror view does the same with **I**.
|
||||
|
||||
## 4. Preparing a node
|
||||
|
||||
**Do this before onboarding.** A node missing its driver or engine registers
|
||||
@@ -428,6 +469,71 @@ sudo llmgw doctor
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### MIG, if you want to cut a card into pieces
|
||||
|
||||
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
|
||||
GPU) instances, each with its own memory, its own share of the card's compute
|
||||
and its own UUID. llmgw places a model on an instance as it would on a card,
|
||||
which makes MIG the way to run several small models on one large card without
|
||||
them competing for it.
|
||||
|
||||
**llmgw does not create the instances.** How a card is cut is your decision,
|
||||
made once with `nvidia-smi` on the node; the agent reads what is there when
|
||||
it starts:
|
||||
|
||||
```
|
||||
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
|
||||
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
|
||||
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
|
||||
nvidia-smi -L # the instances, with their MIG- UUIDs
|
||||
sudo systemctl restart llmgw-agent # so it registers the new devices
|
||||
```
|
||||
|
||||
`-C` matters: a GPU instance with no compute instance inside it is not a
|
||||
device yet. Turning MIG on takes effect at the card's next reset
|
||||
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
|
||||
do not survive a reboot unless something recreates them — NVIDIA's
|
||||
`mig-parted` or a systemd unit of your own.
|
||||
|
||||
**Classes.** A card in MIG mode is reported as its instances instead of as
|
||||
the whole card, and each instance's class is the card's class and the
|
||||
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
|
||||
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
|
||||
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
|
||||
|
||||
```yaml
|
||||
profiles:
|
||||
- name: mig
|
||||
engine: vllm
|
||||
gpu_class: a100-80g-mig-3g.40gb
|
||||
```
|
||||
|
||||
**One instance a replica.** A process can open only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer access between them, so a replica on
|
||||
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
|
||||
MIG class asking for more is refused when the model is added, with that
|
||||
reason. A model too large for one instance wants a larger profile or a whole
|
||||
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
|
||||
instance is offered for a model it holds, and never several together.
|
||||
|
||||
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
|
||||
device 1002 (1000 + 100 × card + instance), and every view shows it as
|
||||
`0/mig2`. The engine is given the instance by its UUID —
|
||||
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
|
||||
build. llama.cpp's Vulkan build cannot see MIG instances at all.
|
||||
|
||||
**What is measured.** Each instance's memory is its own; temperature, power
|
||||
and wear are its card's, shown on each of its instances. Utilisation is not
|
||||
reported for an instance, because no driver measures it per instance: the
|
||||
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
|
||||
published for it rather than a zero.
|
||||
|
||||
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
|
||||
card in MIG mode with none created (nothing can run on it: CUDA will not open
|
||||
the whole card while MIG is on), of GPU instances with no compute instance,
|
||||
and of a mode change waiting for a reset. Such a card is also kept out of
|
||||
placement, with the reason, in `llmgw ctl nodes`.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
```
|
||||
@@ -458,6 +564,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
@@ -492,8 +599,8 @@ engines:
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
|
||||
`engines.trtllm.python`): the agent then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
@@ -544,6 +651,82 @@ which drift between releases:
|
||||
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
|
||||
```
|
||||
|
||||
#### TensorRT-LLM, which you bring
|
||||
|
||||
llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
|
||||
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
|
||||
container; an installer guessing at all of that would fail on most machines it
|
||||
met, a long way into a large download. Install it on the node yourself — a
|
||||
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
|
||||
interpreter in that node's `agent.yaml`:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
trtllm:
|
||||
python: /opt/trtllm/venv/bin/python # python3 inside NVIDIA's container
|
||||
```
|
||||
|
||||
The node then offers the engine `trtllm` whenever that interpreter has the
|
||||
`tensorrt_llm` package; without the line it offers nothing, whatever is on
|
||||
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
|
||||
`--served_model_name`, and publishes none of the queue figures the router
|
||||
reads. Check a node's against the flags llmgw builds:
|
||||
|
||||
```
|
||||
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
|
||||
```
|
||||
|
||||
A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
|
||||
downloads a repository itself, or loads one from the mirror:
|
||||
|
||||
```yaml
|
||||
name: llama-3.1-70b-instruct
|
||||
source: meta-llama/Llama-3.1-70B-Instruct
|
||||
profiles:
|
||||
- name: h100x4
|
||||
engine: trtllm
|
||||
gpu_class: h100-80g
|
||||
tp: 4
|
||||
args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
|
||||
```
|
||||
|
||||
The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
|
||||
--served_model_name …`, through a few lines of Python of its own that do what
|
||||
trtllm-serve has no flag for:
|
||||
|
||||
- **the key.** trtllm-serve checks none; every path but `/health` is made to
|
||||
demand the one the gateway gives the replica, so the port on the node is not
|
||||
a way around the gateway.
|
||||
- **the metrics.** Its options file turns on `return_perf_metrics` and
|
||||
`enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
|
||||
and its queue figures do not move. The router reads
|
||||
`trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
|
||||
`trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
|
||||
is JSON. They are set as the engine iterates, so an engine gone idle shows
|
||||
its last batch until the next request.
|
||||
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
|
||||
are set, after the profile's own `env`.
|
||||
|
||||
What it does not do:
|
||||
|
||||
- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
|
||||
under MPI or Slurm from one of them, which the agents, each starting its own
|
||||
node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
|
||||
or SGLang for one.
|
||||
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
|
||||
TensorRT-LLM builds kernels for.
|
||||
- **take an options file from a profile.** `--config` and
|
||||
`--extra_llm_api_options` are llmgw's; change a setting with `--set
|
||||
PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
|
||||
refused, like the other engines' equivalents: `--trust_remote_code`,
|
||||
`--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
|
||||
`--num_serve_frontends` (both would serve without the key), and the flags the
|
||||
agent sets.
|
||||
|
||||
This engine was written against TensorRT-LLM's documented command line,
|
||||
server and metrics, and tested with stand-ins for them; it has not yet run on a
|
||||
datacenter GPU. Run `engine-check` on the first node before placing a model on it.
|
||||
|
||||
### Weights
|
||||
|
||||
Nothing to do here if you built the mirror in step 3 — the agent fetches what a
|
||||
@@ -911,6 +1094,118 @@ llmgw ctl model canary qwen-7b --abort # or not
|
||||
- **Promote** is the update the canary was a trial of: the model's replicas
|
||||
are replaced one at a time, and `llmgw ctl model rollback` goes back.
|
||||
|
||||
### Quality suites, and holding new weights to one
|
||||
|
||||
A canary says whether new weights serve as fast and as reliably; it does not
|
||||
say whether their answers are as good. A republished quantisation or a chat
|
||||
template that no longer matches can make a model faster and worse at once. A
|
||||
quality suite is a handful of questions with known answers, asked of a model
|
||||
and scored:
|
||||
|
||||
```
|
||||
llmgw ctl eval --suites # what the control plane can run
|
||||
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
|
||||
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
|
||||
llmgw ctl eval --history qwen-7b # the runs kept
|
||||
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
|
||||
```
|
||||
|
||||
It prints each case's score and why, then the suite's: the cases' scores
|
||||
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
|
||||
`--min-score`); it exits non-zero below it, so a script can stop on it.
|
||||
|
||||
- **How it is asked:** on the control plane, through the router as the
|
||||
gateway's own requests, the way a replay is — past admission, quotas,
|
||||
accounting and the guardrails, so the score is the weights' and not a
|
||||
tenant's limits'. It reaches the model exactly as a client's request
|
||||
would, a canary or a model loading on demand included. Answers are asked
|
||||
at temperature 0 unless a case says otherwise, and a reasoning model's
|
||||
`<think>` block is not part of its answer.
|
||||
- **Where suites come from:** `--suite` is a file on the machine running
|
||||
`ctl`, sent whole, or the name of one in the control plane's
|
||||
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
|
||||
models: `arithmetic` (sums and word problems, each scored by the number in
|
||||
the answer) and `json-format` (answers asked for as JSON of a given shape);
|
||||
their files are in `deploy/eval/`, to copy and change. A file in the
|
||||
directory with the same name replaces the built-in one.
|
||||
- **What is kept:** every run, in the store, with the model, the repository
|
||||
and revision the answering weights came from (when known), the suite's name
|
||||
and sha256, when, the score, and each case's answer (up to 2,000
|
||||
characters), score and reason. A run on a canary is filed under the model
|
||||
it is a canary of. Runs are not pruned.
|
||||
|
||||
A suite is YAML (or JSON lines: one case on each line, and optionally a line
|
||||
without a prompt that holds the suite's own settings):
|
||||
|
||||
```yaml
|
||||
name: support # the file's name unless given
|
||||
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
|
||||
judge: llama-70b # grades the cases scored by a rubric
|
||||
system: You are a support agent for Acme. # every case's, unless it has one
|
||||
max_tokens: 256 # every case's; 512 unless given
|
||||
cases:
|
||||
- id: refund-window
|
||||
prompt: How many days do customers have to ask for a refund?
|
||||
number: 30 # the last number in the answer, ±tolerance
|
||||
tolerance: 0
|
||||
- prompt: Which plan includes SSO?
|
||||
contains: [Enterprise] # every string must appear
|
||||
ignore_case: true
|
||||
- prompt: Reply with only the word OK.
|
||||
exact: OK # the whole answer, white space around it aside
|
||||
- prompt: What is today's ISO date format?
|
||||
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
|
||||
- prompt: Give the order as JSON with "id" and "total".
|
||||
json_schema: # the answer, read as JSON, satisfies it
|
||||
type: object
|
||||
required: [id, total]
|
||||
properties: { id: {type: string}, total: {type: number, minimum: 0} }
|
||||
- messages: # a conversation instead of a prompt
|
||||
- { role: user, content: My card was charged twice. }
|
||||
judge: Apologises, and says the duplicate charge is refunded within 5 days.
|
||||
weight: 3 # counts three times; 1 unless given
|
||||
```
|
||||
|
||||
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
|
||||
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
|
||||
still read. The schema supports `type`, `properties`, `required`,
|
||||
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
|
||||
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
|
||||
is read rather than passed unchecked. A `judge` case asks another model on the
|
||||
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
|
||||
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
|
||||
got no answer scores 0 and says why.
|
||||
|
||||
**Holding updates to a suite.** `model update` and `model promote` take
|
||||
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
|
||||
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
|
||||
definition added with `ctl model add` — so every update and promotion of it
|
||||
is held to it, from `ctl`, the console or anywhere:
|
||||
|
||||
```
|
||||
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
|
||||
llmgw ctl model update qwen-7b # asked the suite first
|
||||
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
|
||||
llmgw ctl model gate qwen-7b --off
|
||||
```
|
||||
|
||||
- **Promote** asks the canary the suite before anything moves; below the
|
||||
score the job fails with the score, and the canary stays, with its share,
|
||||
until you take it out.
|
||||
- **Update** runs the new weights first as `MODEL-canary` given none of the
|
||||
model's requests (hidden from clients like any canary), asks it the suite,
|
||||
then either promotes it — the model's replicas are replaced one at a time,
|
||||
as for any update — or takes it out again, leaving the model on its old
|
||||
weights. That needs room for one more replica while it runs; a model that
|
||||
already has a canary is promoted or aborted first. Without a suite, an
|
||||
update swaps the weights in as it always has.
|
||||
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
|
||||
kept with the job's id beside it.
|
||||
|
||||
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
|
||||
control plane has, the score as it comes in), **Its quality runs**, and **Hold
|
||||
its updates to a quality suite…**.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
@@ -1090,6 +1385,46 @@ same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
### Tracing
|
||||
|
||||
Metrics say that a model took four minutes to start; a trace says where they
|
||||
went. Give the control plane and every agent the same OpenTelemetry collector,
|
||||
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
|
||||
|
||||
```yaml
|
||||
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
|
||||
tracing:
|
||||
endpoint: http://otel.internal:4318
|
||||
sample_ratio: 1 # of requests that arrive without a decision of their own
|
||||
```
|
||||
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
|
||||
neither, nothing is traced and nothing is sent: every span is a no-op, and a
|
||||
command to a node carries no trace context.
|
||||
|
||||
What is traced:
|
||||
|
||||
| Trace | Spans |
|
||||
|---|---|
|
||||
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
|
||||
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
|
||||
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
|
||||
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
|
||||
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
|
||||
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
|
||||
| An engine install | `install engine` on the node, a trace of its own. |
|
||||
|
||||
So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
from a control plane that is not tracing. The agent needs its own `tracing`
|
||||
block either way — without one its spans go nowhere, and the control plane's
|
||||
trace simply ends at `send start`.
|
||||
|
||||
## PostgreSQL, when SQLite is no longer the right shape
|
||||
|
||||
SQLite is the default and stays the right answer for most installations: one
|
||||
@@ -1208,6 +1543,68 @@ llmgw keys requests --project Acme/ml/train --off
|
||||
requests past their hours, or of a project turned off, are removed hourly.
|
||||
The console has it as **Keep requests…** on a project.
|
||||
|
||||
## Reusing answers
|
||||
|
||||
A project that asks the same thing again and again — an evaluation run over
|
||||
the same prompts, embeddings of documents that do not change, an agent that
|
||||
retries — can have the same request answered from memory rather than by the
|
||||
model:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/ml/train --ttl 1h # reproducible answers
|
||||
llmgw keys cache --project Acme/ml/eval --ttl 24h --any # every answer
|
||||
llmgw keys cache --project Acme/ml/train # how it stands
|
||||
llmgw keys cache --project Acme/ml/train --off
|
||||
```
|
||||
|
||||
- **What is the same request:** the same project, the same endpoint, the same
|
||||
model (by its catalog name, whichever alias was used) and the same body,
|
||||
with `stream`, `stream_options`, `user`, `metadata`, `store` and
|
||||
`service_tier` left out and the keys in any order. Keys of another project
|
||||
never see the answer, whatever they send.
|
||||
- **Which answers are reused:** a chat or completion asked at `temperature`
|
||||
0 or with a `seed`, and every embedding, rerank and score, which sample
|
||||
nothing. A request sampled above temperature 0 with no seed asks for a new
|
||||
answer, and is asked anew — unless the project says `--any`. Only a whole
|
||||
200 from the model that was asked for is kept: not an error, not a
|
||||
fallback model's or a canary's answer, not one a guardrail changed or
|
||||
stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
|
||||
- **Streams:** an answer given whole is streamed to a client that asks for a
|
||||
stream, as one chunk per choice with the usage chunk if it asked for that;
|
||||
a streamed answer is kept put back together, and served whole to a client
|
||||
that does not stream. Through `/v1/messages` and `/v1/responses` too.
|
||||
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
|
||||
`x-llmgw-cache: no`, is always asked anew. Every answer of a project that
|
||||
reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
|
||||
can be reused), and a hit says its `Age` in seconds.
|
||||
- **Policies still apply:** the limits are checked and the guardrails judge
|
||||
the request before the cache is looked in, and the output policy judges a
|
||||
reused answer as it would a new one. A request continuing a stored
|
||||
conversation, or under a `no_store` policy, is never answered from the
|
||||
cache nor kept in it.
|
||||
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
||||
the model, the key, the rate limit's count — with no tokens and no cost,
|
||||
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
||||
budget and no token limit. The answer itself carries the usage the model
|
||||
reported when it was first given.
|
||||
- **Where it lives:** in each gateway process's memory, at most
|
||||
`router.cache.max_bytes` (128 MB) of answers for all projects together,
|
||||
those used longest ago dropped first; nothing is written to the store, and
|
||||
a restart begins it again. With two control planes behind a balancer each
|
||||
keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
|
||||
applies to what is already kept.
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
max_bytes: 268435456 # 256 MB
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
|
||||
@@ -309,6 +309,28 @@ shown as text. Other schemes are never linked — configuration is not always
|
||||
written by whoever runs the gateway, and a `javascript:` href on a page served to
|
||||
every visitor is a short path to a stored XSS.
|
||||
|
||||
## The docs portal
|
||||
|
||||
The gateway also serves its manual at `/docs`: these pages, rendered, with
|
||||
[clients.md](clients.md) — how to use the gateway from the official OpenAI
|
||||
and Anthropic SDKs — first on its index. The landing page and the account
|
||||
page link to it, so a user's questions about streaming, tools or the Anthropic
|
||||
SDK have an answer that is not you.
|
||||
|
||||
It is public, like the landing page. The client guide is for the person who
|
||||
has no account yet, and the operator documents describe llmgw in general —
|
||||
no address, model, node or key of this installation — which is what the
|
||||
project's repository publishes anyway. A site that would rather not say even
|
||||
that turns sign-in on:
|
||||
|
||||
```yaml
|
||||
landing:
|
||||
docs_sign_in: true
|
||||
```
|
||||
|
||||
A visitor is then sent to sign in first. On a gateway with API keys and no
|
||||
accounts, the portal asks for a key instead, like the API.
|
||||
|
||||
## What the user sees
|
||||
|
||||
They open the link, press the button, and get the key once. The page shows them
|
||||
|
||||
+135
-6
@@ -47,6 +47,8 @@ rather than repeated.
|
||||
| `max_lines` | `int` | [Batch](#batch) |
|
||||
| `concurrency` | `int` | [Batch](#batch) |
|
||||
| `retain_days` | `int` | [Batch](#batch) |
|
||||
| `max_bytes` | `int64` | [Cache](#cache) |
|
||||
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
||||
| `model` | `string` | [Canary](#canary) |
|
||||
| `percent` | `int` | [Canary](#canary) |
|
||||
| `min_requests` | `int` | [Canary](#canary) |
|
||||
@@ -63,6 +65,7 @@ rather than repeated.
|
||||
| `port_range` | `[]int` | [Control](#control) |
|
||||
| `mgmt_interface` | `string` | [Control](#control) |
|
||||
| `timezone` | `string` | [Control](#control) |
|
||||
| `eval_dir` | `string` | [Control](#control) |
|
||||
| `enabled` | `bool` | [Control](#control) |
|
||||
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
||||
| `id` | `string` | [Endpoint](#endpoint) |
|
||||
@@ -77,6 +80,9 @@ rather than repeated.
|
||||
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
||||
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
||||
| `suite` | `string` | [EvalGate](#evalgate) |
|
||||
| `min_score` | `float64` | [EvalGate](#evalgate) |
|
||||
| `judge` | `string` | [EvalGate](#evalgate) |
|
||||
| `url` | `string` | [External](#external) |
|
||||
| `model` | `string` | [External](#external) |
|
||||
| `api_key_file` | `string` | [External](#external) |
|
||||
@@ -90,6 +96,7 @@ rather than repeated.
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
| `docs_sign_in` | `bool` | [Landing](#landing) |
|
||||
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
||||
@@ -110,6 +117,7 @@ rather than repeated.
|
||||
| `adapters` | `[]Adapter` | [Model](#model) |
|
||||
| `fallbacks` | `[]string` | [Model](#model) |
|
||||
| `canary` | `*Canary` | [Model](#model) |
|
||||
| `eval` | `*EvalGate` | [Model](#model) |
|
||||
| `routing_profile` | `string` | [Model](#model) |
|
||||
| `slo` | `SLO` | [Model](#model) |
|
||||
| `context_length` | `int` | [Model](#model) |
|
||||
@@ -205,6 +213,7 @@ rather than repeated.
|
||||
| `tag_labels` | `[]string` | [Router](#router) |
|
||||
| `max_body_bytes` | `int64` | [Router](#router) |
|
||||
| `batch` | `Batch` | [Router](#router) |
|
||||
| `cache` | `Cache` | [Router](#router) |
|
||||
| `trusted_proxies` | `[]string` | [Router](#router) |
|
||||
| `timeouts` | `Timeouts` | [Router](#router) |
|
||||
| `scoring` | `Scoring` | [Router](#router) |
|
||||
@@ -247,7 +256,7 @@ rather than repeated.
|
||||
| `debug_pprof` | `bool` | [Server](#server) |
|
||||
| `agent_insecure` | `bool` | [Server](#server) |
|
||||
| `monitoring` | `Monitoring` | [Server](#server) |
|
||||
| `tracing` | `Tracing` | [Server](#server) |
|
||||
| `tracing` | `obs.Tracing` | [Server](#server) |
|
||||
| `mirror` | `Mirror` | [Server](#server) |
|
||||
| `engines` | `map[string]string` | [Server](#server) |
|
||||
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
||||
@@ -265,6 +274,7 @@ rather than repeated.
|
||||
| `models` | `[]catalog.Model` | [Server](#server) |
|
||||
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
||||
| `bin` | `string` | [SimConfig](#simconfig) |
|
||||
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
||||
| `store` | `string` | [Threads](#threads) |
|
||||
| `retain_hours` | `int` | [Threads](#threads) |
|
||||
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
|
||||
@@ -323,6 +333,7 @@ rather than repeated.
|
||||
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
|
||||
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
|
||||
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
|
||||
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
|
||||
| `name` | `string` | [config.Band](#configband) |
|
||||
| `priority` | `int` | [config.Band](#configband) |
|
||||
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
||||
@@ -332,6 +343,7 @@ rather than repeated.
|
||||
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
||||
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
||||
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
||||
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
||||
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
||||
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
||||
@@ -571,6 +583,22 @@ every interactive request.
|
||||
RetainDays is how long a batch's answer and error files are kept;
|
||||
30 unless set. Uploaded files are kept until deleted.
|
||||
|
||||
## Cache
|
||||
|
||||
### `max_bytes`
|
||||
|
||||
`int64`
|
||||
|
||||
MaxBytes bounds every project's answers together; 128 MB unless set.
|
||||
Past it, the answers used longest ago go first.
|
||||
|
||||
### `max_entry_bytes`
|
||||
|
||||
`int64`
|
||||
|
||||
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
||||
unless set.
|
||||
|
||||
## Canary
|
||||
|
||||
### `model`
|
||||
@@ -666,6 +694,15 @@ MgmtInterface is the ethernet interface NCCL bootstraps over.
|
||||
Timezone is where models' scaling schedules are read, an IANA name
|
||||
like Europe/Istanbul; empty is the control plane's own.
|
||||
|
||||
### `eval_dir`
|
||||
|
||||
`string`
|
||||
|
||||
EvalDir is where quality suites are read from by name, for `llmgw ctl
|
||||
eval MODEL --suite NAME`, a model's eval gate and the console. The
|
||||
suites the binary carries (arithmetic, json-format) are found when it
|
||||
has none of the name; a suite here of the same name replaces one.
|
||||
|
||||
### `enabled`
|
||||
|
||||
`bool`
|
||||
@@ -759,6 +796,28 @@ Labels picks nodes carrying all of these.
|
||||
|
||||
Engines are laid over the cluster's: a version, or "" for none.
|
||||
|
||||
## EvalGate
|
||||
|
||||
### `suite`
|
||||
|
||||
`string`
|
||||
|
||||
Suite is a suite's name in control.eval_dir, or one the binary
|
||||
carries (arithmetic, json-format).
|
||||
|
||||
### `min_score`
|
||||
|
||||
`float64`
|
||||
|
||||
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
|
||||
threshold.
|
||||
|
||||
### `judge`
|
||||
|
||||
`string`
|
||||
|
||||
Judge grades the suite's rubric cases, instead of the judge it names.
|
||||
|
||||
## External
|
||||
|
||||
### `url`
|
||||
@@ -855,6 +914,17 @@ Language is what the web portal speaks to someone who has not chosen
|
||||
a language with the switcher at the foot of every page and whose
|
||||
browser asks for none it speaks: "en" or "tr". Empty is English.
|
||||
|
||||
### `docs_sign_in`
|
||||
|
||||
`bool`
|
||||
|
||||
DocsSignIn serves the docs portal at /docs only to someone signed in.
|
||||
Off, the portal is public like this page: its client guide is for the
|
||||
person who has no account yet, and the operator documents describe
|
||||
llmgw in general — no address, model or key of this installation —
|
||||
which the project's repository publishes anyway. On, a visitor is sent
|
||||
to sign in, and without accounts the portal needs an API key.
|
||||
|
||||
## LlamaCppConfig
|
||||
|
||||
### `bin`
|
||||
@@ -1033,6 +1103,16 @@ moving to it (`llmgw ctl model canary`).
|
||||
|
||||
Takes a [Canary](#canary) block.
|
||||
|
||||
### `eval`
|
||||
|
||||
`*EvalGate`
|
||||
|
||||
Eval is a quality suite new weights must pass before they replace
|
||||
this model's: `llmgw ctl model update` and `model promote` run it
|
||||
against them first, and stop below its score.
|
||||
|
||||
Takes a [EvalGate](#evalgate) block.
|
||||
|
||||
### `routing_profile`
|
||||
|
||||
`string`
|
||||
@@ -1701,6 +1781,16 @@ another. Needs the store.
|
||||
|
||||
Takes a [Batch](#batch) block.
|
||||
|
||||
### `cache`
|
||||
|
||||
`Cache`
|
||||
|
||||
Cache bounds the answers kept for the projects that reuse them
|
||||
(llmgw keys cache). Which projects do, and for how long, is theirs
|
||||
to set; this is only how much memory it may take.
|
||||
|
||||
Takes a [Cache](#cache) block.
|
||||
|
||||
### `trusted_proxies`
|
||||
|
||||
`[]string`
|
||||
@@ -2029,10 +2119,11 @@ Takes a [Monitoring](#monitoring) block.
|
||||
|
||||
### `tracing`
|
||||
|
||||
`Tracing`
|
||||
`obs.Tracing`
|
||||
|
||||
Tracing sends a trace of each inference request to an OpenTelemetry
|
||||
collector. Off unless an endpoint is set here or in
|
||||
Tracing sends traces to an OpenTelemetry collector: each inference
|
||||
request, and on a control plane each placement, replica start and
|
||||
model job. Off unless an endpoint is set here or in
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
|
||||
Takes a [Tracing](#tracing) block.
|
||||
@@ -2184,6 +2275,21 @@ against sim-engines possible without a control plane.
|
||||
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
||||
Empty means whatever started this agent, which is right almost always.
|
||||
|
||||
## TRTLLMConfig
|
||||
|
||||
### `python`
|
||||
|
||||
`string`
|
||||
|
||||
Python is the interpreter of the environment TensorRT-LLM is installed
|
||||
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
|
||||
container. Naming it is what makes this node offer the engine; empty
|
||||
offers nothing.
|
||||
|
||||
The interpreter rather than trtllm-serve, because llmgw starts the
|
||||
server through a few lines of its own that make it demand the
|
||||
gateway's key, which trtllm-serve has no option for.
|
||||
|
||||
## Threads
|
||||
|
||||
### `store`
|
||||
@@ -2282,8 +2388,10 @@ Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
|
||||
|
||||
`float64`
|
||||
|
||||
SampleRatio is the share of requests traced, 0 to 1, when the caller
|
||||
did not decide it with a sampled traceparent of its own. Zero takes 1.
|
||||
SampleRatio is the share of traces kept, 0 to 1, of those nobody
|
||||
upstream already decided. Zero takes 1. A request with a sampled
|
||||
traceparent of its own is kept, and an agent keeps every start the
|
||||
control plane kept, whatever their own ratio says.
|
||||
|
||||
## VLLMConfig
|
||||
|
||||
@@ -2590,6 +2698,18 @@ Takes a [engines.Config](#enginesconfig) block.
|
||||
|
||||
Takes a [supervisor.Config](#supervisorconfig) block.
|
||||
|
||||
### `tracing`
|
||||
|
||||
`obs.Tracing`
|
||||
|
||||
Tracing sends this node's part of each replica start — the weights
|
||||
fetched, the engine launched, the wait for it to answer — to an
|
||||
OpenTelemetry collector, inside the control plane's trace of that
|
||||
start. Give it the collector server.yaml's tracing names. Off unless
|
||||
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
|
||||
|
||||
Takes a [Tracing](#tracing) block.
|
||||
|
||||
## config.Band
|
||||
|
||||
### `name`
|
||||
@@ -2642,6 +2762,15 @@ Takes a [SGLangConfig](#sglangconfig) block.
|
||||
|
||||
Takes a [LlamaCppConfig](#llamacppconfig) block.
|
||||
|
||||
### `trtllm`
|
||||
|
||||
`TRTLLMConfig`
|
||||
|
||||
TRTLLM is TensorRT-LLM, which the agent does not install: a node
|
||||
offers it only when its interpreter is named here.
|
||||
|
||||
Takes a [TRTLLMConfig](#trtllmconfig) block.
|
||||
|
||||
### `sim`
|
||||
|
||||
`SimConfig`
|
||||
|
||||
Reference in new issue
Block a user