Merge features-c2: the docs portal and its links in English and Turkish

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 00:15:25 +02:00
commit e8a62b23b7
201 files changed
+26276 -630

No files matched your search

+400 -3
View File
@@ -252,7 +252,8 @@ knowing before you edit it:
`LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
keys requests presented, is off unless `debug_pprof: true`, and then behind
the token too. To see the dashboard from elsewhere use `/admin` on the
client port — do not move this listener.
client port — do not move this listener. The API itself is described in
[admin-api.md](admin-api.md).
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
budgets mean anything. Without a price every request costs zero and a budget
never binds; without the defaults, a project created without limits is
@@ -383,6 +384,46 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
or from the admin page, or by copying files into the directory by hand — the
mirror is a directory, and anything that puts files in it works.
Files already on the mirror machine go in with `import`, which copies them the
way a download is kept — under a temporary name until whole, the checksum
beside them for nodes to check against, a version never replaced — and checks
that a `.gguf` is one:
```
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
--version 2026-10 --sha256 <the checksum its publisher gave>
```
#### From Ollama
Models you already run in Ollama are GGUFs llama.cpp serves as they are.
On the mirror machine, with Ollama's models readable there:
```
llmgw mirror import-ollama # lists what Ollama has
llmgw mirror import-ollama llama3.2:3b
```
It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
model's GGUF, and a vision model's projector beside it, each checked against
the sha256 Ollama named its file by; a file that does not match is not
mirrored, and nothing of the model is. The version is the tag and the start of
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
`latest` moves in Ollama and a mirror version must not; `--model` and
`--version` choose otherwise. It ends with a model definition to save and
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
with.
Ollama's prompt template is not carried: llama.cpp renders the chat template
inside the GGUF, which for any recent model is the one it was trained on. Its
system prompt and adapters are not carried either, and the import says so.
Models Ollama keeps as separate tensors rather than one GGUF, or as several
weight layers, are refused with a sentence saying so — mirror that model's GGUF
from its publisher instead. When the mirror runs beside the control plane, the
console's Mirror view does the same with **I**.
## 4. Preparing a node
**Do this before onboarding.** A node missing its driver or engine registers
@@ -428,6 +469,71 @@ sudo llmgw doctor
A `-nvml` build is still published beside the portable one; it reads NVLink
from NVML directly, and needs a libc no older than the builder's.
### MIG, if you want to cut a card into pieces
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
GPU) instances, each with its own memory, its own share of the card's compute
and its own UUID. llmgw places a model on an instance as it would on a card,
which makes MIG the way to run several small models on one large card without
them competing for it.
**llmgw does not create the instances.** How a card is cut is your decision,
made once with `nvidia-smi` on the node; the agent reads what is there when
it starts:
```
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
nvidia-smi -L # the instances, with their MIG- UUIDs
sudo systemctl restart llmgw-agent # so it registers the new devices
```
`-C` matters: a GPU instance with no compute instance inside it is not a
device yet. Turning MIG on takes effect at the card's next reset
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
do not survive a reboot unless something recreates them — NVIDIA's
`mig-parted` or a systemd unit of your own.
**Classes.** A card in MIG mode is reported as its instances instead of as
the whole card, and each instance's class is the card's class and the
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
```yaml
profiles:
- name: mig
engine: vllm
gpu_class: a100-80g-mig-3g.40gb
```
**One instance a replica.** A process can open only one MIG device, and
instances have no NVLink or peer-to-peer access between them, so a replica on
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
MIG class asking for more is refused when the model is added, with that
reason. A model too large for one instance wants a larger profile or a whole
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
instance is offered for a model it holds, and never several together.
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
device 1002 (1000 + 100 × card + instance), and every view shows it as
`0/mig2`. The engine is given the instance by its UUID —
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
build. llama.cpp's Vulkan build cannot see MIG instances at all.
**What is measured.** Each instance's memory is its own; temperature, power
and wear are its card's, shown on each of its instances. Utilisation is not
reported for an instance, because no driver measures it per instance: the
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
published for it rather than a zero.
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
card in MIG mode with none created (nothing can run on it: CUDA will not open
the whole card while MIG is on), of GPU instances with no compute instance,
and of a mode change waiting for a reset. Such a card is also kept out of
placement, with the reason, in `llmgw ctl nodes`.
### InfiniBand, if you have it
```
@@ -458,6 +564,7 @@ Nothing to do: the agent installs them once the node joins, under
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
| SGLang | only when you ask for it | the same way as vLLM |
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
node gets the same dependencies whenever it is built, and installed with
@@ -492,8 +599,8 @@ engines:
A changed version reaches every node when it next connects. To use an engine
you installed yourself on a node, name its path in that node's `agent.yaml`
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
then installs none of its own beside it.
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
`engines.trtllm.python`): the agent then installs none of its own beside it.
#### Choosing a node's vLLM build
@@ -544,6 +651,82 @@ which drift between releases:
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
```
#### TensorRT-LLM, which you bring
llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
container; an installer guessing at all of that would fail on most machines it
met, a long way into a large download. Install it on the node yourself — a
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
interpreter in that node's `agent.yaml`:
```yaml
engines:
trtllm:
python: /opt/trtllm/venv/bin/python # python3 inside NVIDIA's container
```
The node then offers the engine `trtllm` whenever that interpreter has the
`tensorrt_llm` package; without the line it offers nothing, whatever is on
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
`--served_model_name`, and publishes none of the queue figures the router
reads. Check a node's against the flags llmgw builds:
```
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
```
A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
downloads a repository itself, or loads one from the mirror:
```yaml
name: llama-3.1-70b-instruct
source: meta-llama/Llama-3.1-70B-Instruct
profiles:
- name: h100x4
engine: trtllm
gpu_class: h100-80g
tp: 4
args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
```
The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
--served_model_name …`, through a few lines of Python of its own that do what
trtllm-serve has no flag for:
- **the key.** trtllm-serve checks none; every path but `/health` is made to
demand the one the gateway gives the replica, so the port on the node is not
a way around the gateway.
- **the metrics.** Its options file turns on `return_perf_metrics` and
`enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
and its queue figures do not move. The router reads
`trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
`trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
is JSON. They are set as the engine iterates, so an engine gone idle shows
its last batch until the next request.
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
are set, after the profile's own `env`.
What it does not do:
- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
under MPI or Slurm from one of them, which the agents, each starting its own
node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
or SGLang for one.
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
TensorRT-LLM builds kernels for.
- **take an options file from a profile.** `--config` and
`--extra_llm_api_options` are llmgw's; change a setting with `--set
PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
refused, like the other engines' equivalents: `--trust_remote_code`,
`--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
`--num_serve_frontends` (both would serve without the key), and the flags the
agent sets.
This engine was written against TensorRT-LLM's documented command line,
server and metrics, and tested with stand-ins for them; it has not yet run on a
datacenter GPU. Run `engine-check` on the first node before placing a model on it.
### Weights
Nothing to do here if you built the mirror in step 3 — the agent fetches what a
@@ -911,6 +1094,118 @@ llmgw ctl model canary qwen-7b --abort # or not
- **Promote** is the update the canary was a trial of: the model's replicas
are replaced one at a time, and `llmgw ctl model rollback` goes back.
### Quality suites, and holding new weights to one
A canary says whether new weights serve as fast and as reliably; it does not
say whether their answers are as good. A republished quantisation or a chat
template that no longer matches can make a model faster and worse at once. A
quality suite is a handful of questions with known answers, asked of a model
and scored:
```
llmgw ctl eval --suites # what the control plane can run
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
llmgw ctl eval --history qwen-7b # the runs kept
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
```
It prints each case's score and why, then the suite's: the cases' scores
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
`--min-score`); it exits non-zero below it, so a script can stop on it.
- **How it is asked:** on the control plane, through the router as the
gateway's own requests, the way a replay is — past admission, quotas,
accounting and the guardrails, so the score is the weights' and not a
tenant's limits'. It reaches the model exactly as a client's request
would, a canary or a model loading on demand included. Answers are asked
at temperature 0 unless a case says otherwise, and a reasoning model's
`<think>` block is not part of its answer.
- **Where suites come from:** `--suite` is a file on the machine running
`ctl`, sent whole, or the name of one in the control plane's
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
models: `arithmetic` (sums and word problems, each scored by the number in
the answer) and `json-format` (answers asked for as JSON of a given shape);
their files are in `deploy/eval/`, to copy and change. A file in the
directory with the same name replaces the built-in one.
- **What is kept:** every run, in the store, with the model, the repository
and revision the answering weights came from (when known), the suite's name
and sha256, when, the score, and each case's answer (up to 2,000
characters), score and reason. A run on a canary is filed under the model
it is a canary of. Runs are not pruned.
A suite is YAML (or JSON lines: one case on each line, and optionally a line
without a prompt that holds the suite's own settings):
```yaml
name: support # the file's name unless given
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
judge: llama-70b # grades the cases scored by a rubric
system: You are a support agent for Acme. # every case's, unless it has one
max_tokens: 256 # every case's; 512 unless given
cases:
- id: refund-window
prompt: How many days do customers have to ask for a refund?
number: 30 # the last number in the answer, ±tolerance
tolerance: 0
- prompt: Which plan includes SSO?
contains: [Enterprise] # every string must appear
ignore_case: true
- prompt: Reply with only the word OK.
exact: OK # the whole answer, white space around it aside
- prompt: What is today's ISO date format?
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
- prompt: Give the order as JSON with "id" and "total".
json_schema: # the answer, read as JSON, satisfies it
type: object
required: [id, total]
properties: { id: {type: string}, total: {type: number, minimum: 0} }
- messages: # a conversation instead of a prompt
- { role: user, content: My card was charged twice. }
judge: Apologises, and says the duplicate charge is refunded within 5 days.
weight: 3 # counts three times; 1 unless given
```
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
still read. The schema supports `type`, `properties`, `required`,
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
is read rather than passed unchecked. A `judge` case asks another model on the
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
got no answer scores 0 and says why.
**Holding updates to a suite.** `model update` and `model promote` take
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
definition added with `ctl model add` — so every update and promotion of it
is held to it, from `ctl`, the console or anywhere:
```
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
llmgw ctl model update qwen-7b # asked the suite first
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
llmgw ctl model gate qwen-7b --off
```
- **Promote** asks the canary the suite before anything moves; below the
score the job fails with the score, and the canary stays, with its share,
until you take it out.
- **Update** runs the new weights first as `MODEL-canary` given none of the
model's requests (hidden from clients like any canary), asks it the suite,
then either promotes it — the model's replicas are replaced one at a time,
as for any update — or takes it out again, leaving the model on its old
weights. That needs room for one more replica while it runs; a model that
already has a canary is promoted or aborted first. Without a suite, an
update swaps the weights in as it always has.
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
kept with the job's id beside it.
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
control plane has, the score as it comes in), **Its quality runs**, and **Hold
its updates to a quality suite…**.
### How many replicas, and how much each one carries
Replica count is changed while the cluster runs, without editing the
@@ -1090,6 +1385,46 @@ same event about the same thing is sent at most once an hour, so a budget that
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
test event to each webhook and says how each answered.
### Tracing
Metrics say that a model took four minutes to start; a trace says where they
went. Give the control plane and every agent the same OpenTelemetry collector,
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
```yaml
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
tracing:
endpoint: http://otel.internal:4318
sample_ratio: 1 # of requests that arrive without a decision of their own
```
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
neither, nothing is traced and nothing is sent: every span is a no-op, and a
command to a node carries no trace context.
What is traced:
| Trace | Spans |
|---|---|
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
| An engine install | `install engine` on the node, a trace of its own. |
So "why did qwen take four minutes to start" is one trace: find `replica start`
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
time went to the download, the launch or the engine loading.
A node follows the control plane's decision: a start the control plane kept is
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
out is not; the agent's ratio decides only for what arrives with no trace, as
from a control plane that is not tracing. The agent needs its own `tracing`
block either way — without one its spans go nowhere, and the control plane's
trace simply ends at `send start`.
## PostgreSQL, when SQLite is no longer the right shape
SQLite is the default and stays the right answer for most installations: one
@@ -1208,6 +1543,68 @@ llmgw keys requests --project Acme/ml/train --off
requests past their hours, or of a project turned off, are removed hourly.
The console has it as **Keep requests…** on a project.
## Reusing answers
A project that asks the same thing again and again — an evaluation run over
the same prompts, embeddings of documents that do not change, an agent that
retries — can have the same request answered from memory rather than by the
model:
```
llmgw keys cache --project Acme/ml/train --ttl 1h # reproducible answers
llmgw keys cache --project Acme/ml/eval --ttl 24h --any # every answer
llmgw keys cache --project Acme/ml/train # how it stands
llmgw keys cache --project Acme/ml/train --off
```
- **What is the same request:** the same project, the same endpoint, the same
model (by its catalog name, whichever alias was used) and the same body,
with `stream`, `stream_options`, `user`, `metadata`, `store` and
`service_tier` left out and the keys in any order. Keys of another project
never see the answer, whatever they send.
- **Which answers are reused:** a chat or completion asked at `temperature`
0 or with a `seed`, and every embedding, rerank and score, which sample
nothing. A request sampled above temperature 0 with no seed asks for a new
answer, and is asked anew — unless the project says `--any`. Only a whole
200 from the model that was asked for is kept: not an error, not a
fallback model's or a canary's answer, not one a guardrail changed or
stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
- **Streams:** an answer given whole is streamed to a client that asks for a
stream, as one chunk per choice with the usage chunk if it asked for that;
a streamed answer is kept put back together, and served whole to a client
that does not stream. Through `/v1/messages` and `/v1/responses` too.
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
`x-llmgw-cache: no`, is always asked anew. Every answer of a project that
reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
can be reused), and a hit says its `Age` in seconds.
- **Policies still apply:** the limits are checked and the guardrails judge
the request before the cache is looked in, and the output policy judges a
reused answer as it would a new one. A request continuing a stored
conversation, or under a `no_store` policy, is never answered from the
cache nor kept in it.
- **What a hit costs:** nothing. It is recorded as a request — status 200,
the model, the key, the rate limit's count — with no tokens and no cost,
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
budget and no token limit. The answer itself carries the usage the model
reported when it was first given.
- **Where it lives:** in each gateway process's memory, at most
`router.cache.max_bytes` (128 MB) of answers for all projects together,
those used longest ago dropped first; nothing is written to the store, and
a restart begins it again. With two control planes behind a balancer each
keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
applies to what is already kept.
```yaml
router:
cache:
max_bytes: 268435456 # 256 MB
max_entry_bytes: 2097152 # 2 MB
```
`llmgw_cache_requests_total{model,result}` counts hits, misses and
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
console has it as **Cache answers…** on a project.
## Batches
Work that can wait — a night's classification, embeddings for a document