Merge features-c2: the docs portal and its links in English and Turkish
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
e8a62b23b7
201 files changed
+26276
-630
No files matched your search
+400
-3
@@ -252,7 +252,8 @@ knowing before you edit it:
|
||||
`LLMGW_ADMIN_TOKEN` gives it from elsewhere. pprof, whose heap holds the API
|
||||
keys requests presented, is off unless `debug_pprof: true`, and then behind
|
||||
the token too. To see the dashboard from elsewhere use `/admin` on the
|
||||
client port — do not move this listener.
|
||||
client port — do not move this listener. The API itself is described in
|
||||
[admin-api.md](admin-api.md).
|
||||
- `default_rpm`, `default_tpm` and `default_price` are what make limits and
|
||||
budgets mean anything. Without a price every request costs zero and a budget
|
||||
never binds; without the defaults, a project created without limits is
|
||||
@@ -383,6 +384,46 @@ llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
|
||||
or from the admin page, or by copying files into the directory by hand — the
|
||||
mirror is a directory, and anything that puts files in it works.
|
||||
|
||||
Files already on the mirror machine go in with `import`, which copies them the
|
||||
way a download is kept — under a temporary name until whole, the checksum
|
||||
beside them for nodes to check against, a version never replaced — and checks
|
||||
that a `.gguf` is one:
|
||||
|
||||
```
|
||||
llmgw mirror import ./qwen2.5-7b-instruct-q4_k_m.gguf --model qwen2.5-7b-instruct \
|
||||
--version 2026-10 --sha256 <the checksum its publisher gave>
|
||||
```
|
||||
|
||||
#### From Ollama
|
||||
|
||||
Models you already run in Ollama are GGUFs llama.cpp serves as they are.
|
||||
On the mirror machine, with Ollama's models readable there:
|
||||
|
||||
```
|
||||
llmgw mirror import-ollama # lists what Ollama has
|
||||
llmgw mirror import-ollama llama3.2:3b
|
||||
```
|
||||
|
||||
It finds Ollama's models directory where Ollama keeps it — `OLLAMA_MODELS`,
|
||||
`~/.ollama/models`, or `/usr/share/ollama/.ollama/models` for Ollama's Linux
|
||||
service, which needs `sudo` to read — or takes `--ollama-dir`. It copies the
|
||||
model's GGUF, and a vision model's projector beside it, each checked against
|
||||
the sha256 Ollama named its file by; a file that does not match is not
|
||||
mirrored, and nothing of the model is. The version is the tag and the start of
|
||||
the digest (`llama3.2/3b-a80c4f17acd5/llama3.2-3b.gguf`), because a tag like
|
||||
`latest` moves in Ollama and a mirror version must not; `--model` and
|
||||
`--version` choose otherwise. It ends with a model definition to save and
|
||||
`llmgw ctl model add -f`, with `--ctx-size` set to the `num_ctx` Ollama ran it
|
||||
with.
|
||||
|
||||
Ollama's prompt template is not carried: llama.cpp renders the chat template
|
||||
inside the GGUF, which for any recent model is the one it was trained on. Its
|
||||
system prompt and adapters are not carried either, and the import says so.
|
||||
Models Ollama keeps as separate tensors rather than one GGUF, or as several
|
||||
weight layers, are refused with a sentence saying so — mirror that model's GGUF
|
||||
from its publisher instead. When the mirror runs beside the control plane, the
|
||||
console's Mirror view does the same with **I**.
|
||||
|
||||
## 4. Preparing a node
|
||||
|
||||
**Do this before onboarding.** A node missing its driver or engine registers
|
||||
@@ -428,6 +469,71 @@ sudo llmgw doctor
|
||||
A `-nvml` build is still published beside the portable one; it reads NVLink
|
||||
from NVML directly, and needs a libc no older than the builder's.
|
||||
|
||||
### MIG, if you want to cut a card into pieces
|
||||
|
||||
An A100, A30, H100, H200 or later card can be split into MIG (Multi-Instance
|
||||
GPU) instances, each with its own memory, its own share of the card's compute
|
||||
and its own UUID. llmgw places a model on an instance as it would on a card,
|
||||
which makes MIG the way to run several small models on one large card without
|
||||
them competing for it.
|
||||
|
||||
**llmgw does not create the instances.** How a card is cut is your decision,
|
||||
made once with `nvidia-smi` on the node; the agent reads what is there when
|
||||
it starts:
|
||||
|
||||
```
|
||||
sudo nvidia-smi -i 0 -mig 1 # MIG mode on card 0; may need a reset
|
||||
sudo nvidia-smi mig -i 0 -lgip # the profiles this card offers
|
||||
sudo nvidia-smi mig -i 0 -cgi 3g.40gb,3g.40gb -C # two instances, each with its compute instance
|
||||
nvidia-smi -L # the instances, with their MIG- UUIDs
|
||||
sudo systemctl restart llmgw-agent # so it registers the new devices
|
||||
```
|
||||
|
||||
`-C` matters: a GPU instance with no compute instance inside it is not a
|
||||
device yet. Turning MIG on takes effect at the card's next reset
|
||||
(`nvidia-smi -i 0 -r` with nothing running on it, or a reboot), and instances
|
||||
do not survive a reboot unless something recreates them — NVIDIA's
|
||||
`mig-parted` or a systemd unit of your own.
|
||||
|
||||
**Classes.** A card in MIG mode is reported as its instances instead of as
|
||||
the whole card, and each instance's class is the card's class and the
|
||||
profile: two `3g.40gb` instances of an A100 80GB are two devices of class
|
||||
`a100-80g-mig-3g.40gb`, an H100's `1g.10gb+me` is `h100-80g-mig-1g.10gb-me`.
|
||||
`llmgw ctl nodes` lists the classes, and a profile names one like any other:
|
||||
|
||||
```yaml
|
||||
profiles:
|
||||
- name: mig
|
||||
engine: vllm
|
||||
gpu_class: a100-80g-mig-3g.40gb
|
||||
```
|
||||
|
||||
**One instance a replica.** A process can open only one MIG device, and
|
||||
instances have no NVLink or peer-to-peer access between them, so a replica on
|
||||
MIG runs on exactly one instance: `tp: 1`, one GPU, one node. A profile on a
|
||||
MIG class asking for more is refused when the model is added, with that
|
||||
reason. A model too large for one instance wants a larger profile or a whole
|
||||
card. `llmgw ctl model plan` and `ctl model serve` plan the same way: an
|
||||
instance is offered for a model it holds, and never several together.
|
||||
|
||||
**How they are numbered and shown.** Placement numbers card 0's instance 2 as
|
||||
device 1002 (1000 + 100 × card + instance), and every view shows it as
|
||||
`0/mig2`. The engine is given the instance by its UUID —
|
||||
`CUDA_VISIBLE_DEVICES=MIG-…` — which works for vLLM and for llama.cpp's CUDA
|
||||
build. llama.cpp's Vulkan build cannot see MIG instances at all.
|
||||
|
||||
**What is measured.** Each instance's memory is its own; temperature, power
|
||||
and wear are its card's, shown on each of its instances. Utilisation is not
|
||||
reported for an instance, because no driver measures it per instance: the
|
||||
console says `n/a on MIG`, and no `llmgw_gpu_utilization_percent` series is
|
||||
published for it rather than a zero.
|
||||
|
||||
`sudo llmgw doctor` lists each card's MIG mode and instances, and warns of a
|
||||
card in MIG mode with none created (nothing can run on it: CUDA will not open
|
||||
the whole card while MIG is on), of GPU instances with no compute instance,
|
||||
and of a mode change waiting for a reset. Such a card is also kept out of
|
||||
placement, with the reason, in `llmgw ctl nodes`.
|
||||
|
||||
### InfiniBand, if you have it
|
||||
|
||||
```
|
||||
@@ -458,6 +564,7 @@ Nothing to do: the agent installs them once the node joins, under
|
||||
| llama.cpp | every node | the release build for its processor and GPU — CUDA 13.3, CUDA 12.8 (x64), or Vulkan for GPUs and drivers the CUDA builds do not cover; the CPU build only where there is no NVIDIA GPU — checked against GitHub's checksum and seen to list the GPU |
|
||||
| vLLM | nodes with an NVIDIA GPU it can run on | a Python environment built by uv from PyPI, about 8 GB |
|
||||
| SGLang | only when you ask for it | the same way as vLLM |
|
||||
| TensorRT-LLM | only nodes where you installed it | not installed by llmgw: see [TensorRT-LLM, which you bring](#tensorrt-llm-which-you-bring) |
|
||||
|
||||
vLLM is resolved as PyPI stood a day after that vLLM was published, so every
|
||||
node gets the same dependencies whenever it is built, and installed with
|
||||
@@ -492,8 +599,8 @@ engines:
|
||||
|
||||
A changed version reaches every node when it next connects. To use an engine
|
||||
you installed yourself on a node, name its path in that node's `agent.yaml`
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`): the agent
|
||||
then installs none of its own beside it.
|
||||
(`engines.llamacpp.bin`, `engines.vllm.bin`, `engines.sglang.python`,
|
||||
`engines.trtllm.python`): the agent then installs none of its own beside it.
|
||||
|
||||
#### Choosing a node's vLLM build
|
||||
|
||||
@@ -544,6 +651,82 @@ which drift between releases:
|
||||
llmgw ctl engine-check vllm --bin /opt/llmgw/engines/vllm/0.30.0-cu13/bin/vllm
|
||||
```
|
||||
|
||||
#### TensorRT-LLM, which you bring
|
||||
|
||||
llmgw does not install TensorRT-LLM. It is a pip package tied to one CUDA, one
|
||||
PyTorch and an MPI the machine has to provide, and NVIDIA's own advice is its
|
||||
container; an installer guessing at all of that would fail on most machines it
|
||||
met, a long way into a large download. Install it on the node yourself — a
|
||||
virtual environment, or NVIDIA's TensorRT-LLM container — and name its
|
||||
interpreter in that node's `agent.yaml`:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
trtllm:
|
||||
python: /opt/trtllm/venv/bin/python # python3 inside NVIDIA's container
|
||||
```
|
||||
|
||||
The node then offers the engine `trtllm` whenever that interpreter has the
|
||||
`tensorrt_llm` package; without the line it offers nothing, whatever is on
|
||||
`PATH`. It needs **TensorRT-LLM 1.3** (1.3.0rc13 or later): 1.2 has no
|
||||
`--served_model_name`, and publishes none of the queue figures the router
|
||||
reads. Check a node's against the flags llmgw builds:
|
||||
|
||||
```
|
||||
llmgw ctl engine-check trtllm --bin /opt/trtllm/venv/bin/python
|
||||
```
|
||||
|
||||
A model for it names a Hugging Face checkpoint, as for vLLM — trtllm-serve
|
||||
downloads a repository itself, or loads one from the mirror:
|
||||
|
||||
```yaml
|
||||
name: llama-3.1-70b-instruct
|
||||
source: meta-llama/Llama-3.1-70B-Instruct
|
||||
profiles:
|
||||
- name: h100x4
|
||||
engine: trtllm
|
||||
gpu_class: h100-80g
|
||||
tp: 4
|
||||
args: ["--max_batch_size", "64", "--free_gpu_memory_fraction", "0.85"]
|
||||
```
|
||||
|
||||
The agent starts `trtllm-serve serve <model> --host … --port … --tp_size …
|
||||
--served_model_name …`, through a few lines of Python of its own that do what
|
||||
trtllm-serve has no flag for:
|
||||
|
||||
- **the key.** trtllm-serve checks none; every path but `/health` is made to
|
||||
demand the one the gateway gives the replica, so the port on the node is not
|
||||
a way around the gateway.
|
||||
- **the metrics.** Its options file turns on `return_perf_metrics` and
|
||||
`enable_iter_perf_stats`, without which `/prometheus/metrics` does not exist
|
||||
and its queue figures do not move. The router reads
|
||||
`trtllm_num_requests_running`, `trtllm_num_requests_waiting` and
|
||||
`trtllm_kv_cache_utilization` there — not `/metrics`, which on trtllm-serve
|
||||
is JSON. They are set as the engine iterates, so an engine gone idle shows
|
||||
its last batch until the next request.
|
||||
- **no usage reports to NVIDIA**: `TRTLLM_NO_USAGE_STATS` and `DO_NOT_TRACK`
|
||||
are set, after the profile's own `env`.
|
||||
|
||||
What it does not do:
|
||||
|
||||
- **span nodes.** A TensorRT-LLM replica larger than one machine is launched
|
||||
under MPI or Slurm from one of them, which the agents, each starting its own
|
||||
node's part, do not do. A profile with `nodes` above 1 is refused; use vLLM
|
||||
or SGLang for one.
|
||||
- **run before Ampere.** Compute capability 8.0 or newer, which is what current
|
||||
TensorRT-LLM builds kernels for.
|
||||
- **take an options file from a profile.** `--config` and
|
||||
`--extra_llm_api_options` are llmgw's; change a setting with `--set
|
||||
PATH=VALUE` instead (`--set kv_cache_config.enable_block_reuse=false`). Also
|
||||
refused, like the other engines' equivalents: `--trust_remote_code`,
|
||||
`--tokenizer`, `--middleware`, `--chat_template`, `--grpc` and
|
||||
`--num_serve_frontends` (both would serve without the key), and the flags the
|
||||
agent sets.
|
||||
|
||||
This engine was written against TensorRT-LLM's documented command line,
|
||||
server and metrics, and tested with stand-ins for them; it has not yet run on a
|
||||
datacenter GPU. Run `engine-check` on the first node before placing a model on it.
|
||||
|
||||
### Weights
|
||||
|
||||
Nothing to do here if you built the mirror in step 3 — the agent fetches what a
|
||||
@@ -911,6 +1094,118 @@ llmgw ctl model canary qwen-7b --abort # or not
|
||||
- **Promote** is the update the canary was a trial of: the model's replicas
|
||||
are replaced one at a time, and `llmgw ctl model rollback` goes back.
|
||||
|
||||
### Quality suites, and holding new weights to one
|
||||
|
||||
A canary says whether new weights serve as fast and as reliably; it does not
|
||||
say whether their answers are as good. A republished quantisation or a chat
|
||||
template that no longer matches can make a model faster and worse at once. A
|
||||
quality suite is a handful of questions with known answers, asked of a model
|
||||
and scored:
|
||||
|
||||
```
|
||||
llmgw ctl eval --suites # what the control plane can run
|
||||
llmgw ctl eval qwen-7b --suite arithmetic # by name, on the control plane
|
||||
llmgw ctl eval qwen-7b --suite ./support.yaml --judge llama-70b --concurrency 8
|
||||
llmgw ctl eval --history qwen-7b # the runs kept
|
||||
llmgw ctl eval --run eval-20261004-101500-1a2b3c4d # one, case by case
|
||||
```
|
||||
|
||||
It prints each case's score and why, then the suite's: the cases' scores
|
||||
weighed, 0 to 1, and whether it reached the suite's `threshold` (or
|
||||
`--min-score`); it exits non-zero below it, so a script can stop on it.
|
||||
|
||||
- **How it is asked:** on the control plane, through the router as the
|
||||
gateway's own requests, the way a replay is — past admission, quotas,
|
||||
accounting and the guardrails, so the score is the weights' and not a
|
||||
tenant's limits'. It reaches the model exactly as a client's request
|
||||
would, a canary or a model loading on demand included. Answers are asked
|
||||
at temperature 0 unless a case says otherwise, and a reasoning model's
|
||||
`<think>` block is not part of its answer.
|
||||
- **Where suites come from:** `--suite` is a file on the machine running
|
||||
`ctl`, sent whole, or the name of one in the control plane's
|
||||
`control.eval_dir` (`/etc/llmgw/eval`). Two are built in and run on small
|
||||
models: `arithmetic` (sums and word problems, each scored by the number in
|
||||
the answer) and `json-format` (answers asked for as JSON of a given shape);
|
||||
their files are in `deploy/eval/`, to copy and change. A file in the
|
||||
directory with the same name replaces the built-in one.
|
||||
- **What is kept:** every run, in the store, with the model, the repository
|
||||
and revision the answering weights came from (when known), the suite's name
|
||||
and sha256, when, the score, and each case's answer (up to 2,000
|
||||
characters), score and reason. A run on a canary is filed under the model
|
||||
it is a canary of. Runs are not pruned.
|
||||
|
||||
A suite is YAML (or JSON lines: one case on each line, and optionally a line
|
||||
without a prompt that holds the suite's own settings):
|
||||
|
||||
```yaml
|
||||
name: support # the file's name unless given
|
||||
threshold: 0.8 # the score to pass at, 0 to 1; 0.8 unless given
|
||||
judge: llama-70b # grades the cases scored by a rubric
|
||||
system: You are a support agent for Acme. # every case's, unless it has one
|
||||
max_tokens: 256 # every case's; 512 unless given
|
||||
cases:
|
||||
- id: refund-window
|
||||
prompt: How many days do customers have to ask for a refund?
|
||||
number: 30 # the last number in the answer, ±tolerance
|
||||
tolerance: 0
|
||||
- prompt: Which plan includes SSO?
|
||||
contains: [Enterprise] # every string must appear
|
||||
ignore_case: true
|
||||
- prompt: Reply with only the word OK.
|
||||
exact: OK # the whole answer, white space around it aside
|
||||
- prompt: What is today's ISO date format?
|
||||
regex: '\d{4}-\d{2}-\d{2}' # matches somewhere in the answer
|
||||
- prompt: Give the order as JSON with "id" and "total".
|
||||
json_schema: # the answer, read as JSON, satisfies it
|
||||
type: object
|
||||
required: [id, total]
|
||||
properties: { id: {type: string}, total: {type: number, minimum: 0} }
|
||||
- messages: # a conversation instead of a prompt
|
||||
- { role: user, content: My card was charged twice. }
|
||||
judge: Apologises, and says the duplicate charge is refunded within 5 days.
|
||||
weight: 3 # counts three times; 1 unless given
|
||||
```
|
||||
|
||||
Each case is scored by exactly one of `exact`, `contains`, `regex`, `number`,
|
||||
`json_schema` or `judge`. A JSON answer in a code fence or inside a sentence is
|
||||
still read. The schema supports `type`, `properties`, `required`,
|
||||
`additionalProperties`, `items`, `enum`, `const`, the numeric and length bounds,
|
||||
`pattern`, `anyOf` and `allOf`; a suite using anything else is refused when it
|
||||
is read rather than passed unchecked. A `judge` case asks another model on the
|
||||
cluster (`judge_model:` on the case, else the suite's `judge:`, else `--judge`)
|
||||
to grade the answer by the rubric, 0 to 1. A case passes at 0.5; a case that
|
||||
got no answer scores 0 and says why.
|
||||
|
||||
**Holding updates to a suite.** `model update` and `model promote` take
|
||||
`--eval SUITE --min-score N` (and `--judge`); or a model carries one — set
|
||||
with `ctl model gate`, or `eval: { suite: arithmetic, min_score: 0.8 }` in a
|
||||
definition added with `ctl model add` — so every update and promotion of it
|
||||
is held to it, from `ctl`, the console or anywhere:
|
||||
|
||||
```
|
||||
llmgw ctl model gate qwen-7b --eval arithmetic --min-score 0.8
|
||||
llmgw ctl model update qwen-7b # asked the suite first
|
||||
llmgw ctl model update qwen-7b --skip-eval # once, without it; the job says so
|
||||
llmgw ctl model gate qwen-7b --off
|
||||
```
|
||||
|
||||
- **Promote** asks the canary the suite before anything moves; below the
|
||||
score the job fails with the score, and the canary stays, with its share,
|
||||
until you take it out.
|
||||
- **Update** runs the new weights first as `MODEL-canary` given none of the
|
||||
model's requests (hidden from clients like any canary), asks it the suite,
|
||||
then either promotes it — the model's replicas are replaced one at a time,
|
||||
as for any update — or takes it out again, leaving the model on its old
|
||||
weights. That needs room for one more replica while it runs; a model that
|
||||
already has a canary is promoted or aborted first. Without a suite, an
|
||||
update swaps the weights in as it always has.
|
||||
- The job's `eval` step says the score (`llmgw ctl jobs JOB`), and the run is
|
||||
kept with the job's id beside it.
|
||||
|
||||
The console has it on a model's menu: **Ask it a quality suite…** (a suite the
|
||||
control plane has, the score as it comes in), **Its quality runs**, and **Hold
|
||||
its updates to a quality suite…**.
|
||||
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
@@ -1090,6 +1385,46 @@ same event about the same thing is sent at most once an hour, so a budget that
|
||||
refuses a thousand requests is one message. `llmgw ctl notify test` sends a
|
||||
test event to each webhook and says how each answered.
|
||||
|
||||
### Tracing
|
||||
|
||||
Metrics say that a model took four minutes to start; a trace says where they
|
||||
went. Give the control plane and every agent the same OpenTelemetry collector,
|
||||
over OTLP/HTTP — Jaeger, Tempo, Honeycomb, anything that takes OTLP:
|
||||
|
||||
```yaml
|
||||
# /etc/llmgw/server.yaml, and the same block in each node's agent.yaml
|
||||
tracing:
|
||||
endpoint: http://otel.internal:4318
|
||||
sample_ratio: 1 # of requests that arrive without a decision of their own
|
||||
```
|
||||
|
||||
`OTEL_EXPORTER_OTLP_ENDPOINT` does the same when the file does not say. With
|
||||
neither, nothing is traced and nothing is sent: every span is a no-op, and a
|
||||
command to a node carries no trace context.
|
||||
|
||||
What is traced:
|
||||
|
||||
| Trace | Spans |
|
||||
|---|---|
|
||||
| An inference request (`llmgw-server`) | `inference /v1/…` with `admission`, `schedule` and an `upstream` per attempt beneath it; a caller's `traceparent` is continued and the engine is sent one. |
|
||||
| A reconcile pass that did something | `reconcile`, and beneath it what it did: `scale up` with `place` (the profile, GPU class, pool, fabric and nodes chosen, or every candidate turned down and why) and `replica start`; `scale down`; `drain replica` and `stop replica`, each with its reason; `evict`, `preempt` with its victims; `reap replica` with the error and when it is tried again; `restart replica`, `restart refused`; `node unreachable`, `node lost`. A pass that found nothing to do is not traced. |
|
||||
| A replica's start | `replica start` stays open from placement until the replica serves, fails or is stopped, with each shard's progress as an event on it and how long it took. A `send start` per shard beneath it is the command to the node, and the node's spans (`llmgw-agent`) continue the same trace: `start shard`, with `fetch weights` (local, cached, downloaded or fetched by another shard, and a `download` with its bytes), `launch engine` (and `wait for rendezvous` on a worker of a multi-node replica) and `wait for engine` (how often the engine was asked, and its last answer if it never came up). |
|
||||
| A stop on a node | `stop shard` and `stop engine` in the trace of the control plane's `stop replica`, saying whether the engine had to be killed. |
|
||||
| An adoption | `adopt`, when a node that registers reports engines already running, with how many nobody claimed. |
|
||||
| A model job | `serve model` or `update model`, with a span per step — `registry`, `mirror`, `check`, `catalog`, `placement`, `smoke test`, `replicas` — and from the placement step a link to the replica's own start. |
|
||||
| An engine install | `install engine` on the node, a trace of its own. |
|
||||
|
||||
So "why did qwen take four minutes to start" is one trace: find `replica start`
|
||||
with `llmgw.model` = qwen, and the node's spans beneath it say whether the
|
||||
time went to the download, the launch or the engine loading.
|
||||
|
||||
A node follows the control plane's decision: a start the control plane kept is
|
||||
traced on the node whatever the agent's own `sample_ratio`, and one it sampled
|
||||
out is not; the agent's ratio decides only for what arrives with no trace, as
|
||||
from a control plane that is not tracing. The agent needs its own `tracing`
|
||||
block either way — without one its spans go nowhere, and the control plane's
|
||||
trace simply ends at `send start`.
|
||||
|
||||
## PostgreSQL, when SQLite is no longer the right shape
|
||||
|
||||
SQLite is the default and stays the right answer for most installations: one
|
||||
@@ -1208,6 +1543,68 @@ llmgw keys requests --project Acme/ml/train --off
|
||||
requests past their hours, or of a project turned off, are removed hourly.
|
||||
The console has it as **Keep requests…** on a project.
|
||||
|
||||
## Reusing answers
|
||||
|
||||
A project that asks the same thing again and again — an evaluation run over
|
||||
the same prompts, embeddings of documents that do not change, an agent that
|
||||
retries — can have the same request answered from memory rather than by the
|
||||
model:
|
||||
|
||||
```
|
||||
llmgw keys cache --project Acme/ml/train --ttl 1h # reproducible answers
|
||||
llmgw keys cache --project Acme/ml/eval --ttl 24h --any # every answer
|
||||
llmgw keys cache --project Acme/ml/train # how it stands
|
||||
llmgw keys cache --project Acme/ml/train --off
|
||||
```
|
||||
|
||||
- **What is the same request:** the same project, the same endpoint, the same
|
||||
model (by its catalog name, whichever alias was used) and the same body,
|
||||
with `stream`, `stream_options`, `user`, `metadata`, `store` and
|
||||
`service_tier` left out and the keys in any order. Keys of another project
|
||||
never see the answer, whatever they send.
|
||||
- **Which answers are reused:** a chat or completion asked at `temperature`
|
||||
0 or with a `seed`, and every embedding, rerank and score, which sample
|
||||
nothing. A request sampled above temperature 0 with no seed asks for a new
|
||||
answer, and is asked anew — unless the project says `--any`. Only a whole
|
||||
200 from the model that was asked for is kept: not an error, not a
|
||||
fallback model's or a canary's answer, not one a guardrail changed or
|
||||
stopped, not one longer than `router.cache.max_entry_bytes` (1 MB).
|
||||
- **Streams:** an answer given whole is streamed to a client that asks for a
|
||||
stream, as one chunk per choice with the usage chunk if it asked for that;
|
||||
a streamed answer is kept put back together, and served whole to a client
|
||||
that does not stream. Through `/v1/messages` and `/v1/responses` too.
|
||||
- **The client decides too:** `Cache-Control: no-cache` (or `no-store`), or
|
||||
`x-llmgw-cache: no`, is always asked anew. Every answer of a project that
|
||||
reuses answers says `x-llmgw-cache: hit`, `miss` or `bypass` (not one that
|
||||
can be reused), and a hit says its `Age` in seconds.
|
||||
- **Policies still apply:** the limits are checked and the guardrails judge
|
||||
the request before the cache is looked in, and the output policy judges a
|
||||
reused answer as it would a new one. A request continuing a stored
|
||||
conversation, or under a `no_store` policy, is never answered from the
|
||||
cache nor kept in it.
|
||||
- **What a hit costs:** nothing. It is recorded as a request — status 200,
|
||||
the model, the key, the rate limit's count — with no tokens and no cost,
|
||||
marked as a cache hit (`llmgw ctl request <id>` says so), so it spends no
|
||||
budget and no token limit. The answer itself carries the usage the model
|
||||
reported when it was first given.
|
||||
- **Where it lives:** in each gateway process's memory, at most
|
||||
`router.cache.max_bytes` (128 MB) of answers for all projects together,
|
||||
those used longest ago dropped first; nothing is written to the store, and
|
||||
a restart begins it again. With two control planes behind a balancer each
|
||||
keeps its own. A change of setting is seen within 30 seconds; a shorter TTL
|
||||
applies to what is already kept.
|
||||
|
||||
```yaml
|
||||
router:
|
||||
cache:
|
||||
max_bytes: 268435456 # 256 MB
|
||||
max_entry_bytes: 2097152 # 2 MB
|
||||
```
|
||||
|
||||
`llmgw_cache_requests_total{model,result}` counts hits, misses and
|
||||
bypasses, `llmgw_cache_bytes` and `llmgw_cache_entries` what is held. The
|
||||
console has it as **Cache answers…** on a project.
|
||||
|
||||
## Batches
|
||||
|
||||
Work that can wait — a night's classification, embeddings for a document
|
||||
|
||||
Reference in new issue
Block a user