deniz/llmgw
1
0 Fork 0
forked from ilker/llmgw
Code Pull Requests Activity
Files
646ff996c794cd1a9677ffd7f84b064d17a616cf
llmgw/docs/reference.md
T

2418 lines
58 KiB
Markdown
Raw Blame History

# Configuration reference
Every setting llmgw reads from a file, with the type it takes and what it is
for.
**Generated from the source.** The descriptions are the comments on the fields
themselves, so this cannot describe a setting that does not exist or miss one
that does — and a test fails when it has drifted. Do not edit it; edit the
comment and run `go generate ./...`.
Where a setting takes another block, the block is listed under its own heading
rather than repeated.
## Every key
| Key | Type | In |
|---|---|---|
| `name` | `string` | [Adapter](#adapter) |
| `source` | `string` | [Adapter](#adapter) |
| `scale` | `float64` | [Adapter](#adapter) |
| `enabled` | `bool` | [Admission](#admission) |
| `bands` | `[]Band` | [Admission](#admission) |
| `default_band` | `string` | [Admission](#admission) |
| `band_header` | `string` | [Admission](#admission) |
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
| `queue_capacity_total` | `int` | [Admission](#admission) |
| `interval` | `time.Duration` | [Admission](#admission) |
| `enabled` | `bool` | [Auth](#auth) |
| `store` | `string` | [Auth](#auth) |
| `cache_ttl` | `time.Duration` | [Auth](#auth) |
| `negative_ttl` | `time.Duration` | [Auth](#auth) |
| `default_rpm` | `int` | [Auth](#auth) |
| `default_tpm` | `int` | [Auth](#auth) |
| `signup_open` | `bool` | [Auth](#auth) |
| `oidc` | `OIDC` | [Auth](#auth) |
| `scim` | `SCIM` | [Auth](#auth) |
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
| `personal` | `Personal` | [Auth](#auth) |
| `pricing` | `map[string]Price` | [Auth](#auth) |
| `default_price` | `Price` | [Auth](#auth) |
| `flush_interval` | `time.Duration` | [Auth](#auth) |
| `batch_size` | `int` | [Auth](#auth) |
| `buffer_size` | `int` | [Auth](#auth) |
| `usage_retention` | `time.Duration` | [Auth](#auth) |
| `tools` | `bool` | [Capabilities](#capabilities) |
| `vision` | `bool` | [Capabilities](#capabilities) |
| `embeddings` | `bool` | [Capabilities](#capabilities) |
| `interval` | `time.Duration` | [Control](#control) |
| `base_backoff` | `time.Duration` | [Control](#control) |
| `max_backoff` | `time.Duration` | [Control](#control) |
| `unreachable_after` | `time.Duration` | [Control](#control) |
| `lost_after` | `time.Duration` | [Control](#control) |
| `port_range` | `[]int` | [Control](#control) |
| `mgmt_interface` | `string` | [Control](#control) |
| `enabled` | `bool` | [Control](#control) |
| `lease_ttl` | `time.Duration` | [Control](#control) |
| `id` | `string` | [Endpoint](#endpoint) |
| `url` | `string` | [Endpoint](#endpoint) |
| `api_key` | `string` | [Endpoint](#endpoint) |
| `models` | `[]string` | [Endpoint](#endpoint) |
| `engine` | `string` | [Endpoint](#endpoint) |
| `pool` | `string` | [Endpoint](#endpoint) |
| `weight` | `float64` | [Endpoint](#endpoint) |
| `priority` | `int` | [Endpoint](#endpoint) |
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
| `url` | `string` | [External](#external) |
| `model` | `string` | [External](#external) |
| `api_key_file` | `string` | [External](#external) |
| `format` | `string` | [GuardModel](#guardmodel) |
| `categories` | `[]string` | [GuardModel](#guardmodel) |
| `model` | `string` | [JudgePolicy](#judgepolicy) |
| `input` | `string` | [JudgePolicy](#judgepolicy) |
| `output` | `string` | [JudgePolicy](#judgepolicy) |
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
| `api` | `string` | [Mirror](#mirror) |
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
| `registry` | `string` | [Mirror](#mirror) |
| `timeout` | `time.Duration` | [Mirror](#mirror) |
| `update_check` | `time.Duration` | [Mirror](#mirror) |
| `name` | `string` | [Model](#model) |
| `aliases` | `[]string` | [Model](#model) |
| `description` | `string` | [Model](#model) |
| `source` | `string` | [Model](#model) |
| `profiles` | `[]Profile` | [Model](#model) |
| `scaling` | `Scaling` | [Model](#model) |
| `class` | `string` | [Model](#model) |
| `owner` | `string` | [Model](#model) |
| `external` | `*External` | [Model](#model) |
| `adapters` | `[]Adapter` | [Model](#model) |
| `fallbacks` | `[]string` | [Model](#model) |
| `routing_profile` | `string` | [Model](#model) |
| `slo` | `SLO` | [Model](#model) |
| `context_length` | `int` | [Model](#model) |
| `max_output` | `int` | [Model](#model) |
| `capabilities` | `Capabilities` | [Model](#model) |
| `role` | `string` | [Model](#model) |
| `guard` | `*GuardModel` | [Model](#model) |
| `origin` | `*Origin` | [Model](#model) |
| `prometheus` | `string` | [Monitoring](#monitoring) |
| `window` | `time.Duration` | [Monitoring](#monitoring) |
| `timeout` | `time.Duration` | [Monitoring](#monitoring) |
| `issuer` | `string` | [OIDC](#oidc) |
| `client_id` | `string` | [OIDC](#oidc) |
| `client_secret_file` | `string` | [OIDC](#oidc) |
| `redirect_url` | `string` | [OIDC](#oidc) |
| `label` | `string` | [OIDC](#oidc) |
| `scopes` | `[]string` | [OIDC](#oidc) |
| `groups_claim` | `string` | [OIDC](#oidc) |
| `create_accounts` | `bool` | [OIDC](#oidc) |
| `roles` | `map[string][]string` | [OIDC](#oidc) |
| `memberships` | `map[string]string` | [OIDC](#oidc) |
| `no_passwords` | `bool` | [OIDC](#oidc) |
| `nvidia_smi` | `string` | [Options](#options) |
| `ib_root` | `string` | [Options](#options) |
| `prefer_nvml` | `bool` | [Options](#options) |
| `skip_gpus` | `bool` | [Options](#options) |
| `repo` | `string` | [Origin](#origin) |
| `revision` | `string` | [Origin](#origin) |
| `files` | `[]OriginFile` | [Origin](#origin) |
| `previous` | `*PreviousOrigin` | [Origin](#origin) |
| `path` | `string` | [OriginFile](#originfile) |
| `sha256` | `string` | [OriginFile](#originfile) |
| `bytes` | `int64` | [OriginFile](#originfile) |
| `name` | `string` | [Pattern](#pattern) |
| `regex` | `string` | [Pattern](#pattern) |
| `action` | `string` | [Pattern](#pattern) |
| `disabled` | `bool` | [Personal](#personal) |
| `rpm` | `int` | [Personal](#personal) |
| `tpm` | `int` | [Personal](#personal) |
| `budget` | `float64` | [Personal](#personal) |
| `period` | `string` | [Personal](#personal) |
| `mode` | `string` | [Policy](#policy) |
| `fail` | `string` | [Policy](#policy) |
| `no_store` | `bool` | [Policy](#policy) |
| `input` | `Rules` | [Policy](#policy) |
| `output` | `Rules` | [Policy](#policy) |
| `tools` | `Tools` | [Policy](#policy) |
| `max_output_tokens` | `int` | [Policy](#policy) |
| `system_prompt` | `string` | [Policy](#policy) |
| `drop_client_system` | `bool` | [Policy](#policy) |
| `require_capabilities` | `bool` | [Policy](#policy) |
| `validate_structured` | `bool` | [Policy](#policy) |
| `judge` | `*JudgePolicy` | [Policy](#policy) |
| `stream` | `string` | [Policy](#policy) |
| `enabled` | `bool` | [Prefix](#prefix) |
| `max_nodes` | `int` | [Prefix](#prefix) |
| `min_match` | `int` | [Prefix](#prefix) |
| `ttl` | `time.Duration` | [Prefix](#prefix) |
| `match_ratio` | `float64` | [Prefix](#prefix) |
| `balance_abs` | `float64` | [Prefix](#prefix) |
| `balance_rel` | `float64` | [Prefix](#prefix) |
| `source` | `string` | [PreviousOrigin](#previousorigin) |
| `revision` | `string` | [PreviousOrigin](#previousorigin) |
| `files` | `[]OriginFile` | [PreviousOrigin](#previousorigin) |
| `name` | `string` | [Profile](#profile) |
| `engine` | `string` | [Profile](#profile) |
| `gpu_class` | `string` | [Profile](#profile) |
| `tp` | `int` | [Profile](#profile) |
| `pp` | `int` | [Profile](#profile) |
| `gpus_per_node` | `int` | [Profile](#profile) |
| `nodes` | `int` | [Profile](#profile) |
| `args` | `[]string` | [Profile](#profile) |
| `env` | `map[string]string` | [Profile](#profile) |
| `min_memory_gb` | `int` | [Profile](#profile) |
| `cpu_cores` | `int` | [Profile](#profile) |
| `min_disk_gb` | `int` | [Profile](#profile) |
| `node_selector` | `map[string]string` | [Profile](#profile) |
| `weight` | `float64` | [Profile](#profile) |
| `priority` | `int` | [Profile](#profile) |
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
| `activation_wait` | `time.Duration` | [Router](#router) |
| `scrape_interval` | `time.Duration` | [Router](#router) |
| `scrape_timeout` | `time.Duration` | [Router](#router) |
| `stale_after` | `time.Duration` | [Router](#router) |
| `retries` | `int` | [Router](#router) |
| `cooldown` | `time.Duration` | [Router](#router) |
| `failure_threshold` | `int` | [Router](#router) |
| `default_profile` | `string` | [Router](#router) |
| `prefix` | `Prefix` | [Router](#router) |
| `session_header` | `string` | [Router](#router) |
| `inject_usage` | `bool` | [Router](#router) |
| `tenant_label` | `string` | [Router](#router) |
| `max_body_bytes` | `int64` | [Router](#router) |
| `trusted_proxies` | `[]string` | [Router](#router) |
| `timeouts` | `Timeouts` | [Router](#router) |
| `scoring` | `Scoring` | [Router](#router) |
| `admission` | `Admission` | [Router](#router) |
| `pii` | `map[string]string` | [Rules](#rules) |
| `injection` | `string` | [Rules](#rules) |
| `deny` | `[]Pattern` | [Rules](#rules) |
| `max_chars` | `int` | [Rules](#rules) |
| `token_file` | `string` | [SCIM](#scim) |
| `python` | `string` | [SGLangConfig](#sglangconfig) |
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
| `min` | `int` | [Scaling](#scaling) |
| `max` | `int` | [Scaling](#scaling) |
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
| `target_waiting` | `float64` | [Scoring](#scoring) |
| `target_inflight` | `float64` | [Scoring](#scoring) |
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
| `session_load_factor` | `float64` | [Scoring](#scoring) |
| `listen` | `string` | [Server](#server) |
| `admin_listen` | `string` | [Server](#server) |
| `metrics_listen` | `string` | [Server](#server) |
| `agent_listen` | `string` | [Server](#server) |
| `auth` | `Auth` | [Server](#server) |
| `threads` | `Threads` | [Server](#server) |
| `landing` | `Landing` | [Server](#server) |
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
| `max_connections` | `int` | [Server](#server) |
| `admin_token_file` | `string` | [Server](#server) |
| `admin_remote` | `bool` | [Server](#server) |
| `debug_pprof` | `bool` | [Server](#server) |
| `agent_insecure` | `bool` | [Server](#server) |
| `monitoring` | `Monitoring` | [Server](#server) |
| `tracing` | `Tracing` | [Server](#server) |
| `mirror` | `Mirror` | [Server](#server) |
| `engines` | `map[string]string` | [Server](#server) |
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
| `backup_dir` | `string` | [Server](#server) |
| `guard` | `guard.Config` | [Server](#server) |
| `firewall_networks` | `[]string` | [Server](#server) |
| `backup_recipients` | `[]string` | [Server](#server) |
| `releases` | `string` | [Server](#server) |
| `router` | `Router` | [Server](#server) |
| `control` | `Control` | [Server](#server) |
| `models` | `[]catalog.Model` | [Server](#server) |
| `endpoints` | `[]Endpoint` | [Server](#server) |
| `bin` | `string` | [SimConfig](#simconfig) |
| `store` | `string` | [Threads](#threads) |
| `retain_hours` | `int` | [Threads](#threads) |
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
| `response_header` | `time.Duration` | [Timeouts](#timeouts) |
| `generation` | `time.Duration` | [Timeouts](#timeouts) |
| `first_token` | `time.Duration` | [Timeouts](#timeouts) |
| `idle` | `time.Duration` | [Timeouts](#timeouts) |
| `total` | `time.Duration` | [Timeouts](#timeouts) |
| `allow` | `[]string` | [Tools](#tools) |
| `deny` | `[]string` | [Tools](#tools) |
| `on_denied` | `string` | [Tools](#tools) |
| `max_calls_per_turn` | `int` | [Tools](#tools) |
| `validate_arguments` | `bool` | [Tools](#tools) |
| `endpoint` | `string` | [Tracing](#tracing) |
| `sample_ratio` | `float64` | [Tracing](#tracing) |
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
| `name` | `string` | [admission.Band](#admissionband) |
| `priority` | `int` | [admission.Band](#admissionband) |
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
| `default_band` | `string` | [admission.Config](#admissionconfig) |
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
| `node_id` | `string` | [agent.Config](#agentconfig) |
| `server` | `string` | [agent.Config](#agentconfig) |
| `servers` | `[]string` | [agent.Config](#agentconfig) |
| `hostname` | `string` | [agent.Config](#agentconfig) |
| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
| `engines_dir` | `string` | [agent.Config](#agentconfig) |
| `weights_cache` | `string` | [agent.Config](#agentconfig) |
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
| `name` | `string` | [config.Band](#configband) |
| `priority` | `int` | [config.Band](#configband) |
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
| `default` | `[]string` | [guard.Config](#guardconfig) |
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
| `run_as` | `string` | [supervisor.Config](#supervisorconfig) |
| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
## Adapter
### `name`
`string`
Name is what clients ask for, like a model name.
### `source`
`string`
Source is the adapter file (GGUF for llama.cpp), fetched by each node
as the model's weights are.
### `scale`
`float64`
Scale is how strongly it is applied; zero takes 1.
## Admission
### `enabled`
`bool`
Enabled turns the layer on. Off, every request goes straight through,
which is the v0.2.0 behaviour.
### `bands`
`[]Band`
### `default_band`
`string`
### `band_header`
`string`
BandHeader lets a client declare its band. Until authentication lands in
v1.0.0 this is trusted, so the port must not be exposed to callers who
should not be able to promote themselves.
### `queue_ttl`
`time.Duration`
### `queue_capacity_per_band`
`int`
### `queue_capacity_total`
`int`
QueueCapacityTotal bounds everything queued across bands. Reached, a
request is not refused while a lower band has any queued: the newest of
the lowest band is turned away instead, with a 503 and Retry-After.
Zero takes half of all the bands' capacities together.
### `interval`
`time.Duration`
## Auth
### `enabled`
`bool`
### `store`
`string`
Store is the SQLite file holding keys, tenancy and usage.
### `cache_ttl`
`time.Duration`
CacheTTL bounds how long a revoked key keeps working, traded against a
database read per key per TTL.
### `negative_ttl`
`time.Duration`
NegativeTTL is how long a rejected key is remembered as rejected, so a
client looping with a bad key cannot become a database read per request.
### `default_rpm`
`int`
DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.
Unlimited used to be the default, and a project created without limits is
the easy path — the "default" project on this cluster still is one. The
first sign of that is the bill or the outage. Zero keeps the old
behaviour for anyone who wants it, but it should be a choice rather than
an omission. A fallback, not a ceiling: an explicit limit at any scope
wins, above or below.
### `default_tpm`
`int`
### `signup_open`
`bool`
SignupOpen serves the signup form. Accounts it creates are pending until
an operator approves them, so this is not a way in — but turning it off
means accounts are made with `llmgw users create` instead.
### `oidc`
`OIDC`
OIDC is sign-in through an identity provider.
Takes a [OIDC](#oidc) block.
### `scim`
`SCIM`
SCIM is account provisioning by the identity provider.
Takes a [SCIM](#scim) block.
### `max_key_ttl`
`time.Duration`
MaxKeyTTL is how long any API key may live; a key asked for with no
expiry is given this. Organizations, teams and projects can set
shorter ones (llmgw keys ttl). Zero is no limit.
### `personal`
`Personal`
Personal is what a person may do on their own, outside any project: make
personal keys and use the chat page personally, within these limits.
Takes a [Personal](#personal) block.
### `pricing`
`map[string]Price`
Pricing is what each model costs, for budgets. A model absent from the
table costs whatever DefaultPrice says, which is nothing unless set.
### `default_price`
`Price`
Takes a [config.Price](#configprice) block.
### `flush_interval`
`time.Duration`
FlushInterval and BatchSize decide when usage records are written.
### `batch_size`
`int`
### `buffer_size`
`int`
BufferSize bounds the queue. Beyond it records are dropped rather than
blocking a request: losing accounting is bad, turning a stalled database
into an outage is worse.
### `usage_retention`
`time.Duration`
UsageRetention is how long the record of each request is kept: who sent
it, which replica answered, how long it queued — what `ctl request`
reads. Zero keeps it for ever. Totals, charts and budgets are read from
five-minute sums kept beside the records, which are not pruned, so this
bounds the table that grows by a row a request without changing any
figure a page shows. At least a day when set.
## Capabilities
### `tools`
`bool`
### `vision`
`bool`
### `embeddings`
`bool`
## Control
### `interval`
`time.Duration`
Interval is how often desired and actual state are compared.
### `base_backoff`
`time.Duration`
BaseBackoff and MaxBackoff bound how fast a failing model is retried.
### `max_backoff`
`time.Duration`
### `unreachable_after`
`time.Duration`
UnreachableAfter and LostAfter stage node expiry. They are deliberately
far apart: a network blip must not tear down replicas that still serve,
and rebuilding a multi-node replica costs tens of minutes of loading.
### `lost_after`
`time.Duration`
### `port_range`
`[]int`
PortRange is the per-node range for engine HTTP and dist-init ports.
### `mgmt_interface`
`string`
MgmtInterface is the ethernet interface NCCL bootstraps over.
### `enabled`
`bool`
Enabled runs the control plane even when configuration declares no
models.
Without it, a deployment whose catalog lives entirely in the database
cannot start: control plane mode is inferred from the config file, so an
empty models block means "router only", the process refuses to start for
want of anything to serve, and the models it would have loaded are never
read. That is a cluster that cannot be managed at runtime because it is
managed at runtime. AgentListen cannot carry this signal — it has a
default, so every deployment would become a control plane.
### `lease_ttl`
`time.Duration`
LeaseTTL is how long this control plane's claim to be the one placing
work stands without being renewed.
Only meaningful with a shared store, where a second control plane can
see the claim. Long enough to survive a slow query or a brief network
stall, short enough that a standby takes over before anyone has finished
reading the alert. Zero takes the default.
## Endpoint
### `id`
`string`
### `url`
`string`
### `api_key`
`string`
APIKey is sent to the engine as a bearer token, for an engine started
with one.
### `models`
`[]string`
### `engine`
`string`
Engine names the inference engine, which decides what its metrics are
called. Empty means vLLM's names.
### `pool`
`string`
### `weight`
`float64`
### `priority`
`int`
## EngineOverride
### `gpu_class`
`string`
GPUClass picks nodes with a GPU of this class ("v100-32g").
### `nodes`
`[]string`
Nodes picks nodes by id.
### `labels`
`map[string]string`
Labels picks nodes carrying all of these.
### `engines`
`map[string]string`
Engines are laid over the cluster's: a version, or "" for none.
## External
### `url`
`string`
URL is the provider's base address, without /v1: https://api.openai.com.
### `model`
`string`
Model is what the provider calls it; requests are rewritten to it.
### `api_key_file`
`string`
APIKeyFile holds the provider's key, on the control plane. A file
rather than the key itself, so the key is in neither the catalog, the
store nor the audit log; empty for a provider that asks for none.
## GuardModel
### `format`
`string`
Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
json ({"safe": false, "categories": ["..."]}).
### `categories`
`[]string`
Categories the model is asked about, for shieldgemma — each a policy
sentence — and the names reported for llamaguard3's S1, S2, ….
## JudgePolicy
### `model`
`string`
Model is a catalog model with role: guard.
### `input`
`string`
Input and Output are what an unsafe verdict does: flag or block.
Empty does not ask the judge about that direction.
### `output`
`string`
### `categories`
`[]string`
Categories, when set, are the only ones that count: Llama Guard's
codes (S1, S10) or the names the model reports.
### `latency_budget`
`time.Duration`
LatencyBudget is how long the judge is waited for before the policy's
fail applies. Zero is 250ms.
## Landing
### `contact`
`string`
Contact is where someone without a key should go — an address, a URL, a
channel name. Rendered as a link when it is one, as text otherwise.
### `notice`
`string`
Notice is an optional line of your own, for whatever the reader needs to
know that this page cannot guess.
## LlamaCppConfig
### `bin`
`string`
Bin is the llama-server binary. It is the server, not the CLI: the
gateway talks OpenAI over HTTP and needs the one that serves it.
Empty means the agent installs llama.cpp itself, at the version the
control plane asks for. Setting it opts out of that.
### `parallel`
`int`
Parallel is how many requests a server handles at once, when a profile
does not say. llama.cpp defaults to one, which would make every second
request wait behind the first no matter what the router decided.
### `threads`
`int`
Threads is how many CPU threads a shard with no GPU computes on. Zero
takes half this machine's CPUs.
Not all of them, which is llama.cpp's own default: two busy servers
that each spin on every CPU fight for them, and on an 8-vCPU node
that made 7B generation fall from 10 tokens a second to under 1 each.
At half, one server alone is about a third slower and two together
keep 5.5 each. A node that only ever runs one CPU replica does better
with all of them; say so here.
## Mirror
### `api`
`string`
API is the mirror's management address, scheme included, for example
https://models.internal:8082. Empty means no mirror is configured.
The management port, not the one nodes fetch bytes from. Those are two
listeners on the mirror precisely so this one can be locked down without
re-certifying every agent's fetcher.
### `tls`
`tlsconf.Config`
TLS authenticates to the mirror's management API, with the same CA the
agents use. Empty leaves the connection in plain HTTP, which is only
acceptable on a network where nothing untrusted can reach that port —
the mirror says as much in its own log when it starts that way.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `registry`
`string`
Registry is the model registry the mirror page browses, empty for the
public Hugging Face. Set it to an internal mirror of the registry where
the control plane cannot reach the internet — or leave the whole mirror
block out, which turns the explorer off rather than giving it a search
box that can only time out.
### `timeout`
`time.Duration`
Timeout bounds a management call. It does not bound a download: the
mirror answers a fetch as soon as it has started one, and the bytes keep
arriving long after this deadline has passed.
### `update_check`
`time.Duration`
UpdateCheck is how often the registry is asked whether the files of
the models served from it have changed there: a repository can publish
new weights under the same name. Only told, never applied — moving a
model to them is `llmgw ctl model update`. 0 never asks.
## Model
### `name`
`string`
### `aliases`
`[]string`
### `description`
`string`
Description is what this model is for, in a line.
Written by whoever added it, because nobody else can say it. A registry
offers a pipeline tag and a wall of README, and neither answers the
question a person picking from a list is asking: which of these nine
names should I send this request to.
It reaches clients through /v1/models, which is the only view most of
them ever see. A gateway serving nine models by name alone makes that
choice by guesswork.
### `source`
`string`
Source is the weights path or a HuggingFace id.
### `profiles`
`[]Profile`
### `scaling`
`Scaling`
Takes a [Scaling](#scaling) block.
### `class`
`string`
Class is how much this model's capacity is worth against another's,
when there is not enough for both: critical, standard or opportunistic.
Empty is standard, or opportunistic for a model loaded on demand. See
EffectiveClass.
### `owner`
`string`
Owner is the part of the tenancy tree whose capacity quota this model
counts against, written org:ID, team:ID or project:ID. Empty is shared:
held by the cluster, counted against nobody.
### `external`
`*External`
External is a model served by someone else — an OpenAI-compatible API
outside the cluster — rather than by replicas placed here. Such a model
has no profiles and nothing is placed for it; the router sends its
requests to the provider. Most useful named in another model's
fallbacks: used only when the cluster has no capacity left. Its
prompts leave the cluster, so which keys may name it is worth an
allowlist.
Takes a [External](#external) block.
### `adapters`
`[]Adapter`
Adapters are LoRA adapters served on this model's replicas, each asked
for by its own name as though it were a model. Every replica loads
every adapter, applying one only to a request that names it.
### `fallbacks`
`[]string`
### `routing_profile`
`string`
RoutingProfile selects how the router scores endpoints for this model.
### `slo`
`SLO`
Takes a [SLO](#slo) block.
### `context_length`
`int`
ContextLength is how many tokens one request may use, prompt and
completion together, and MaxOutput bounds the completion alone. Zero
means unknown, which is what every model was until now.
The gateway knew neither, so an over-long prompt was not refused: it was
admitted, given a replica, and failed at the engine, and a client sizing
its prompts from /v1/models had nothing to read. Worse, the number is not
the one an operator writes in the engine's arguments — llama.cpp divides
--ctx-size across its parallel slots, so a pool of 8192 across four slots
is 2048 a request, and nothing said so.
### `max_output`
`int`
### `capabilities`
`Capabilities`
Capabilities is what this model can do beyond plain completion.
Takes a [Capabilities](#capabilities) block.
### `role`
`string`
Role "guard" is a model that judges the gateway's traffic for the
guardrails rather than serving clients: hidden from /v1/models and the
chat page, refused to any key that names it, critical by default, and
always running. Guard says how its answers are read.
### `guard`
`*GuardModel`
Takes a [GuardModel](#guardmodel) block.
### `origin`
`*Origin`
Origin is the registry repository the weights came from, at which
commit, and each file's checksum there. Written when a model is served
from the registry; for one that was not, worked out from the mirror's
record of where it fetched the file. It is what lets the control plane
say that the repository has published different weights since, and
move the model to them.
Takes a [Origin](#origin) block.
## Monitoring
### `prometheus`
`string`
Prometheus is the server's address, for example
http://monitoring.internal:9090. Empty leaves the charts off the
dashboard rather than showing an error where one should be.
### `window`
`time.Duration`
Window is how far back the charts look. Zero takes an hour.
### `timeout`
`time.Duration`
Timeout bounds one query. It runs while somebody waits for a page, so a
monitoring system that is slow today is skipped rather than waited for.
## OIDC
### `issuer`
`string`
Issuer is the provider's address; its /.well-known/openid-configuration
says the rest.
### `client_id`
`string`
ClientID and the file holding its secret, as registered with the
provider.
### `client_secret_file`
`string`
### `redirect_url`
`string`
RedirectURL is where the provider sends people back:
https://<gateway>/login/oidc/callback. Empty works it out from the
request, which is right unless a proxy changes the address.
### `label`
`string`
Label is what the sign-in button says: "Sign in with <label>".
### `scopes`
`[]string`
Scopes asked for besides openid; empty is email and profile, and
groups when Roles or Memberships use them.
### `groups_claim`
`string`
GroupsClaim names the ID token claim holding groups; empty is
"groups".
### `create_accounts`
`bool`
CreateAccounts makes an account, active, for a person the provider
vouches for and the gateway does not know. Off, only existing accounts
sign in this way.
### `roles`
`map[string][]string`
Roles maps a staff role (viewer, operator, billing, admin) to the
groups that give it. Set at every sign-in: someone taken out of the
group loses the role the next time.
### `memberships`
`map[string]string`
Memberships maps a group to a place in the tree — org:NAME,
team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in.
### `no_passwords`
`bool`
PasswordSignIn off refuses passwords on the sign-in form: people come
in through the provider only. Administrators still can, so a broken
provider does not lock everybody out.
## Options
### `nvidia_smi`
`string`
NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.
Worth configuring on nodes where the driver lives outside the default
paths, which containerized driver installs often do.
### `ib_root`
`string`
IBRoot is the sysfs InfiniBand directory.
### `prefer_nvml`
`bool`
PreferNVML uses the NVML backend when this binary has one. NVLink
topology is read from nvidia-smi topo -m on every build; NVML, where it
is used, is the more exact source of the peers.
### `skip_gpus`
`bool`
SkipGPUs omits GPU discovery. A node without nvidia-smi is already
found to have none, so this is for one whose driver is broken.
## Origin
### `repo`
`string`
Repo is the repository, owner/name.
### `revision`
`string`
Revision is the commit the files were fetched at.
### `files`
`[]OriginFile`
Files are the weights fetched from it, with what the registry
published as each one's sha256. A new commit that leaves them alone —
a README edited — is not an update.
### `previous`
`*PreviousOrigin`
Previous is what the last update replaced, kept so it can be gone back
to while its files are still on the mirror.
Takes a [PreviousOrigin](#previousorigin) block.
## OriginFile
### `path`
`string`
### `sha256`
`string`
### `bytes`
`int64`
## Pattern
### `name`
`string`
### `regex`
`string`
### `action`
`string`
## Personal
### `disabled`
`bool`
Disabled refuses personal keys and personal chat; people then use the
gateway only through projects they are members of.
### `rpm`
`int`
### `tpm`
`int`
### `budget`
`float64`
Budget is in whole units per Period ("day" or "month").
### `period`
`string`
## Policy
### `mode`
`string`
Mode: enforce acts, shadow only records what it would have done, off
does nothing. Empty is enforce.
### `fail`
`string`
Fail: what happens when a policy cannot judge — a judge model down or
over its latency budget. open serves, closed refuses. Empty is open.
### `no_store`
`bool`
NoStore keeps nothing of the request: no stored conversation, no
memory notes, no prefix routing, no fallback outside the cluster.
### `input`
`Rules`
Takes a [Rules](#rules) block.
### `output`
`Rules`
Takes a [Rules](#rules) block.
### `tools`
`Tools`
Takes a [Tools](#tools) block.
### `max_output_tokens`
`int`
MaxOutputTokens is put in a request that asks for no limit; one
asking for more is lowered to it.
### `system_prompt`
`string`
SystemPrompt is put before every request's messages.
### `drop_client_system`
`bool`
DropClientSystem removes the caller's own system and developer
messages, so only the pinned prompt instructs the model.
### `require_capabilities`
`bool`
RequireCapabilities refuses a request offering tools, or sending
images, to a model whose catalog entry does not say it takes them.
### `validate_structured`
`bool`
ValidateStructured checks an answer against the JSON schema the
request asked for in response_format: one that does not fit is
stopped — or, streamed a sentence at a time, flagged, having been
sent already.
### `judge`
`*JudgePolicy`
Judge has a judge model on the cluster read the text too.
Takes a [JudgePolicy](#judgepolicy) block.
### `stream`
`string`
Stream: sentence scans a streamed answer a sentence at a time,
releasing each once it passes; buffer_all holds the whole answer
until it is judged. Empty is sentence.
## Prefix
### `enabled`
`bool`
### `max_nodes`
`int`
MaxNodes bounds the tree, MinMatch is the shortest match worth acting on
in bytes, and TTL is how long an entry is believed — an engine evicts on
its own schedule, so an old record may describe a cache that is gone.
### `min_match`
`int`
### `ttl`
`time.Duration`
### `match_ratio`
`float64`
MatchRatio is the share of the prompt that must match to be worth acting
on. BalanceAbs and BalanceRel are the imbalance guard, and both must trip
before affinity is abandoned.
### `balance_abs`
`float64`
### `balance_rel`
`float64`
## PreviousOrigin
### `source`
`string`
### `revision`
`string`
### `files`
`[]OriginFile`
## Profile
### `name`
`string`
### `engine`
`string`
### `gpu_class`
`string`
### `tp`
`int`
### `pp`
`int`
### `gpus_per_node`
`int`
GPUsPerNode defaults to TP for a single-node profile.
### `nodes`
`int`
Nodes above 1 makes this a multi-node profile, which requires a fabric.
### `args`
`[]string`
### `env`
`map[string]string`
### `min_memory_gb`
`int`
MinMemoryGB is the host RAM this profile needs to run.
The number that matters is the model's working set: the weights, plus the
KV cache for the whole context pool, plus the engine's own overhead. For
a GPU profile that lives in the cards and this is small; for a CPU
profile it is the whole thing, and getting it wrong means an OOM on a
node that reported itself healthy.
Zero means unconstrained, which is every profile that existed before
this. A node that does not report its memory is not excluded either —
an older agent must not have its nodes quietly emptied by an upgrade of
the control plane.
### `cpu_cores`
`int`
CPUCores is the cores a replica of a profile with no GPU is given:
reserved out of its node's CPUs by placement, as memory is, and what
its engine computes on (llama.cpp's --threads).
Reserved because an engine takes every core it is told about, and two
on one node that each think they have all of them spin against each
other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
to under one each. Zero reserves nothing and leaves the thread count to
the node, which is every profile from before this.
### `min_disk_gb`
`int`
MinDiskGB is the free space a node needs before it can hold these
weights.
Only checked when the node does not already have them: a node holding
the file needs no room for it, and requiring the space twice would keep
a model off the one node best placed to run it.
Declared rather than derived, because the size of a model that has never
been fetched anywhere is not knowable from here — and a wrong guess
fails late, halfway through a download, which is the failure this exists
to prevent. Zero means unconstrained, and a node that reports no storage
is not excluded: unknown is not full.
### `node_selector`
`map[string]string`
NodeSelector restricts this profile to nodes carrying every one of these
labels, with these values.
A CPU profile otherwise fits everywhere, because there is no card to be
free — which is right for a 0.5B model and wrong for a 14B one. Nothing
in the inventory says how much memory a node has, so the operator says
which nodes are meant for a model instead of the scheduler inferring it.
Agents have reported labels since the beginning; this is what reads them.
### `weight`
`float64`
Weight is the router's capacity weight. It should come from ctl bench
rather than a guess (plan §6.2.1).
### `priority`
`int`
Priority orders profiles during placement; highest that fits wins.
### `startup_timeout`
`time.Duration`
StartupTimeout bounds model loading. A 70B model across two nodes takes
tens of minutes, so this is per profile rather than global.
## Router
### `activation_wait`
`time.Duration`
ActivationWait is how long a request waits for an on-demand model to
load before the client is told to come back. Zero takes the default.
The one real decision in on-demand loading. Refusing outright makes the
feature useless, since the first request for every model would fail;
waiting without a bound turns a request into a job, and a client that
budgeted thirty seconds ends up holding a connection for ten minutes.
### `scrape_interval`
`time.Duration`
### `scrape_timeout`
`time.Duration`
### `stale_after`
`time.Duration`
StaleAfter is how long an endpoint may go unscraped before being scored
down (plan §7.6).
### `retries`
`int`
Retries is how many *additional* endpoints may be tried after a failure,
and only while nothing has been written to the client (plan §7.7).
### `cooldown`
`time.Duration`
Cooldown is how long a failing endpoint leaves the candidate pool.
### `failure_threshold`
`int`
FailureThreshold is the consecutive failure count that opens a breaker.
1 turns the breaker into a pure cooldown.
### `default_profile`
`string`
DefaultProfile names the routing profile used when a model picks none.
### `prefix`
`Prefix`
Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the
router uses session affinity alone, which is tier 0 and already most of
the win for multi-turn traffic.
Takes a [Prefix](#prefix) block.
### `session_header`
`string`
SessionHeader carries the session id for affinity (plan §7.5 tier 0).
### `inject_usage`
`bool`
InjectUsage asks engines for usage even when the client did not, so token
accounting works regardless of client behaviour.
### `tenant_label`
`string`
TenantLabel is the level of the tree the per-tenant request series are
labelled by — llmgw_tenant_requests_total and its siblings: "org",
"team" or "project", or "none" for no such series. A label per project
on a cluster of thousands is more series than Prometheus wants, and a
team is what an alert is usually about; "team" is the default.
### `max_body_bytes`
`int64`
MaxBodyBytes bounds the request body the router will read.
### `trusted_proxies`
`[]string`
TrustedProxies are the reverse proxies in front of the client port,
as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
are believed; anyone else's are not, since they decide the address
written into the setup scripts and whose attempts are counted where.
### `timeouts`
`Timeouts`
Takes a [Timeouts](#timeouts) block.
### `scoring`
`Scoring`
Takes a [Scoring](#scoring) block.
### `admission`
`Admission`
Takes a [Admission](#admission) block.
## Rules
### `pii`
`map[string]string`
PII maps a kind (email, phone, iban, card, tckn, ip, secret) to what is
done when it is found: flag, redact or block.
### `injection`
`string`
Injection is what is done on signs of prompt injection: flag or block.
### `deny`
`[]Pattern`
Deny are patterns, matched against the text as a model reads it.
### `max_chars`
`int`
MaxChars refuses text longer than this; zero is no limit.
## SCIM
### `token_file`
`string`
TokenFile holds the bearer token the provider sends. Empty turns
provisioning off.
## SGLangConfig
### `python`
`string`
Python is the interpreter of the SGLang virtualenv. SGLang is launched as
a module rather than a console script, which is the documented entry
point and avoids depending on the script being on PATH.
## SLO
### `ttft_p99`
`time.Duration`
### `tpot_p99`
`time.Duration`
## Scaling
### `min`
`int`
### `max`
`int`
### `idle_timeout`
`time.Duration`
IdleTimeout is how long an on-demand model may go unasked-for before its
replicas are stopped. Zero takes the cluster default.
Minutes rather than seconds, always. Stopping a replica throws away the
KV cache prefix routing spent its effort filling, and a model that flaps
in and out loses that entirely — so the saving has to be worth more than
the cache, which it only is over a real idle period.
## Scoring
### `target_waiting`
`float64`
### `target_inflight`
`float64`
### `kv_ceiling`
`float64`
### `session_load_factor`
`float64`
## Server
### `listen`
`string`
Listen is the client-facing OpenAI API address.
### `admin_listen`
`string`
AdminListen serves the dashboard, the management API, health and pprof.
Loopback by default, and it should stay there: pprof can dump the process
heap, and API keys are in that heap in plaintext. Every comment beside
this setting has always said not to expose it, while the default bound
every interface — a default that contradicts its own documentation is a
footgun, so the default now matches the advice.
### `metrics_listen`
`string`
MetricsListen serves Prometheus metrics and nothing else.
Separate from AdminListen because the reason that one is loopback is
pprof, not metrics — and metrics are the one thing on it that something
else genuinely needs to reach. Sharing a listener meant a fleet could
either be monitored or keep its heap private, which is not a choice
anybody should have to make.
Empty means off, and metrics remain available on AdminListen either way,
so nothing that scrapes it today has to change.
### `agent_listen`
`string`
AgentListen is where node agents connect. Separate from the client port
so the fleet-facing surface can be firewalled independently.
### `auth`
`Auth`
Auth turns on API keys on the client-facing port.
Takes a [Auth](#auth) block.
### `threads`
`Threads`
Threads keeps conversations and agent memory without API keys, for a
gateway on one trusted machine. With auth on they live in the auth store,
per project, and this block is ignored.
Takes a [Threads](#threads) block.
### `landing`
`Landing`
Landing configures the page served at "/".
Takes a [Landing](#landing) block.
### `agent_tls`
`tlsconf.Config`
AgentTLS authenticates the fleet-facing port. It deliberately does not
cover the client-facing one: that surface is fronted by a publicly
trusted certificate, while this one is mutual and private.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `max_connections`
`int`
MaxConnections bounds the client port's open connections; past it new
ones wait in the kernel's backlog. 0 is no bound.
### `admin_token_file`
`string`
AdminTokenFile holds the token the admin API's /api and /debug paths
demand. Empty is admin-token in the systemd unit's state directory
(/var/lib/llmgw), or secrets/admin-token beside this file for a server
run by hand. Made, 0600, when the server starts and it does not exist;
`sudo llmgw ctl` and the console read it from there.
### `admin_remote`
`bool`
AdminRemote allows admin_listen on an address other than loopback. The
token still guards the API, but it then crosses the network in clear.
### `debug_pprof`
`bool`
DebugPprof serves Go's profiler on the admin listener, behind the token.
Off by default: a heap profile holds the API keys requests presented.
### `agent_insecure`
`bool`
AgentInsecure lets a control plane run its agent port without
agent_tls: plaintext, and any machine that reaches it may register as
any node — be given replicas, and so be sent users' prompts — or, in
the other direction, a machine in the path may tell every agent what
to run. Refused unless said here, and then the port listens on
loopback unless agent_listen names an address. For one machine, a
test, or a network nothing untrusted reaches.
### `monitoring`
`Monitoring`
Monitoring is where the dashboard reads its history from.
Takes a [Monitoring](#monitoring) block.
### `tracing`
`Tracing`
Tracing sends a trace of each inference request to an OpenTelemetry
collector. Off unless an endpoint is set here or in
OTEL_EXPORTER_OTLP_ENDPOINT.
Takes a [Tracing](#tracing) block.
### `mirror`
`Mirror`
Mirror points the control plane at a model mirror it can administer.
Empty means there is none, and /admin/mirror stays unmounted.
Takes a [Mirror](#mirror) block.
### `engines`
`map[string]string`
Engines are the engines every node installs, and at which version. The
agents install them themselves — the right build for their hardware,
checked against its publisher's checksum and seen to run — so a node is
never trusted to already have what somebody once put there.
Unset keeps DefaultEngines. A version replaces the default; an empty
one leaves that engine to be installed by hand.
### `engine_overrides`
`[]EngineOverride`
EngineOverrides give some nodes different engines from the rest: a
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
all on nodes meant only for llama.cpp. Each picks nodes and lays its
engines over the cluster's, in order, later ones last; an empty version
removes that engine from the nodes it picks.
### `backup_dir`
`string`
BackupDir is where `llmgw server backup` writes, and where the newest
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
Empty means /var/backups/llmgw-server.
### `guard`
`guard.Config`
Guard is the library of guardrail policies: what requests and answers
are checked for, and what is done. Applied to every request through
guard.default, and to parts of the tenancy tree by attaching them
(llmgw ctl guard attach). Empty checks nothing.
Takes a [guard.Config](#guardconfig) block.
### `firewall_networks`
`[]string`
FirewallNetworks are networks, besides those each machine reaches the
cluster on, that llmgw's internal ports answer: a monitoring host on
another subnet, nodes behind a router. Written into every machine's
firewall by `server init`, `node add` and `server add`.
### `backup_recipients`
`[]string`
BackupRecipients encrypt every archive `llmgw server backup` writes, to
whoever holds one of their keys: age recipients (age1…) or SSH public
keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
the certificate authority's key and every secret in the store, and is
copied off the machine; encrypted, the copy is safe wherever it lands.
Empty writes archives unencrypted, readable by root only.
### `releases`
`string`
Releases is where `llmgw ctl upgrade` fetches release binaries from,
for this machine and — through their agents — every node. Empty means
the project's own releases.
### `router`
`Router`
Takes a [Router](#router) block.
### `control`
`Control`
Takes a [Control](#control) block.
### `models`
`[]catalog.Model`
### `endpoints`
`[]Endpoint`
Endpoints is the phase-1 static replica list. It is only consulted when
no model declares a profile, which is what makes a router-only smoke test
against sim-engines possible without a control plane.
## SimConfig
### `bin`
`string`
Bin is the llmgw binary itself; the engine is one of its subcommands.
Empty means whatever started this agent, which is right almost always.
## Threads
### `store`
`string`
Store is the SQLite file. Empty means nothing is kept.
### `retain_hours`
`int`
RetainHours is how long idle conversations and notes are kept. Required
with Store: a store with no expiry keeps everything anyone typed, by
omission.
## Timeouts
### `dial`
`time.Duration`
### `response_header`
`time.Duration`
ResponseHeader bounds the wait for headers on a streaming request. It is
a liveness check there, because headers arrive before generation starts.
### `generation`
`time.Duration`
Generation bounds a non-streaming request, where the engine sends no
headers until the completion is done. Setting response_header alone used
to cap both, so 60s read as a liveness guard and behaved as a limit of
about a hundred tokens on a slow model.
### `first_token`
`time.Duration`
### `idle`
`time.Duration`
### `total`
`time.Duration`
## Tools
### `allow`
`[]string`
Allow, when set, are the only tool names a request may offer and a
model may call.
### `deny`
`[]string`
Deny are tool names never offered or called.
### `on_denied`
`string`
OnDenied is strip (the tool is removed from the request) or block.
Empty is strip.
### `max_calls_per_turn`
`int`
MaxCallsPerTurn stops an answer calling more tools than this.
### `validate_arguments`
`bool`
ValidateArguments checks a call's arguments against the parameters
schema the request declared for the tool.
## Tracing
### `endpoint`
`string`
Endpoint is the collector, for example http://otel.internal:4318.
Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
### `sample_ratio`
`float64`
SampleRatio is the share of requests traced, 0 to 1, when the caller
did not decide it with a sampled traceparent of its own. Zero takes 1.
## VLLMConfig
### `bin`
`string`
Bin is the vllm executable.
### `launch_mode`
`string`
LaunchMode selects the multi-node mechanism: "mp", the default, or
"ray".
mp is what vLLM documents today — every node runs serve with its rank,
the workers headless — and needs nothing but vLLM. ray is the older way,
kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
node and has not been checked against a real cluster.
### `ray_bin`
`string`
RayBin is used when LaunchMode is "ray".
## accounting.Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## admission.Band
### `name`
`string`
### `priority`
`int`
Priority orders bands; higher is served first.
### `saturation_ceiling`
`float64`
SaturationCeiling is the pool saturation above which this band waits.
1.0 means "never hold this band back".
## admission.Config
### `bands`
`[]Band`
### `default_band`
`string`
### `queue_ttl`
`time.Duration`
QueueTTL drops a request that has waited this long. Sending the GPU work
the client has already given up on helps nobody.
### `queue_capacity_per_band`
`int`
QueueCapacityPerBand bounds each band's queue. Rejecting immediately
beats accumulating silently.
### `queue_capacity_total`
`int`
QueueCapacityTotal bounds everything queued, across bands. When it is
reached, a request is not refused while a lower band has requests
queued: the newest of the lowest band is turned away instead
(displacement), since the oldest have waited longest and are nearest
their TTL. Zero takes half of every band's capacity together.
### `interval`
`time.Duration`
Interval is how often the dispatcher re-checks saturation. Releases also
wake it, so this is only the backstop for saturation falling on its own.
## agent.Config
### `node_id`
`string`
NodeID defaults to the hostname. It must equal the common name of the
agent's certificate, so set it only where the two differ.
### `server`
`string`
Server is the control plane, and Servers is the same thing when there is
more than one. Both are accepted: every configuration written before
there could be a second control plane names one address, and none of
them should have to change.
The order is tried in order, and a refusal moves to the next. Only the
leader accepts an agent, so a standby is skipped in the time it takes to
be told so — which is what makes failover a reconnect rather than an
edit to every node's configuration.
### `servers`
`[]string`
### `hostname`
`string`
Hostname defaults to the machine's.
### `mgmt_addr`
`string`
MgmtAddr is the address engines are reached on. It defaults to this
machine's address on the route to the control plane, which is the one
the control plane and the router can reach.
### `labels`
`map[string]string`
Labels are free-form facts about a node for a profile's node_selector.
Nothing needs them: memory, GPUs, engines and disk are measured, and a
label restating one of those goes stale the day the machine changes.
### `mgmt_interface`
`string`
MgmtInterface is the ethernet interface NCCL bootstraps over. It
defaults to the interface holding MgmtAddr, then eth0.
### `fabrics`
`[]inventory.FabricAttachment`
Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
default subnet prefix is the same on every fabric, so nothing can be
inferred from the wire (plan §5.2).
### `heartbeat_interval`
`time.Duration`
### `reconnect_base`
`time.Duration`
ReconnectBase is the first reconnect delay; it backs off from there.
### `reconnect_max`
`time.Duration`
### `adopt_ready_timeout`
`time.Duration`
AdoptReadyTimeout bounds how long an adopted engine is given to answer a
health check before the agent stops waiting on it.
Shorter than a startup timeout on purpose: this engine is already
running, so it is not loading a model, it is either serving or it is not.
### `tls`
`tlsconf.Config`
TLS authenticates both ends of the control-plane stream. Leaving it unset
keeps the plaintext behaviour, which only belongs on a trusted network.
Takes a [tlsconf.Config](#tlsconfconfig) block.
### `model_dirs`
`[]string`
ModelDirs are the directories this node keeps model weights in, beyond
WeightsCache, which is always looked in. Storage differs at every installation,
so llmgw observes it rather than owning it: a Lustre mount, an image
exported over the fabric, local NVMe and a copy made by hand all look
identical from here, and placement only needs to know whether the file
is present.
### `engines_dir`
`string`
EnginesDir is where the agent installs engines, one directory per
build. Defaults to /opt/llmgw/engines.
### `weights_cache`
`string`
WeightsCache is where a remote source is downloaded to. Defaults to the
first entry of ModelDirs, because a node that says where its models live
has already answered this question, and otherwise to
/var/lib/llmgw-weights.
Fetching is the simple arrangement: every node keeps its own copy. The
exotic ones need no code — weights on a shared filer or an image exported
over the fabric are simply present, and present means nothing is fetched.
### `discovery`
`discovery.Options`
Takes a [Options](#options) block.
### `engines`
`engines.Config`
Takes a [engines.Config](#enginesconfig) block.
### `supervisor`
`supervisor.Config`
Takes a [supervisor.Config](#supervisorconfig) block.
## config.Band
### `name`
`string`
### `priority`
`int`
### `saturation_ceiling`
`float64`
## config.Price
### `prompt_per_1k`
`int64`
### `completion_per_1k`
`int64`
## engines.Config
### `disabled`
`[]string`
Disabled are engines this node never installs or advertises, whatever
the control plane asks: vLLM on a node of cards no build of it runs on,
that is meant for llama.cpp alone.
### `vllm`
`VLLMConfig`
Takes a [VLLMConfig](#vllmconfig) block.
### `sglang`
`SGLangConfig`
Takes a [SGLangConfig](#sglangconfig) block.
### `llamacpp`
`LlamaCppConfig`
Takes a [LlamaCppConfig](#llamacppconfig) block.
### `sim`
`SimConfig`
Takes a [SimConfig](#simconfig) block.
## guard.Config
### `default`
`[]string`
### `policies`
`map[string]Policy`
## supervisor.Config
### `work_dir`
`string`
WorkDir holds pid files and per-shard scratch space.
### `log_lines`
`int`
LogLines is how many lines of engine output to retain per shard.
### `stop_timeout`
`time.Duration`
StopTimeout is how long a process gets to exit after SIGTERM before it is
killed. Engines need time to free GPU memory cleanly.
### `health_interval`
`time.Duration`
HealthInterval is how often a starting shard is polled.
### `rendezvous_timeout`
`time.Duration`
RendezvousTimeout bounds the wait for a multi-node rendezvous point to
start listening.
### `alive_grace`
`time.Duration`
AliveGrace is how long a non-serving rank must stay up before it counts
as ready. There is no external signal for a worker, so staying alive is
the only evidence available.
### `run_as`
`string`
RunAs is the user engines run as when the agent is root; empty is
llmgw-engine, "root" keeps them root.
## tlsconf.Config
### `ca`
`string`
CA is the bundle used to verify the other side. In a real deployment this
is whichever authority issued the fleet's certificates.
### `cert`
`string`
Cert and Key are this component's own identity.
### `key`
`string`
### `insecure`
`bool`
Insecure disables TLS entirely. It exists for local runs against
sim-engines; anything using it must stay on a trusted network.
View Git Blame Copy Permalink