llmgw/docs/reference.md

Configuration reference

Every setting llmgw reads from a file, with the type it takes and what it is for.

Generated from the source. The descriptions are the comments on the fields themselves, so this cannot describe a setting that does not exist or miss one that does — and a test fails when it has drifted. Do not edit it; edit the comment and run go generate ./....

Where a setting takes another block, the block is listed under its own heading rather than repeated.

Every key

Key	Type	In
name	string	Adapter
source	string	Adapter
scale	float64	Adapter
owner	string	Adapter
enabled	bool	Admission
bands	[]Band	Admission
default_band	string	Admission
band_header	string	Admission
queue_ttl	time.Duration	Admission
queue_capacity_per_band	int	Admission
queue_capacity_total	int	Admission
interval	time.Duration	Admission
default	map[string]any	AgentSettings
nodes	map[string]map[string]any	AgentSettings
enabled	bool	Auth
store	string	Auth
cache_ttl	time.Duration	Auth
negative_ttl	time.Duration	Auth
default_rpm	int	Auth
default_tpm	int	Auth
signup_open	bool	Auth
oidc	OIDC	Auth
scim	SCIM	Auth
max_key_ttl	time.Duration	Auth
personal	Personal	Auth
pricing	map[string]Price	Auth
default_price	Price	Auth
flush_interval	time.Duration	Auth
batch_size	int	Auth
buffer_size	int	Auth
usage_retention	time.Duration	Auth
max_file_bytes	int64	Batch
max_lines	int	Batch
concurrency	int	Batch
retain_days	int	Batch
max_bytes	int64	Cache
max_entry_bytes	int64	Cache
model	string	Canary
percent	int	Canary
min_requests	int	Canary
max_error_increase	float64	Canary
max_ttft_ratio	float64	Canary
tools	bool	Capabilities
vision	bool	Capabilities
embeddings	bool	Capabilities
images	bool	Capabilities
meshes	bool	Capabilities
workflow	string	ComfyUI
checkpoint	string	ComfyUI
output	string	ComfyUI
nodes	[]CustomNode	ComfyUI
environment	string	ComfyUI
dir	string	ComfyUIConfig
python	string	ComfyUIConfig
interval	time.Duration	Control
base_backoff	time.Duration	Control
max_backoff	time.Duration	Control
unreachable_after	time.Duration	Control
lost_after	time.Duration	Control
port_range	[]int	Control
mgmt_interface	string	Control
timezone	string	Control
eval_dir	string	Control
enabled	bool	Control
lease_ttl	time.Duration	Control
repo	string	CustomNode
commit	string	CustomNode
requirements	[]string	CustomNode
id	string	Endpoint
url	string	Endpoint
api_key	string	Endpoint
models	[]string	Endpoint
engine	string	Endpoint
workflow	string	Endpoint
pool	string	Endpoint
weight	float64	Endpoint
priority	int	Endpoint
gpu_class	string	EngineOverride
nodes	[]string	EngineOverride
labels	map[string]string	EngineOverride
engines	map[string]string	EngineOverride
suite	string	EvalGate
min_score	float64	EvalGate
judge	string	EvalGate
url	string	External
model	string	External
api_key_file	string	External
peer	string	External
api	string	External
workflow	string	External
output	string	External
budgets	[]FacilityBudget	Facility
margin	float64	Facility
gpu_watts	float64	Facility
cpu_core_watts	float64	Facility
node_watts	float64	Facility
measured_fresh	time.Duration	Facility
domain	string	FacilityBudget
cap_watts	float64	FacilityBudget
other_watts	float64	FacilityBudget
name	string	Federation
interval	time.Duration	Federation
enabled	bool	FirewallSettings
format	string	GuardModel
categories	[]string	GuardModel
model	string	JudgePolicy
models	[]string	JudgePolicy
input	string	JudgePolicy
output	string	JudgePolicy
categories	[]string	JudgePolicy
controversial	string	JudgePolicy
latency_budget	time.Duration	JudgePolicy
tokenize_timeout	time.Duration	KVIndex
min_blocks	int	KVIndex
max_blocks	int	KVIndex
contact	string	Landing
notice	string	Landing
language	string	Landing
docs_sign_in	bool	Landing
id	string	License
name	string	License
url	string	License
notice	string	License
commercial	*bool	License
excluded_regions	[]string	License
revenue_cap_usd	int64	License
attribution	string	License
bin	string	LlamaCppConfig
parallel	int	LlamaCppConfig
threads	int	LlamaCppConfig
api	string	Mirror
tls	tlsconf.Config	Mirror
registry	string	Mirror
timeout	time.Duration	Mirror
update_check	time.Duration	Mirror
name	string	Model
aliases	[]string	Model
description	string	Model
source	string	Model
profiles	[]Profile	Model
scaling	Scaling	Model
class	string	Model
owner	string	Model
external	*External	Model
comfyui	*ComfyUI	Model
license	*License	Model
adapters	[]Adapter	Model
max_loaded_adapters	int	Model
fallbacks	[]string	Model
canary	*Canary	Model
eval	*EvalGate	Model
routing_profile	string	Model
slo	SLO	Model
context_length	int	Model
max_output	int	Model
capabilities	Capabilities	Model
query_prefix	string	Model
role	string	Model
guard	*GuardModel	Model
origin	*Origin	Model
prometheus	string	Monitoring
window	time.Duration	Monitoring
timeout	time.Duration	Monitoring
issuer	string	OIDC
client_id	string	OIDC
client_secret_file	string	OIDC
redirect_url	string	OIDC
label	string	OIDC
scopes	[]string	OIDC
groups_claim	string	OIDC
create_accounts	bool	OIDC
roles	map[string][]string	OIDC
memberships	map[string]string	OIDC
no_passwords	bool	OIDC
mirror	string	Offline
checksums	string	Offline
nvidia_smi	string	Options
ib_root	string	Options
prefer_nvml	bool	Options
skip_gpus	bool	Options
repo	string	Origin
revision	string	Origin
files	[]OriginFile	Origin
previous	*PreviousOrigin	Origin
path	string	OriginFile
sha256	string	OriginFile
bytes	int64	OriginFile
name	string	Pattern
regex	string	Pattern
action	string	Pattern
disabled	bool	Personal
rpm	int	Personal
tpm	int	Personal
budget	float64	Personal
period	string	Personal
mode	string	Policy
fail	string	Policy
no_store	bool	Policy
input	Rules	Policy
output	Rules	Policy
tools	Tools	Policy
max_output_tokens	int	Policy
system_prompt	string	Policy
drop_client_system	bool	Policy
require_capabilities	bool	Policy
validate_structured	bool	Policy
judge	*JudgePolicy	Policy
stream	string	Policy
enabled	bool	Prefix
max_nodes	int	Prefix
min_match	int	Prefix
ttl	time.Duration	Prefix
match_ratio	float64	Prefix
balance_abs	float64	Prefix
balance_rel	float64	Prefix
source	string	PreviousOrigin
revision	string	PreviousOrigin
files	[]OriginFile	PreviousOrigin
name	string	Profile
engine	string	Profile
engine_version	string	Profile
gpu_class	string	Profile
source	string	Profile
tp	int	Profile
pp	int	Profile
gpus_per_node	int	Profile
nodes	int	Profile
args	[]string	Profile
env	map[string]string	Profile
min_memory_gb	int	Profile
cpu_cores	int	Profile
min_disk_gb	int	Profile
node_selector	map[string]string	Profile
power_watts	float64	Profile
keep_within	string	Profile
weight	float64	Profile
priority	int	Profile
startup_timeout	time.Duration	Profile
kv_events	bool	Profile
activation_wait	time.Duration	Router
scrape_interval	time.Duration	Router
scrape_timeout	time.Duration	Router
stale_after	time.Duration	Router
retries	int	Router
cooldown	time.Duration	Router
failure_threshold	int	Router
default_profile	string	Router
prefix	Prefix	Router
kv_index	KVIndex	Router
session_header	string	Router
inject_usage	bool	Router
tenant_label	string	Router
tag_labels	[]string	Router
max_body_bytes	int64	Router
batch	Batch	Router
cache	Cache	Router
trusted_proxies	[]string	Router
timeouts	Timeouts	Router
scoring	Scoring	Router
admission	Admission	Router
pii	map[string]string	Rules
injection	string	Rules
deny	[]Pattern	Rules
max_chars	int	Rules
token_file	string	SCIM
python	string	SGLangConfig
ttft_p99	time.Duration	SLO
tpot_p99	time.Duration	SLO
days	[]string	ScaleWindow
from	string	ScaleWindow
to	string	ScaleWindow
min	int	ScaleWindow
max	int	ScaleWindow
min	int	Scaling
max	int	Scaling
idle_timeout	time.Duration	Scaling
schedule	[]ScaleWindow	Scaling
prewarm	time.Duration	Scaling
target_waiting	float64	Scoring
target_inflight	float64	Scoring
kv_ceiling	float64	Scoring
session_load_factor	float64	Scoring
vault	secrets.Vault	Secrets
master_key	string	Secrets
listen	string	Server
admin_listen	string	Server
metrics_listen	string	Server
agent_listen	string	Server
auth	Auth	Server
threads	Threads	Server
landing	Landing	Server
agent_tls	tlsconf.Config	Server
max_connections	int	Server
admin_token_file	string	Server
admin_remote	bool	Server
debug_pprof	bool	Server
agent_insecure	bool	Server
monitoring	Monitoring	Server
tracing	obs.Tracing	Server
mirror	Mirror	Server
offline	Offline	Server
engines	map[string]string	Server
engine_overrides	[]EngineOverride	Server
backup_dir	string	Server
guard	guard.Config	Server
secrets	Secrets	Server
notify	notify.Config	Server
mail	mail.Config	Server
serving	ServingSettings	Server
firewall	FirewallSettings	Server
agents	AgentSettings	Server
firewall_networks	[]string	Server
backup_recipients	[]string	Server
backup_copy_to	[]string	Server
backup_copy_key	string	Server
releases	string	Server
router	Router	Server
control	Control	Server
models	[]catalog.Model	Server
facility	Facility	Server
federation	Federation	Server
endpoints	[]Endpoint	Server
stopped	bool	ServingSettings
bin	string	SimConfig
python	string	TRTLLMConfig
store	string	Threads
retain_hours	int	Threads
dial	time.Duration	Timeouts
response_header	time.Duration	Timeouts
generation	time.Duration	Timeouts
first_token	time.Duration	Timeouts
idle	time.Duration	Timeouts
total	time.Duration	Timeouts
allow	[]string	Tools
deny	[]string	Tools
on_denied	string	Tools
max_calls_per_turn	int	Tools
validate_arguments	bool	Tools
endpoint	string	Tracing
sample_ratio	float64	Tracing
bin	string	VLLMConfig
launch_mode	string	VLLMConfig
ray_bin	string	VLLMConfig
addr	string	Vault
token_file	string	Vault
role_id_file	string	Vault
secret_id_file	string	Vault
ca_file	string	Vault
namespace	string	Vault
url	string	Webhook
format	string	Webhook
events	[]string	Webhook
secret_file	string	Webhook
prompt_per_1k	int64	accounting.Price
completion_per_1k	int64	accounting.Price
per_image	int64	accounting.Price
per_mesh	int64	accounting.Price
name	string	admission.Band
priority	int	admission.Band
saturation_ceiling	float64	admission.Band
queue_ttl	time.Duration	admission.Band
bands	[]Band	admission.Config
default_band	string	admission.Config
queue_ttl	time.Duration	admission.Config
queue_capacity_per_band	int	admission.Config
queue_capacity_total	int	admission.Config
interval	time.Duration	admission.Config
node_id	string	agent.Config
server	string	agent.Config
servers	[]string	agent.Config
hostname	string	agent.Config
mgmt_addr	string	agent.Config
labels	map[string]string	agent.Config
mgmt_interface	string	agent.Config
fabrics	[]inventory.FabricAttachment	agent.Config
heartbeat_interval	time.Duration	agent.Config
reconnect_base	time.Duration	agent.Config
reconnect_max	time.Duration	agent.Config
adopt_ready_timeout	time.Duration	agent.Config
tls	tlsconf.Config	agent.Config
model_dirs	[]string	agent.Config
engines_dir	string	agent.Config
weights_cache	string	agent.Config
discovery	discovery.Options	agent.Config
engines	engines.Config	agent.Config
supervisor	supervisor.Config	agent.Config
tracing	obs.Tracing	agent.Config
name	string	config.Band
priority	int	config.Band
saturation_ceiling	float64	config.Band
queue_ttl	time.Duration	config.Band
prompt_per_1k	int64	config.Price
completion_per_1k	int64	config.Price
per_image	int64	config.Price
per_mesh	int64	config.Price
disabled	[]string	engines.Config
vllm	VLLMConfig	engines.Config
sglang	SGLangConfig	engines.Config
llamacpp	LlamaCppConfig	engines.Config
trtllm	TRTLLMConfig	engines.Config
sim	SimConfig	engines.Config
comfyui	ComfyUIConfig	engines.Config
enabled	*bool	guard.Config
default	[]string	guard.Config
policies	map[string]Policy	guard.Config
smtp	string	mail.Config
from	string	mail.Config
username	string	mail.Config
password_file	string	mail.Config
base_url	string	mail.Config
webhooks	[]Webhook	notify.Config
host	string	notify.Config
work_dir	string	supervisor.Config
log_lines	int	supervisor.Config
stop_timeout	time.Duration	supervisor.Config
health_interval	time.Duration	supervisor.Config
rendezvous_timeout	time.Duration	supervisor.Config
alive_grace	time.Duration	supervisor.Config
run_as	string	supervisor.Config
ca	string	tlsconf.Config
cert	string	tlsconf.Config
key	string	tlsconf.Config
insecure	bool	tlsconf.Config

Adapter

name

string

Name is what clients ask for, like a model name.

source

string

Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT adapter directory (a mirror version ending in "/", a path on the node, or a Hugging Face repository id). Fetched by each node as the model's weights are.

scale

float64

Scale is how strongly it is applied; zero takes 1.

owner

string

Owner is the part of the tenancy tree the adapter belongs to — org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in it see it: to anyone else it is a model that does not exist. An adapter trained on an organisation's documents carries some of what they say in its weights. Empty is every caller the model allows.

Admission

enabled

bool

Enabled turns the layer on. Off, every request goes straight through, which is the v0.2.0 behaviour.

bands

[]Band

default_band

string

band_header

string

BandHeader lets a client declare its band. Until authentication lands in v1.0.0 this is trusted, so the port must not be exposed to callers who should not be able to promote themselves.

queue_ttl

time.Duration

queue_capacity_per_band

int

queue_capacity_total

int

QueueCapacityTotal bounds everything queued across bands. Reached, a request is not refused while a lower band has any queued: the newest of the lowest band is turned away instead, with a 503 and Retry-After. Zero takes half of all the bands' capacities together.

interval

time.Duration

AgentSettings

default

map[string]any

Default is agent.yaml's keys for every node.

nodes

map[string]map[string]any

Nodes are agent.yaml's keys for one node each, by node id, over Default.

Auth

enabled

bool

store

string

Store is the SQLite file holding keys, tenancy and usage.

cache_ttl

time.Duration

CacheTTL bounds how long a revoked key keeps working, traded against a database read per key per TTL.

negative_ttl

time.Duration

NegativeTTL is how long a rejected key is remembered as rejected, so a client looping with a bad key cannot become a database read per request.

default_rpm

int

DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.

Unlimited used to be the default, and a project created without limits is the easy path — the "default" project on this cluster still is one. The first sign of that is the bill or the outage. Zero keeps the old behaviour for anyone who wants it, but it should be a choice rather than an omission. A fallback, not a ceiling: an explicit limit at any scope wins, above or below.

default_tpm

int

signup_open

bool

SignupOpen serves the signup form. Accounts it creates are pending until an operator approves them, so this is not a way in — but turning it off means accounts are made with llmgw users create instead.

oidc

OIDC

OIDC is sign-in through an identity provider.

Takes a OIDC block.

scim

SCIM

SCIM is account provisioning by the identity provider.

Takes a SCIM block.

max_key_ttl

time.Duration

MaxKeyTTL is how long any API key may live; a key asked for with no expiry is given this. Organizations, teams and projects can set shorter ones (llmgw keys ttl). Zero is no limit.

personal

Personal

Personal is what a person may do on their own, outside any project: make personal keys and use the chat page personally, within these limits.

Takes a Personal block.

pricing

map[string]Price

Pricing is what each model costs, for budgets. A model absent from the table costs whatever DefaultPrice says, which is nothing unless set.

default_price

Price

Takes a config.Price block.

flush_interval

time.Duration

FlushInterval and BatchSize decide when usage records are written.

batch_size

int

buffer_size

int

BufferSize bounds the queue. Beyond it records are dropped rather than blocking a request: losing accounting is bad, turning a stalled database into an outage is worse.

usage_retention

time.Duration

UsageRetention is how long the record of each request is kept: who sent it, which replica answered, how long it queued — what ctl request reads. Zero keeps it for ever. Totals, charts and budgets are read from five-minute sums kept beside the records, which are not pruned, so this bounds the table that grows by a row a request without changing any figure a page shows. At least a day when set.

Batch

max_file_bytes

int64

MaxFileBytes bounds one uploaded file; 100 MB unless set.

max_lines

int

MaxLines bounds the requests in one batch; 50 000 unless set.

concurrency

int

Concurrency is how many of one batch's requests are in flight at once; 4 unless set. They queue in the lowest admission band, behind every interactive request.

retain_days

int

RetainDays is how long a batch's answer and error files are kept; 30 unless set. Uploaded files are kept until deleted.

Cache

max_bytes

int64

MaxBytes bounds every project's answers together; 128 MB unless set. Past it, the answers used longest ago go first.

max_entry_bytes

int64

MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB unless set.

Canary

model

string

Model is the candidate: a catalog model, usually this one at newer weights, which clients cannot name themselves.

percent

int

Percent of requests go to it, 0 to 100.

min_requests

int

The candidate is taken out — rolled back — when, over at least MinRequests of its own, its error rate is MaxErrorIncrease above this model's (0.05 is five points), or its mean time to the first token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.

max_error_increase

float64

max_ttft_ratio

float64

Capabilities

tools

bool

vision

bool

embeddings

bool

images

bool

Images is a model that makes pictures, asked for at /v1/images/generations. Implied by an external model whose API is ComfyUI's.

meshes

bool

Meshes is a model that makes 3D models from a picture, asked for as a job at /v1/3d/generations. Implied by a ComfyUI model whose output is meshes.

ComfyUI

workflow

string

Workflow is a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane, in ComfyUI's API format.

checkpoint

string

Checkpoint is the file the workflow's {{checkpoint}} names: the source's own name unless set, which a source that is a set of files must.

output

string

Output is what the workflow makes: pictures (the default), asked for at /v1/images/generations, or meshes, asked for as a job at /v1/3d/generations from a picture.

nodes

[]CustomNode

Nodes are custom nodes the workflow needs, put in each replica's own custom_nodes at the commit named.

environment

string

Environment names the models that may share one ComfyUI replica: every model of an environment agrees on the custom nodes they both name, so one replica can carry all of them and ComfyUI moves between their weights itself. Empty is an environment of the model alone.

ComfyUIConfig

dir

string

Dir is ComfyUI's directory, where main.py is. Readable by the user engines run as.

python

string

Python is the interpreter of its environment; Dir's .venv when unset.

Control

interval

time.Duration

Interval is how often desired and actual state are compared.

base_backoff

time.Duration

BaseBackoff and MaxBackoff bound how fast a failing model is retried.

max_backoff

time.Duration

unreachable_after

time.Duration

UnreachableAfter and LostAfter stage node expiry. They are deliberately far apart: a network blip must not tear down replicas that still serve, and rebuilding a multi-node replica costs tens of minutes of loading.

lost_after

time.Duration

port_range

[]int

PortRange is the per-node range for engine HTTP and dist-init ports.

mgmt_interface

string

MgmtInterface is the ethernet interface NCCL bootstraps over.

timezone

string

Timezone is where models' scaling schedules are read, an IANA name like Europe/Istanbul; empty is the control plane's own.

eval_dir

string

EvalDir is where quality suites are read from by name, for llmgw ctl eval MODEL --suite NAME, a model's eval gate and the console. The suites the binary carries (arithmetic, json-format) are found when it has none of the name; a suite here of the same name replaces one.

enabled

bool

Enabled runs the control plane even when configuration declares no models.

Without it, a deployment whose catalog lives entirely in the database cannot start: control plane mode is inferred from the config file, so an empty models block means "router only", the process refuses to start for want of anything to serve, and the models it would have loaded are never read. That is a cluster that cannot be managed at runtime because it is managed at runtime. AgentListen cannot carry this signal — it has a default, so every deployment would become a control plane.

lease_ttl

time.Duration

LeaseTTL is how long this control plane's claim to be the one placing work stands without being renewed.

Only meaningful with a shared store, where a second control plane can see the claim. Long enough to survive a slow query or a brief network stall, short enough that a standby takes over before anyone has finished reading the alert. Zero takes the default.

CustomNode

repo

string

Repo is the repository, https only.

commit

string

Commit is the full commit the node is used at: a branch moves, and a node that changed under a model is a model that changed unasked.

requirements

[]string

Requirements are the PyPI packages the node needs, by name and version range only. They are installed beside the replica, never into ComfyUI's environment: a package named here at the range named, and every other one at the version ComfyUI already runs. One named here is first on ComfyUI's path, so its range must suit ComfyUI too.

Endpoint

id

string

url

string

api_key

string

APIKey is sent to the engine as a bearer token, for an engine started with one.

models

[]string

engine

string

Engine names the inference engine, which decides what its metrics are called. Empty means vLLM's names.

workflow

string

Workflow is the ComfyUI workflow for engine: comfyui, a built-in's name or its JSON; flux-schnell unless set.

pool

string

weight

float64

priority

int

EngineOverride

gpu_class

string

GPUClass picks nodes with a GPU of this class ("v100-32g").

nodes

[]string

Nodes picks nodes by id.

labels

map[string]string

Labels picks nodes carrying all of these.

engines

map[string]string

Engines are laid over the cluster's: a version, or "" for none.

EvalGate

suite

string

Suite is a suite's name in control.eval_dir, or one the binary carries (arithmetic, json-format).

min_score

float64

MinScore is the score, 0 to 1, to pass at; zero is the suite's own threshold.

judge

string

Judge grades the suite's rubric cases, instead of the judge it names.

External

url

string

URL is the provider's base address, without /v1: https://api.openai.com.

model

string

Model is what the provider calls it; requests are rewritten to it.

api_key_file

string

APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none.

peer

string

Peer names the federated llmgw cluster this model is offered by, for a model that federation added (llmgw ctl federation add) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up.

api

string

API is what the provider speaks: OpenAI's, unless this says comfyui — a ComfyUI server, asked for pictures at /v1/images/generations and sent the workflow below with the request's prompt in it. Model is then the checkpoint file the workflow loads, as ComfyUI lists it.

workflow

string

Workflow is the ComfyUI workflow, in its API format: a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane.

output

string

Output is what it makes: pictures, or meshes (see ComfyUI.Output).

Facility

budgets

[]FacilityBudget

Budgets are the domains with a power cap.

margin

float64

Margin is the share of every cap kept free, 0.1 for ten percent. Zero keeps none, for caps already given derated.

gpu_watts

float64

GPUWatts is a card's draw when it does not report its power limit: an agent without nvidia-smi's power fields, or a node that has not sent a heartbeat yet. Zero counts such a card as drawing nothing.

cpu_core_watts

float64

CPUCoreWatts is the draw of a core reserved by a replica with no GPU (cpu_cores). Zero counts such replicas as drawing nothing.

node_watts

float64

NodeWatts is what a node draws besides its cards and reserved cores — idle CPUs, memory, fans, disks — counted for every node in a capped domain, busy or not.

measured_fresh

time.Duration

MeasuredFresh is how old a node's power reading may be and still be counted against its domains. Zero takes two minutes.

FacilityBudget

domain

string

Domain is level:name, as the nodes' labels give it: rack:r12-07, pdu:r12-07-a, row:12.

cap_watts

float64

CapWatts is what the domain may draw. For a dual-fed rack it is what one feed carries alone, derated as the facility derates it (usually 80%), since that is what the rack has to survive.

other_watts

float64

OtherWatts is drawn in the domain by what llmgw does not place on: switches, storage, machines outside the cluster.

Federation

name

string

Name is this cluster as a peer reads it, in the federated_from tag sent with a request to a peer set to receive tags (ctl federation add --send-tags). Empty is this machine's host name.

interval

time.Duration

Interval is how often each peer is asked whether it is ready, timed, and what it offers. A peer that does not answer is taken out of routing until it does. Zero is 10s.

FirewallSettings

enabled

bool

Enabled keeps llmgw's internal ports — engines, node exporter, the agents' stream, the mirror, the store — to the cluster's networks on every machine. False removes llmgw's nftables table everywhere, for a site whose own firewall does that job; the rules are kept, and true puts them back. Changed in the cluster's settings, it is in force on every node and control plane within seconds.

GuardModel

format

string

Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or "Controversial", and the categories), llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({"safe": false, "categories": ["..."]}).

categories

[]string

Categories the model is asked about, for shieldgemma — each a policy sentence — and the names reported for llamaguard3's S1, S2, ….

JudgePolicy

model

string

Model is a catalog model with role: guard.

models

[]string

Models are judges asked in turn, instead of Model: the first that answers gives the verdict. A judge on a GPU first and one on CPUs after it is the GPU's speed while it is there, and the CPUs' when it is not — busy, failed, or the node down.

input

string

Input and Output are what an unsafe verdict does: flag or block. Empty does not ask the judge about that direction.

output

string

categories

[]string

Categories, when set, are the only ones that count: Llama Guard's codes (S1, S10) or the names the model reports.

controversial

string

Controversial is what Qwen3Guard's third verdict counts as: safe (the default) or unsafe.

latency_budget

time.Duration

LatencyBudget is how long each judge is waited for before the next is asked, or, after the last, the policy's fail applies. Zero is 250ms.

KVIndex

tokenize_timeout

time.Duration

TokenizeTimeout bounds the call that turns a request into token ids, made to one of the model's replicas before the request is placed. Past it the request is routed as tier 1 would. Zero takes 250ms.

min_blocks

int

MinBlocks is the fewest leading blocks a replica must hold to be preferred for them; fewer is coincidence — a chat template's opening tokens — rather than a prefix worth following. Zero takes 4, which is 64 tokens at vLLM's usual block size of 16.

max_blocks

int

MaxBlocks bounds how many of a prompt's blocks are hashed and looked up. Beyond it a longer match still helps, but the difference no longer decides anything. Zero takes 4096.

Landing

contact

string

Contact is where someone without a key should go — an address, a URL, a channel name. Rendered as a link when it is one, as text otherwise.

notice

string

Notice is an optional line of your own, for whatever the reader needs to know that this page cannot guess.

language

string

Language is what the web portal speaks to someone who has not chosen a language with the switcher at the foot of every page and whose browser asks for none it speaks: "en" or "tr". Empty is English.

docs_sign_in

bool

DocsSignIn serves the docs portal at /docs only to someone signed in. Off, the portal is public like this page: its client guide is for the person who has no account yet, and the operator documents describe llmgw in general — no address, model or key of this installation — which the project's repository publishes anyway. On, a visitor is sent to sign in, and without accounts the portal needs an API key.

License

id

string

ID names the licence for a machine: an SPDX identifier where there is one (MIT, Apache-2.0, CC-BY-NC-4.0), else a lowercase name of the licence's own (stability-community, tencent-hunyuan-community). What a person's or a site's choice of licences matches on. Name is for people.

name

string

Name is the licence's own: MIT, Apache-2.0.

url

string

notice

string

Notice is what must accompany what was made: attribution, terms on the output, what the weights were trained on.

commercial

*bool

Commercial says whether what is made may be used commercially: nil is not said, which is not the same as yes.

excluded_regions

[]string

ExcludedRegions are where the licence grants nothing: ISO 3166 codes, and EU for the union (Hunyuan's: EU, UK, KR).

revenue_cap_usd

int64

RevenueCapUSD is the yearly revenue above which the licence asks for another agreement (Stability's community licence: 1000000). Zero is none.

attribution

string

Attribution is what a product using the output must show: "Powered by Stability AI".

LlamaCppConfig

bin

string

Bin is the llama-server binary. It is the server, not the CLI: the gateway talks OpenAI over HTTP and needs the one that serves it. Empty means the agent installs llama.cpp itself, at the version the control plane asks for. Setting it opts out of that.

parallel

int

Parallel is how many requests a server handles at once, when a profile does not say. llama.cpp defaults to one, which would make every second request wait behind the first no matter what the router decided.

threads

int

Threads is how many CPU threads a shard with no GPU computes on. Zero takes half this machine's CPUs.

Not all of them, which is llama.cpp's own default: two busy servers that each spin on every CPU fight for them, and on an 8-vCPU node that made 7B generation fall from 10 tokens a second to under 1 each. At half, one server alone is about a third slower and two together keep 5.5 each. A node that only ever runs one CPU replica does better with all of them; say so here.

Mirror

api

string

API is the mirror's management address, scheme included, for example https://models.internal:8082. Empty means no mirror is configured.

The management port, not the one nodes fetch bytes from. Those are two listeners on the mirror precisely so this one can be locked down without re-certifying every agent's fetcher.

tls

tlsconf.Config

TLS authenticates to the mirror's management API, with the same CA the agents use. Empty leaves the connection in plain HTTP, which is only acceptable on a network where nothing untrusted can reach that port — the mirror says as much in its own log when it starts that way.

Takes a tlsconf.Config block.

registry

string

Registry is the model registry the mirror page browses, empty for the public Hugging Face. Set it to an internal mirror of the registry where the control plane cannot reach the internet — or leave the whole mirror block out, which turns the explorer off rather than giving it a search box that can only time out.

timeout

time.Duration

Timeout bounds a management call. It does not bound a download: the mirror answers a fetch as soon as it has started one, and the bytes keep arriving long after this deadline has passed.

update_check

time.Duration

UpdateCheck is how often the registry is asked whether the files of the models served from it have changed there: a repository can publish new weights under the same name. Only told, never applied — moving a model to them is llmgw ctl model update. 0 never asks.

Model

name

string

aliases

[]string

description

string

Description is what this model is for, in a line.

Written by whoever added it, because nobody else can say it. A registry offers a pipeline tag and a wall of README, and neither answers the question a person picking from a list is asking: which of these nine names should I send this request to.

It reaches clients through /v1/models, which is the only view most of them ever see. A gateway serving nine models by name alone makes that choice by guesswork.

source

string

Source is the weights path or a HuggingFace id.

profiles

[]Profile

scaling

Scaling

Takes a Scaling block.

class

string

Class is how much this model's capacity is worth against another's, when there is not enough for both: critical, standard or opportunistic. Empty is standard, or opportunistic for a model loaded on demand. See EffectiveClass.

owner

string

Owner is the part of the tenancy tree whose capacity quota this model counts against, written org:ID, team:ID or project:ID. Empty is shared: held by the cluster, counted against nobody.

external

*External

External is a model served by someone else — an OpenAI-compatible API outside the cluster — rather than by replicas placed here. Such a model has no profiles and nothing is placed for it; the router sends its requests to the provider. Most useful named in another model's fallbacks: used only when the cluster has no capacity left. Its prompts leave the cluster, so which keys may name it is worth an allowlist.

Takes a External block.

comfyui

*ComfyUI

ComfyUI is how a model whose profiles run engine comfyui makes its pictures: the workflow, and the file it loads.

Takes a ComfyUI block.

license

*License

License is the terms of what the model makes, for a model whose output somebody ships: shown with every result and written into a mesh.

Takes a License block.

adapters

[]Adapter

Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it — unless MaxLoadedAdapters is set.

max_loaded_adapters

int

MaxLoadedAdapters, above zero, has each replica hold at most this many adapters at once, loaded when a request first asks for one and unloaded least recently used first to make room. The router sends a request for an adapter to a replica that already holds it while that one has room to take it. vLLM profiles only: llama-server loads adapters at start and cannot load one afterwards. Zero loads every adapter on every replica at start.

fallbacks

[]string

canary

*Canary

Canary sends a share of this model's requests to another catalog model — the same model at new weights — to compare the two before moving to it (llmgw ctl model canary).

Takes a Canary block.

eval

*EvalGate

Eval is a quality suite new weights must pass before they replace this model's: llmgw ctl model update and model promote run it against them first, and stop below its score.

Takes a EvalGate block.

routing_profile

string

RoutingProfile selects how the router scores endpoints for this model.

slo

SLO

Takes a SLO block.

context_length

int

ContextLength is how many tokens one request may use, prompt and completion together, and MaxOutput bounds the completion alone. Zero means unknown, which is what every model was until now.

The gateway knew neither, so an over-long prompt was not refused: it was admitted, given a replica, and failed at the engine, and a client sizing its prompts from /v1/models had nothing to read. Worse, the number is not the one an operator writes in the engine's arguments — llama.cpp divides --ctx-size across its parallel slots, so a pool of 8192 across four slots is 2048 a request, and nothing said so.

max_output

int

capabilities

Capabilities

Capabilities is what this model can do beyond plain completion.

Takes a Capabilities block.

query_prefix

string

QueryPrefix is put before a question an embedding model embeds for a knowledge-base search; documents are embedded without it. Models trained with an instruction on the query side want it, Qwen3-Embedding "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: " for one.

role

string

Role "guard" is a model that judges the gateway's traffic for the guardrails rather than serving clients: hidden from /v1/models and the chat page, refused to any key that names it, critical by default, and always running. Guard says how its answers are read.

guard

*GuardModel

Takes a GuardModel block.

origin

*Origin

Origin is the registry repository the weights came from, at which commit, and each file's checksum there. Written when a model is served from the registry; for one that was not, worked out from the mirror's record of where it fetched the file. It is what lets the control plane say that the repository has published different weights since, and move the model to them.

Takes a Origin block.

Monitoring

prometheus

string

Prometheus is the server's address, for example http://monitoring.internal:9090. Empty leaves the charts off the dashboard rather than showing an error where one should be.

window

time.Duration

Window is how far back the charts look. Zero takes an hour.

timeout

time.Duration

Timeout bounds one query. It runs while somebody waits for a page, so a monitoring system that is slow today is skipped rather than waited for.

OIDC

issuer

string

Issuer is the provider's address; its /.well-known/openid-configuration says the rest.

client_id

string

ClientID and the file holding its secret, as registered with the provider.

client_secret_file

string

redirect_url

string

RedirectURL is where the provider sends people back: https:///login/oidc/callback. Empty works it out from the request, which is right unless a proxy changes the address.

label

string

Label is what the sign-in button says: "Sign in with ".

scopes

[]string

Scopes asked for besides openid; empty is email and profile, and groups when Roles or Memberships use them.

groups_claim

string

GroupsClaim names the ID token claim holding groups; empty is "groups".

create_accounts

bool

CreateAccounts makes an account, active, for a person the provider vouches for and the gateway does not know. Off, only existing accounts sign in this way.

roles

map[string][]string

Roles maps a staff role (viewer, operator, billing, admin) to the groups that give it. Set at every sign-in: someone taken out of the group loses the role the next time.

memberships

map[string]string

Memberships maps a group to a place in the tree — org:NAME, team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in.

no_passwords

bool

PasswordSignIn off refuses passwords on the sign-in form: people come in through the provider only. Administrators still can, so a broken provider does not lock everybody out.

Offline

mirror

string

Mirror is the mirror's file address, as nodes reach it: http://10.0.0.5:8081 — the port model files are served on, not the management API. Set, the cluster is offline: nodes install engines from it and never from their publishers, releases come from it unless releases names another place, and no registry is asked whether a model has newer weights. Python engines (vLLM, SGLang) come from PyPI, which no bundle carries; on an offline node they are installed by hand.

checksums

string

Checksums is the file of engine checksums this control plane trusts, kept by llmgw bundle import and llmgw bundle trust; every engine file a node fetches from the mirror is checked against it. Read again whenever it changes. Empty is /etc/llmgw/offline.sha256.

Options

nvidia_smi

string

NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH. Worth configuring on nodes where the driver lives outside the default paths, which containerized driver installs often do.

ib_root

string

IBRoot is the sysfs InfiniBand directory.

prefer_nvml

bool

PreferNVML uses the NVML backend when this binary has one. NVLink topology is read from nvidia-smi topo -m on every build; NVML, where it is used, is the more exact source of the peers.

skip_gpus

bool

SkipGPUs omits GPU discovery. A node without nvidia-smi is already found to have none, so this is for one whose driver is broken.

Origin

repo

string

Repo is the repository, owner/name.

revision

string

Revision is the commit the files were fetched at.

files

[]OriginFile

Files are the weights fetched from it, with what the registry published as each one's sha256. A new commit that leaves them alone — a README edited — is not an update.

previous

*PreviousOrigin

Previous is what the last update replaced, kept so it can be gone back to while its files are still on the mirror.

Takes a PreviousOrigin block.

OriginFile

path

string

sha256

string

bytes

int64

Pattern

name

string

regex

string

action

string

Personal

disabled

bool

Disabled refuses personal keys and personal chat; people then use the gateway only through projects they are members of.

rpm

int

tpm

int

budget

float64

Budget is in whole units per Period ("day" or "month").

period

string

Policy

mode

string

Mode: enforce acts, shadow only records what it would have done, off does nothing. Empty is enforce.

fail

string

Fail: what happens when a policy cannot judge — a judge model down or over its latency budget. open serves, closed refuses. Empty is open.

no_store

bool

NoStore keeps nothing of the request: no stored conversation, no memory notes, no prefix routing, no fallback outside the cluster.

input

Rules

Takes a Rules block.

output

Rules

Takes a Rules block.

tools

Tools

Takes a Tools block.

max_output_tokens

int

MaxOutputTokens is put in a request that asks for no limit; one asking for more is lowered to it.

system_prompt

string

SystemPrompt is put before every request's messages.

drop_client_system

bool

DropClientSystem removes the caller's own system and developer messages, so only the pinned prompt instructs the model.

require_capabilities

bool

RequireCapabilities refuses a request offering tools, or sending images, to a model whose catalog entry does not say it takes them.

validate_structured

bool

ValidateStructured checks an answer against the JSON schema the request asked for in response_format: one that does not fit is stopped — or, streamed a sentence at a time, flagged, having been sent already.

judge

*JudgePolicy

Judge has a judge model on the cluster read the text too.

Takes a JudgePolicy block.

stream

string

Stream: sentence scans a streamed answer a sentence at a time, releasing each once it passes; buffer_all holds the whole answer until it is judged. Empty is sentence.

Prefix

enabled

bool

max_nodes

int

MaxNodes bounds the tree, MinMatch is the shortest match worth acting on in bytes, and TTL is how long an entry is believed — an engine evicts on its own schedule, so an old record may describe a cache that is gone.

min_match

int

ttl

time.Duration

match_ratio

float64

MatchRatio is the share of the prompt that must match to be worth acting on. BalanceAbs and BalanceRel are the imbalance guard, and both must trip before affinity is abandoned.

balance_abs

float64

balance_rel

float64

PreviousOrigin

source

string

revision

string

files

[]OriginFile

Profile

name

string

engine

string

engine_version

string

EngineVersion restricts the profile to nodes whose installed engine's version starts with it: "0.11" takes vLLM 0.11.0 and 0.11.1, not 0.10.2. Empty takes whatever version the node has.

gpu_class

string

source

string

Source is the weights this profile loads, when not the model's: an embedding model served by vLLM from its repository on a card and by llama.cpp from a GGUF on CPUs is one model with two sources. Empty loads the model's. Updates and canaries move the model's source; a profile with its own keeps it.

tp

int

pp

int

gpus_per_node

int

GPUsPerNode defaults to TP for a single-node profile.

nodes

int

Nodes above 1 makes this a multi-node profile, which requires a fabric.

args

[]string

env

map[string]string

min_memory_gb

int

MinMemoryGB is the host RAM this profile needs to run.

The number that matters is the model's working set: the weights, plus the KV cache for the whole context pool, plus the engine's own overhead. For a GPU profile that lives in the cards and this is small; for a CPU profile it is the whole thing, and getting it wrong means an OOM on a node that reported itself healthy.

Zero means unconstrained, which is every profile that existed before this. A node that does not report its memory is not excluded either — an older agent must not have its nodes quietly emptied by an upgrade of the control plane.

cpu_cores

int

CPUCores is the cores a replica of a profile with no GPU is given: reserved out of its node's CPUs by placement, as memory is, and what its engine computes on (llama.cpp's --threads).

Reserved because an engine takes every core it is told about, and two on one node that each think they have all of them spin against each other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second to under one each. Zero reserves nothing and leaves the thread count to the node, which is every profile from before this.

min_disk_gb

int

MinDiskGB is the free space a node needs before it can hold these weights.

Only checked when the node does not already have them: a node holding the file needs no room for it, and requiring the space twice would keep a model off the one node best placed to run it.

Declared rather than derived, because the size of a model that has never been fetched anywhere is not knowable from here — and a wrong guess fails late, halfway through a download, which is the failure this exists to prevent. Zero means unconstrained, and a node that reports no storage is not excluded: unknown is not full.

node_selector

map[string]string

NodeSelector restricts this profile to nodes carrying every one of these labels, with these values.

A CPU profile otherwise fits everywhere, because there is no card to be free — which is right for a 0.5B model and wrong for a 14B one. Nothing in the inventory says how much memory a node has, so the operator says which nodes are meant for a model instead of the scheduler inferring it. Agents have reported labels since the beginning; this is what reads them.

power_watts

float64

PowerWatts is what one replica of this profile draws, in watts, split evenly over its nodes, counted against the power cap of every rack, PDU or other domain its nodes are in (facility.budgets in server.yaml). Zero derives it: each card at the power limit it reports, each reserved core at facility.cpu_core_watts. Only matters where a domain has a cap.

keep_within

string

KeepWithin keeps a multi-node replica's nodes inside one rack, row, room or site, read from their llmgw.io/rack (and so on) labels: for a fabric whose leaves do not reach across racks, or cables that do not. Nodes without the label are passed over for such a profile. Empty is no constraint.

weight

float64

Weight is the router's capacity weight. It should come from ctl bench rather than a guess (plan §6.2.1).

priority

int

Priority orders profiles during placement; highest that fits wins.

startup_timeout

time.Duration

StartupTimeout bounds model loading. A 70B model across two nodes takes tens of minutes, so this is per profile rather than global.

kv_events

bool

KVEvents has vLLM publish what its prefix cache holds, and the router route this model's requests by it: to the replica holding the most of a prompt's leading blocks (plan §7.5 tier 2). vLLM only.

Off by default, because it costs something on every request: the router asks a replica to tokenize the prompt before placing it, a round trip and the chat template run twice. It pays where prompts share long prefixes that the approximate tree guesses wrong — many replicas, conversations that move between them, evictions. The engine is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED, which the router's hashing assumes; arguments overriding either leave the index empty, and the agent says so in its log.

Router

activation_wait

time.Duration

ActivationWait is how long a request waits for an on-demand model to load before the client is told to come back. Zero takes the default.

The one real decision in on-demand loading. Refusing outright makes the feature useless, since the first request for every model would fail; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds ends up holding a connection for ten minutes.

scrape_interval

time.Duration

scrape_timeout

time.Duration

stale_after

time.Duration

StaleAfter is how long an endpoint may go unscraped before being scored down (plan §7.6).

retries

int

Retries is how many additional endpoints may be tried after a failure, and only while nothing has been written to the client (plan §7.7).

cooldown

time.Duration

Cooldown is how long a failing endpoint leaves the candidate pool.

failure_threshold

int

FailureThreshold is the consecutive failure count that opens a breaker. 1 turns the breaker into a pure cooldown.

default_profile

string

DefaultProfile names the routing profile used when a model picks none.

prefix

Prefix

Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the router uses session affinity alone, which is tier 0 and already most of the win for multi-turn traffic.

Takes a Prefix block.

kv_index

KVIndex

KVIndex tunes the precise KV index (plan §7.5 tier 2). Which models use it is the catalog's to say, by kv_events on a vLLM profile; this is only how.

Takes a KVIndex block.

session_header

string

SessionHeader carries the session id for affinity (plan §7.5 tier 0).

inject_usage

bool

InjectUsage asks engines for usage even when the client did not, so token accounting works regardless of client behaviour.

tenant_label

string

TenantLabel is the level of the tree the per-tenant request series are labelled by — llmgw_tenant_requests_total and its siblings: "org", "team" or "project", or "none" for no such series. A label per project on a cluster of thousands is more series than Prometheus wants, and a team is what an alert is usually about; "team" is the default.

tag_labels

[]string

TagLabels are the request tags (metadata, x-llmgw-tags) counted in llmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value. Only these: a tag is the client's to set, and every value of one is a series. The first 100 values of each are counted apart, the rest as "other". Empty counts none; the usage records keep every tag anyway.

max_body_bytes

int64

MaxBodyBytes bounds the request body the router will read.

batch

Batch

Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of requests served at the lowest priority, its answers written to another. Needs the store.

Takes a Batch block.

cache

Cache

Cache bounds the answers kept for the projects that reuse them (llmgw keys cache). Which projects do, and for how long, is theirs to set; this is only how much memory it may take.

Takes a Cache block.

trusted_proxies

[]string

TrustedProxies are the reverse proxies in front of the client port, as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host are believed; anyone else's are not, since they decide the address written into the setup scripts and whose attempts are counted where.

timeouts

Timeouts

Takes a Timeouts block.

scoring

Scoring

Takes a Scoring block.

admission

Admission

Takes a Admission block.

Rules

pii

map[string]string

PII maps a kind (email, phone, iban, card, tckn, ip, secret) to what is done when it is found: flag, redact or block.

injection

string

Injection is what is done on signs of prompt injection: flag or block.

deny

[]Pattern

Deny are patterns, matched against the text as a model reads it.

max_chars

int

MaxChars refuses text longer than this; zero is no limit.

SCIM

token_file

string

TokenFile holds the bearer token the provider sends. Empty turns provisioning off.

SGLangConfig

python

string

Python is the interpreter of the SGLang virtualenv. SGLang is launched as a module rather than a console script, which is the documented entry point and avoids depending on the script being on PATH.

SLO

ttft_p99

time.Duration

TTFTp99 is the time to the first token 99 requests in 100 are within.

tpot_p99

time.Duration

TPOTp99 is the time per output token, after the first, 99 streamed requests in 100 are within.

ScaleWindow

days

[]string

Days are mon … sun, or weekdays, weekends; empty is every day.

from

string

From and To are HH:MM; a window whose To is before its From runs past midnight.

to

string

min

int

max

int

Max, when set, is the window's ceiling for load scaling; zero keeps the model's own.

Scaling

min

int

max

int

idle_timeout

time.Duration

IdleTimeout is how long an on-demand model may go unasked-for before its replicas are stopped. Zero takes the cluster default.

Minutes rather than seconds, always. Stopping a replica throws away the KV cache prefix routing spent its effort filling, and a model that flaps in and out loses that entirely — so the saving has to be worth more than the cache, which it only is over a real idle period.

schedule

[]ScaleWindow

Schedule changes min and max by the time of day, in the control plane's control.timezone: more replicas kept through working hours, fewer at night, an on-demand model kept loaded while people are at work. The first window that holds now wins; outside them all, Min and Max apply.

prewarm

time.Duration

Prewarm starts a window this long before it opens, so its replicas have loaded by then: loading a model is minutes, and the people who arrive at nine are the reason for the window.

Scoring

target_waiting

float64

target_inflight

float64

kv_ceiling

float64

session_load_factor

float64

Secrets

vault

secrets.Vault

Vault is how vault: references and vault-transit: are reached.

Takes a Vault block.

master_key

string

MasterKey seals the secrets the store keeps for itself — what every engine's key is derived from — so a copy of the store, or of a backup, does not hold them: vault-transit:KEY seals with Vault's transit engine, which never lets the key out; otherwise a reference to 32 random bytes (openssl rand -hex 32), which then must be kept as safely as the store. Values from before it was set are sealed the first time they are read.

Server

listen

string

Listen is the client-facing OpenAI API address.

admin_listen

string

AdminListen serves the dashboard, the management API, health and pprof.

Loopback by default, and it should stay there: pprof can dump the process heap, and API keys are in that heap in plaintext. Every comment beside this setting has always said not to expose it, while the default bound every interface — a default that contradicts its own documentation is a footgun, so the default now matches the advice.

metrics_listen

string

MetricsListen serves Prometheus metrics and nothing else.

Separate from AdminListen because the reason that one is loopback is pprof, not metrics — and metrics are the one thing on it that something else genuinely needs to reach. Sharing a listener meant a fleet could either be monitored or keep its heap private, which is not a choice anybody should have to make.

Empty means off, and metrics remain available on AdminListen either way, so nothing that scrapes it today has to change.

agent_listen

string

AgentListen is where node agents connect. Separate from the client port so the fleet-facing surface can be firewalled independently.

auth

Auth

Auth turns on API keys on the client-facing port.

Takes a Auth block.

threads

Threads

Threads keeps conversations and agent memory without API keys, for a gateway on one trusted machine. With auth on they live in the auth store, per project, and this block is ignored.

Takes a Threads block.

landing

Landing

Landing configures the page served at "/".

Takes a Landing block.

agent_tls

tlsconf.Config

AgentTLS authenticates the fleet-facing port. It deliberately does not cover the client-facing one: that surface is fronted by a publicly trusted certificate, while this one is mutual and private.

Takes a tlsconf.Config block.

max_connections

int

MaxConnections bounds the client port's open connections; past it new ones wait in the kernel's backlog. 0 is no bound.

admin_token_file

string

AdminTokenFile holds the token the admin API's /api and /debug paths demand. Empty is admin-token in the systemd unit's state directory (/var/lib/llmgw), or secrets/admin-token beside this file for a server run by hand. Made, 0600, when the server starts and it does not exist; sudo llmgw ctl and the console read it from there.

admin_remote

bool

AdminRemote allows admin_listen on an address other than loopback. The token still guards the API, but it then crosses the network in clear.

debug_pprof

bool

DebugPprof serves Go's profiler on the admin listener, behind the token. Off by default: a heap profile holds the API keys requests presented.

agent_insecure

bool

AgentInsecure lets a control plane run its agent port without agent_tls: plaintext, and any machine that reaches it may register as any node — be given replicas, and so be sent users' prompts — or, in the other direction, a machine in the path may tell every agent what to run. Refused unless said here, and then the port listens on loopback unless agent_listen names an address. For one machine, a test, or a network nothing untrusted reaches.

monitoring

Monitoring

Monitoring is where the dashboard reads its history from.

Takes a Monitoring block.

tracing

obs.Tracing

Tracing sends traces to an OpenTelemetry collector: each inference request, and on a control plane each placement, replica start and model job. Off unless an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.

Takes a Tracing block.

mirror

Mirror

Mirror points the control plane at a model mirror it can administer. Empty means there is none, and /admin/mirror stays unmounted.

Takes a Mirror block.

offline

Offline

Offline is a cluster that reaches nothing outside itself: its nodes install engines, and upgrades fetch releases, from the mirror, out of a bundle carried in (llmgw bundle create, llmgw bundle import).

Takes a Offline block.

engines

map[string]string

Engines are the engines every node installs, and at which version. The agents install them themselves — the right build for their hardware, checked against its publisher's checksum and seen to run — so a node is never trusted to already have what somebody once put there.

Unset keeps DefaultEngines. A version replaces the default; an empty one leaves that engine to be installed by hand.

engine_overrides

[]EngineOverride

EngineOverrides give some nodes different engines from the rest: a vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at all on nodes meant only for llama.cpp. Each picks nodes and lays its engines over the cluster's, in order, later ones last; an empty version removes that engine from the nodes it picks.

backup_dir

string

BackupDir is where llmgw server backup writes, and where the newest archive's age is read for llmgw_backup_last_success_timestamp_seconds. Empty means /var/backups/llmgw-server.

guard

guard.Config

Guard is the library of guardrail policies: what requests and answers are checked for, and what is done. Applied to every request through guard.default, and to parts of the tenancy tree by attaching them (llmgw ctl guard attach). Empty checks nothing.

Takes a guard.Config block.

secrets

Secrets

Secrets is where secrets kept outside files are read from, and the key the store's own secrets are sealed under.

Takes a Secrets block.

notify

notify.Config

Notify sends what needs a person — a model parked, a node lost, a budget spent, new weights published — to webhooks, signed, without a Prometheus and Alertmanager in between. Empty sends nothing.

Takes a notify.Config block.

mail

mail.Config

Mail is where the gateway's few emails go out from: a link to set a forgotten password. Unset, none is sent and people ask an operator.

Takes a mail.Config block.

serving

ServingSettings

Serving stops and starts the whole cluster's service (llmgw stop, llmgw start).

Takes a ServingSettings block.

firewall

FirewallSettings

Firewall turns llmgw's firewall on every machine on or off.

Takes a FirewallSettings block.

agents

AgentSettings

Agents are the nodes' agent settings: agent.yaml's keys but those an agent needs to start, for every node (default) and for one (nodes.). Sent to the agents as they change; an agent whose settings changed starts again on them, its engines serving throughout.

Takes a AgentSettings block.

firewall_networks

[]string

FirewallNetworks are networks, besides those each machine reaches the cluster on, that llmgw's internal ports answer: a monitoring host on another subnet, nodes behind a router. Written into every machine's firewall by server init, node add and server add.

backup_recipients

[]string

BackupRecipients encrypt every archive llmgw server backup writes, to whoever holds one of their keys: age recipients (age1…) or SSH public keys (ssh-ed25519 …, ssh-rsa …), as age takes them. An archive holds the certificate authority's key and every secret in the store, and is copied off the machine; encrypted, the copy is safe wherever it lands. Empty writes archives unencrypted, readable by root only.

backup_copy_to

[]string

BackupCopyTo is where each archive is copied once written, so it survives this machine: a directory (a mounted share, another disk) or ssh://user@host[:port]/path, reached with BackupCopyKey and checked against /etc/llmgw/backup_known_hosts, which learns a host the first time. The same number of archives is kept there as here. Refused without backup_recipients: an archive leaves the machine encrypted.

backup_copy_key

string

BackupCopyKey is the SSH private key for ssh:// copies.

releases

string

Releases is where llmgw ctl upgrade fetches release binaries from, for this machine and — through their agents — every node. Empty means the project's own releases.

router

Router

Takes a Router block.

control

Control

Takes a Control block.

models

[]catalog.Model

facility

Facility

Facility is the power caps of racks, PDUs and other domains, and how a replica's draw is estimated against them. Where a node is comes from its llmgw.io/site, room, row, rack and pdu labels; holding a domain for maintenance is llmgw ctl facility hold, not configuration.

Takes a Facility block.

federation

Federation

Federation is how this cluster takes part in federation with other llmgw clusters. The peers themselves are added at runtime, with llmgw ctl federation add, and kept in the store.

Takes a Federation block.

endpoints

[]Endpoint

Endpoints is the phase-1 static replica list. It is only consulted when no model declares a profile, which is what makes a router-only smoke test against sim-engines possible without a control plane.

ServingSettings

stopped

bool

Stopped takes every model down and answers every request for one with 503 (system_stopped), leaving the control planes, agents and mirror running and every model's definition as it is: llmgw stop sets it, llmgw start clears it, and the models come back as their catalog entries ask. In force on every control plane within seconds.

SimConfig

bin

string

Bin is the llmgw binary itself; the engine is one of its subcommands. Empty means whatever started this agent, which is right almost always.

TRTLLMConfig

python

string

Python is the interpreter of the environment TensorRT-LLM is installed in: /bin/python for a virtual environment, python3 in NVIDIA's container. Naming it is what makes this node offer the engine; empty offers nothing.

The interpreter rather than trtllm-serve, because llmgw starts the server through a few lines of its own that make it demand the gateway's key, which trtllm-serve has no option for.

Threads

store

string

Store is the SQLite file. Empty means nothing is kept.

retain_hours

int

RetainHours is how long idle conversations and notes are kept. Required with Store: a store with no expiry keeps everything anyone typed, by omission.

Timeouts

dial

time.Duration

response_header

time.Duration

ResponseHeader bounds the wait for headers on a streaming request. It is a liveness check there, because headers arrive before generation starts.

generation

time.Duration

Generation bounds a non-streaming request, where the engine sends no headers until the completion is done. Setting response_header alone used to cap both, so 60s read as a liveness guard and behaved as a limit of about a hundred tokens on a slow model.

first_token

time.Duration

idle

time.Duration

total

time.Duration

Tools

allow

[]string

Allow, when set, are the only tool names a request may offer and a model may call.

deny

[]string

Deny are tool names never offered or called.

on_denied

string

OnDenied is strip (the tool is removed from the request) or block. Empty is strip.

max_calls_per_turn

int

MaxCallsPerTurn stops an answer calling more tools than this.

validate_arguments

bool

ValidateArguments checks a call's arguments against the parameters schema the request declared for the tool.

Tracing

endpoint

string

Endpoint is the collector, for example http://otel.internal:4318. Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.

sample_ratio

float64

SampleRatio is the share of traces kept, 0 to 1, of those nobody upstream already decided. Zero takes 1. A request with a sampled traceparent of its own is kept, and an agent keeps every start the control plane kept, whatever their own ratio says.

VLLMConfig

bin

string

Bin is the vllm executable.

launch_mode

string

LaunchMode selects the multi-node mechanism: "mp", the default, or "ray".

mp is what vLLM documents today — every node runs serve with its rank, the workers headless — and needs nothing but vLLM. ray is the older way, kept for a vLLM too old for mp's multi-node flags; it needs Ray on every node and has not been checked against a real cluster.

ray_bin

string

RayBin is used when LaunchMode is "ray".

Vault

addr

string

Addr is Vault's address, like https://vault.internal:8200; empty takes VAULT_ADDR.

token_file

string

TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with AppRole when RoleIDFile and SecretIDFile are set.

role_id_file

string

secret_id_file

string

ca_file

string

CAFile verifies Vault's certificate; empty uses the system's.

namespace

string

Namespace is Vault Enterprise's or OpenBao's namespace.

Webhook

url

string

format

string

Format is json (the event, with X-Llmgw-Signature when SecretFile is set) or slack ({"text": …}, which Slack, Mattermost and Rocket.Chat incoming webhooks take).

events

[]string

Events are the kinds sent here; empty is all.

secret_file

string

SecretFile holds the key the body is signed with: X-Llmgw-Signature is sha256=.

accounting.Price

prompt_per_1k

int64

completion_per_1k

int64

per_image

int64

PerImage is what one picture costs, for a model that makes them: a picture's cost is the card's time, which its prompt's few tokens do not measure.

per_mesh

int64

PerMesh is what one 3D model costs: minutes of a card, which no token count measures.

admission.Band

name

string

priority

int

Priority orders bands; higher is served first.

saturation_ceiling

float64

SaturationCeiling is the pool saturation above which this band waits. 1.0 means "never hold this band back".

queue_ttl

time.Duration

QueueTTL, set, is how long this band's requests may wait instead of the controller's: a band for bulk work — a knowledge-base import, a backfill — whose requests had rather wait ten minutes behind people than be turned away after thirty seconds.

admission.Config

bands

[]Band

default_band

string

queue_ttl

time.Duration

QueueTTL drops a request that has waited this long. Sending the GPU work the client has already given up on helps nobody.

queue_capacity_per_band

int

QueueCapacityPerBand bounds each band's queue. Rejecting immediately beats accumulating silently.

queue_capacity_total

int

QueueCapacityTotal bounds everything queued, across bands. When it is reached, a request is not refused while a lower band has requests queued: the newest of the lowest band is turned away instead (displacement), since the oldest have waited longest and are nearest their TTL. Zero takes half of every band's capacity together.

interval

time.Duration

Interval is how often the dispatcher re-checks saturation. Releases also wake it, so this is only the backstop for saturation falling on its own.

agent.Config

node_id

string

NodeID defaults to the hostname. It must equal the common name of the agent's certificate, so set it only where the two differ.

server

string

Server is the control plane, and Servers is the same thing when there is more than one. Both are accepted: every configuration written before there could be a second control plane names one address, and none of them should have to change.

The order is tried in order, and a refusal moves to the next. Only the leader accepts an agent, so a standby is skipped in the time it takes to be told so — which is what makes failover a reconnect rather than an edit to every node's configuration.

servers

[]string

hostname

string

Hostname defaults to the machine's.

mgmt_addr

string

MgmtAddr is the address engines are reached on. It defaults to this machine's address on the route to the control plane, which is the one the control plane and the router can reach.

labels

map[string]string

Labels are free-form facts about a node for a profile's node_selector. Nothing needs them: memory, GPUs, engines and disk are measured, and a label restating one of those goes stale the day the machine changes.

mgmt_interface

string

MgmtInterface is the ethernet interface NCCL bootstraps over. It defaults to the interface holding MgmtAddr, then eth0.

fabrics

[]inventory.FabricAttachment

Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's default subnet prefix is the same on every fabric, so nothing can be inferred from the wire (plan §5.2).

heartbeat_interval

time.Duration

reconnect_base

time.Duration

ReconnectBase is the first reconnect delay; it backs off from there.

reconnect_max

time.Duration

adopt_ready_timeout

time.Duration

AdoptReadyTimeout bounds how long an adopted engine is given to answer a health check before the agent stops waiting on it.

Shorter than a startup timeout on purpose: this engine is already running, so it is not loading a model, it is either serving or it is not.

tls

tlsconf.Config

TLS authenticates both ends of the control-plane stream. Leaving it unset keeps the plaintext behaviour, which only belongs on a trusted network.

Takes a tlsconf.Config block.

model_dirs

[]string

ModelDirs are the directories this node keeps model weights in, beyond WeightsCache, which is always looked in. Storage differs at every installation, so llmgw observes it rather than owning it: a Lustre mount, an image exported over the fabric, local NVMe and a copy made by hand all look identical from here, and placement only needs to know whether the file is present.

engines_dir

string

EnginesDir is where the agent installs engines, one directory per build. Defaults to /opt/llmgw/engines.

weights_cache

string

WeightsCache is where a remote source is downloaded to. Defaults to the first entry of ModelDirs, because a node that says where its models live has already answered this question, and otherwise to /var/lib/llmgw-weights.

Fetching is the simple arrangement: every node keeps its own copy. The exotic ones need no code — weights on a shared filer or an image exported over the fabric are simply present, and present means nothing is fetched.

discovery

discovery.Options

Takes a Options block.

engines

engines.Config

Takes a engines.Config block.

supervisor

supervisor.Config

Takes a supervisor.Config block.

tracing

obs.Tracing

Tracing sends this node's part of each replica start — the weights fetched, the engine launched, the wait for it to answer — to an OpenTelemetry collector, inside the control plane's trace of that start. Give it the collector server.yaml's tracing names. Off unless an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.

Takes a Tracing block.

config.Band

name

string

priority

int

saturation_ceiling

float64

queue_ttl

time.Duration

QueueTTL is how long this band's requests may wait for room; the admission's queue_ttl unless set. Long for a band of bulk work, which had rather wait behind people than be turned away.

config.Price

prompt_per_1k

int64

completion_per_1k

int64

per_image

int64

PerImage is the cost of one picture, for a model that makes them (/v1/images/generations), on top of its prompt's tokens.

per_mesh

int64

PerMesh is the cost of one 3D model (/v1/3d/generations).

engines.Config

disabled

[]string

Disabled are engines this node never installs or advertises, whatever the control plane asks: vLLM on a node of cards no build of it runs on, that is meant for llama.cpp alone.

vllm

VLLMConfig

Takes a VLLMConfig block.

sglang

SGLangConfig

Takes a SGLangConfig block.

llamacpp

LlamaCppConfig

Takes a LlamaCppConfig block.

trtllm

TRTLLMConfig

TRTLLM is TensorRT-LLM, which the agent does not install: a node offers it only when its interpreter is named here.

Takes a TRTLLMConfig block.

sim

SimConfig

Takes a SimConfig block.

comfyui

ComfyUIConfig

ComfyUI makes pictures; offered by a node whose ComfyUI is named here.

Takes a ComfyUIConfig block.

guard.Config

enabled

*bool

Enabled turns every policy on or off at once: false checks nothing, whatever is defined or attached. Unset is on.

default

[]string

policies

map[string]Policy

mail.Config

smtp

string

SMTP is the relay, host:port: 587 with STARTTLS, 465 with TLS from the start, 25 as the relay allows. Empty sends nothing.

from

string

From is the sender, llmgw@example.com or "llmgw llmgw@example.com".

username

string

Username and PasswordFile log in to the relay, when it wants that. The password is a secret reference: a file, env:NAME or vault:path.

password_file

string

base_url

string

BaseURL is the gateway's public address, which links in mail point at: https://llm.example.com. Never taken from a request, whose Host a sender chooses and could point a reset link at their own site.

notify.Config

webhooks

[]Webhook

host

string

Host names the gateway in messages; empty is this machine's name.

supervisor.Config

work_dir

string

WorkDir holds pid files and per-shard scratch space.

log_lines

int

LogLines is how many lines of engine output to retain per shard.

stop_timeout

time.Duration

StopTimeout is how long a process gets to exit after SIGTERM before it is killed. Engines need time to free GPU memory cleanly.

health_interval

time.Duration

HealthInterval is how often a starting shard is polled.

rendezvous_timeout

time.Duration

RendezvousTimeout bounds the wait for a multi-node rendezvous point to start listening.

alive_grace

time.Duration

AliveGrace is how long a non-serving rank must stay up before it counts as ready. There is no external signal for a worker, so staying alive is the only evidence available.

run_as

string

RunAs is the user engines run as when the agent is root; empty is llmgw-engine, "root" keeps them root.

tlsconf.Config

ca

string

CA is the bundle used to verify the other side. In a real deployment this is whichever authority issued the fleet's certificates.

cert

string

Cert and Key are this component's own identity.

key

string

insecure

bool

Insecure disables TLS entirely. It exists for local runs against sim-engines; anything using it must stay on a trusted network.