llmgw/docs/reference.md

Configuration reference

Every setting llmgw reads from a file, with the type it takes and what it is for.

Generated from the source. The descriptions are the comments on the fields themselves, so this cannot describe a setting that does not exist or miss one that does — and a test fails when it has drifted. Do not edit it; edit the comment and run go generate ./....

Where a setting takes another block, the block is listed under its own heading rather than repeated.

Every key

Key	Type	In
enabled	bool	Admission
bands	[]Band	Admission
default_band	string	Admission
band_header	string	Admission
queue_ttl	time.Duration	Admission
queue_capacity_per_band	int	Admission
interval	time.Duration	Admission
enabled	bool	Auth
store	string	Auth
cache_ttl	time.Duration	Auth
negative_ttl	time.Duration	Auth
default_rpm	int	Auth
default_tpm	int	Auth
signup_open	bool	Auth
pricing	map[string]Price	Auth
default_price	Price	Auth
flush_interval	time.Duration	Auth
batch_size	int	Auth
buffer_size	int	Auth
name	string	Band
priority	int	Band
saturation_ceiling	float64	Band
tools	bool	Capabilities
vision	bool	Capabilities
embeddings	bool	Capabilities
node_id	string	Config
server	string	Config
servers	[]string	Config
hostname	string	Config
mgmt_addr	string	Config
labels	map[string]string	Config
mgmt_interface	string	Config
fabrics	[]inventory.FabricAttachment	Config
heartbeat_interval	time.Duration	Config
reconnect_base	time.Duration	Config
reconnect_max	time.Duration	Config
adopt_ready_timeout	time.Duration	Config
tls	tlsconf.Config	Config
model_dirs	[]string	Config
weights_cache	string	Config
discovery	discovery.Options	Config
engines	engines.Config	Config
supervisor	supervisor.Config	Config
interval	time.Duration	Control
base_backoff	time.Duration	Control
max_backoff	time.Duration	Control
unreachable_after	time.Duration	Control
lost_after	time.Duration	Control
port_range	[]int	Control
mgmt_interface	string	Control
lease_ttl	time.Duration	Control
id	string	Endpoint
url	string	Endpoint
models	[]string	Endpoint
engine	string	Endpoint
pool	string	Endpoint
weight	float64	Endpoint
priority	int	Endpoint
contact	string	Landing
notice	string	Landing
bin	string	LlamaCppConfig
parallel	int	LlamaCppConfig
name	string	Model
aliases	[]string	Model
source	string	Model
profiles	[]Profile	Model
scaling	Scaling	Model
fallbacks	[]string	Model
routing_profile	string	Model
slo	SLO	Model
context_length	int	Model
max_output	int	Model
capabilities	Capabilities	Model
nvidia_smi	string	Options
ib_root	string	Options
prefer_nvml	bool	Options
skip_gpus	bool	Options
enabled	bool	Prefix
max_nodes	int	Prefix
min_match	int	Prefix
ttl	time.Duration	Prefix
match_ratio	float64	Prefix
balance_abs	float64	Prefix
balance_rel	float64	Prefix
prompt_per_1k	int64	Price
completion_per_1k	int64	Price
name	string	Profile
engine	string	Profile
gpu_class	string	Profile
tp	int	Profile
pp	int	Profile
gpus_per_node	int	Profile
nodes	int	Profile
args	[]string	Profile
env	map[string]string	Profile
min_memory_gb	int	Profile
min_disk_gb	int	Profile
node_selector	map[string]string	Profile
weight	float64	Profile
priority	int	Profile
startup_timeout	time.Duration	Profile
activation_wait	time.Duration	Router
scrape_interval	time.Duration	Router
scrape_timeout	time.Duration	Router
stale_after	time.Duration	Router
retries	int	Router
cooldown	time.Duration	Router
failure_threshold	int	Router
default_profile	string	Router
prefix	Prefix	Router
session_header	string	Router
inject_usage	bool	Router
max_body_bytes	int64	Router
timeouts	Timeouts	Router
scoring	Scoring	Router
admission	Admission	Router
python	string	SGLangConfig
ttft_p99	time.Duration	SLO
tpot_p99	time.Duration	SLO
min	int	Scaling
max	int	Scaling
idle_timeout	time.Duration	Scaling
target_waiting	float64	Scoring
target_inflight	float64	Scoring
kv_ceiling	float64	Scoring
session_load_factor	float64	Scoring
listen	string	Server
admin_listen	string	Server
metrics_listen	string	Server
agent_listen	string	Server
auth	Auth	Server
threads	Threads	Server
landing	Landing	Server
agent_tls	tlsconf.Config	Server
router	Router	Server
control	Control	Server
models	[]catalog.Model	Server
endpoints	[]Endpoint	Server
bin	string	SimConfig
store	string	Threads
retain_hours	int	Threads
dial	time.Duration	Timeouts
response_header	time.Duration	Timeouts
generation	time.Duration	Timeouts
first_token	time.Duration	Timeouts
idle	time.Duration	Timeouts
total	time.Duration	Timeouts
bin	string	VLLMConfig
launch_mode	string	VLLMConfig
ray_bin	string	VLLMConfig

Admission

enabled

bool

Enabled turns the layer on. Off, every request goes straight through, which is the v0.2.0 behaviour.

bands

[]Band

default_band

string

band_header

string

BandHeader lets a client declare its band. Until authentication lands in v1.0.0 this is trusted, so the port must not be exposed to callers who should not be able to promote themselves.

queue_ttl

time.Duration

queue_capacity_per_band

int

interval

time.Duration

Auth

enabled

bool

store

string

Store is the SQLite file holding keys, tenancy and usage.

cache_ttl

time.Duration

CacheTTL bounds how long a revoked key keeps working, traded against a database read per key per TTL.

negative_ttl

time.Duration

NegativeTTL is how long a rejected key is remembered as rejected, so a client looping with a bad key cannot become a database read per request.

default_rpm

int

DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.

Unlimited used to be the default, and a project created without limits is the easy path — the "default" project on this cluster still is one. The first sign of that is the bill or the outage. Zero keeps the old behaviour for anyone who wants it, but it should be a choice rather than an omission. A fallback, not a ceiling: an explicit limit at any scope wins, above or below.

default_tpm

int

signup_open

bool

SignupOpen serves the signup form. Accounts it creates are pending until an operator approves them, so this is not a way in — but turning it off means accounts are made with llmgw users create instead.

pricing

map[string]Price

Pricing is what each model costs, for budgets. A model absent from the table costs whatever DefaultPrice says, which is nothing unless set.

default_price

Price

Takes a Price block.

flush_interval

time.Duration

FlushInterval and BatchSize decide when usage records are written.

batch_size

int

buffer_size

int

BufferSize bounds the queue. Beyond it records are dropped rather than blocking a request: losing accounting is bad, turning a stalled database into an outage is worse.

Band

name

string

priority

int

saturation_ceiling

float64

Capabilities

tools

bool

vision

bool

embeddings

bool

Config

node_id

string

server

string

Server is the control plane, and Servers is the same thing when there is more than one. Both are accepted: every configuration written before there could be a second control plane names one address, and none of them should have to change.

The order is tried in order, and a refusal moves to the next. Only the leader accepts an agent, so a standby is skipped in the time it takes to be told so — which is what makes failover a reconnect rather than an edit to every node's configuration.

servers

[]string

hostname

string

mgmt_addr

string

labels

map[string]string

mgmt_interface

string

MgmtInterface is the ethernet interface NCCL bootstraps over.

fabrics

[]inventory.FabricAttachment

Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's default subnet prefix is the same on every fabric, so nothing can be inferred from the wire (plan §5.2).

heartbeat_interval

time.Duration

reconnect_base

time.Duration

ReconnectBase is the first reconnect delay; it backs off from there.

reconnect_max

time.Duration

adopt_ready_timeout

time.Duration

AdoptReadyTimeout bounds how long an adopted engine is given to answer a health check before the agent stops waiting on it.

Shorter than a startup timeout on purpose: this engine is already running, so it is not loading a model, it is either serving or it is not.

tls

tlsconf.Config

TLS authenticates both ends of the control-plane stream. Leaving it unset keeps the plaintext behaviour, which only belongs on a trusted network.

Takes a Config block.

model_dirs

[]string

ModelDirs are the directories this node keeps model weights in.

Empty by default, and empty means the node is not asked to look — which is not the same as having nothing. Storage differs at every installation, so llmgw observes it rather than owning it: a Lustre mount, an image exported over the fabric, local NVMe and a copy made by hand all look identical from here, and placement only needs to know whether the file is present.

weights_cache

string

WeightsCache is where a remote source is downloaded to. Defaults to the first entry of ModelDirs, because a node that says where its models live has already answered this question.

Fetching is the simple arrangement: every node keeps its own copy. The exotic ones need no code — weights on a shared filer or an image exported over the fabric are simply present, and present means nothing is fetched.

discovery

discovery.Options

Takes a Options block.

engines

engines.Config

Takes a Config block.

supervisor

supervisor.Config

Takes a Config block.

Control

interval

time.Duration

Interval is how often desired and actual state are compared.

base_backoff

time.Duration

BaseBackoff and MaxBackoff bound how fast a failing model is retried.

max_backoff

time.Duration

unreachable_after

time.Duration

UnreachableAfter and LostAfter stage node expiry. They are deliberately far apart: a network blip must not tear down replicas that still serve, and rebuilding a multi-node replica costs tens of minutes of loading.

lost_after

time.Duration

port_range

[]int

PortRange is the per-node range for engine HTTP and dist-init ports.

mgmt_interface

string

MgmtInterface is the ethernet interface NCCL bootstraps over.

lease_ttl

time.Duration

LeaseTTL is how long this control plane's claim to be the one placing work stands without being renewed.

Only meaningful with a shared store, where a second control plane can see the claim. Long enough to survive a slow query or a brief network stall, short enough that a standby takes over before anyone has finished reading the alert. Zero takes the default.

Endpoint

id

string

url

string

models

[]string

engine

string

Engine names the inference engine, which decides what its metrics are called. Empty means vLLM's names.

pool

string

weight

float64

priority

int

Landing

contact

string

Contact is where someone without a key should go — an address, a URL, a channel name. Rendered as a link when it is one, as text otherwise.

notice

string

Notice is an optional line of your own, for whatever the reader needs to know that this page cannot guess.

LlamaCppConfig

bin

string

Bin is the llama-server binary. It is the server, not the CLI: the gateway talks OpenAI over HTTP and needs the one that serves it.

parallel

int

Parallel is how many requests a server handles at once, when a profile does not say. llama.cpp defaults to one, which would make every second request wait behind the first no matter what the router decided.

Model

name

string

aliases

[]string

source

string

Source is the weights path or a HuggingFace id.

profiles

[]Profile

scaling

Scaling

Takes a Scaling block.

fallbacks

[]string

routing_profile

string

RoutingProfile selects how the router scores endpoints for this model.

slo

SLO

Takes a SLO block.

context_length

int

ContextLength is how many tokens one request may use, prompt and completion together, and MaxOutput bounds the completion alone. Zero means unknown, which is what every model was until now.

The gateway knew neither, so an over-long prompt was not refused: it was admitted, given a replica, and failed at the engine, and a client sizing its prompts from /v1/models had nothing to read. Worse, the number is not the one an operator writes in the engine's arguments — llama.cpp divides --ctx-size across its parallel slots, so a pool of 8192 across four slots is 2048 a request, and nothing said so.

max_output

int

capabilities

Capabilities

Capabilities is what this model can do beyond plain completion.

Takes a Capabilities block.

Options

nvidia_smi

string

NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH. Worth configuring on nodes where the driver lives outside the default paths, which containerized driver installs often do.

ib_root

string

IBRoot is the sysfs InfiniBand directory.

prefer_nvml

bool

PreferNVML uses the NVML backend when this binary has one. It is the only source of NVLink topology.

skip_gpus

bool

SkipGPUs omits GPU discovery, for nodes that have none.

Prefix

enabled

bool

max_nodes

int

MaxNodes bounds the tree, MinMatch is the shortest match worth acting on in bytes, and TTL is how long an entry is believed — an engine evicts on its own schedule, so an old record may describe a cache that is gone.

min_match

int

ttl

time.Duration

match_ratio

float64

MatchRatio is the share of the prompt that must match to be worth acting on. BalanceAbs and BalanceRel are the imbalance guard, and both must trip before affinity is abandoned.

balance_abs

float64

balance_rel

float64

Price

prompt_per_1k

int64

completion_per_1k

int64

Profile

name

string

engine

string

gpu_class

string

tp

int

pp

int

gpus_per_node

int

GPUsPerNode defaults to TP for a single-node profile.

nodes

int

Nodes above 1 makes this a multi-node profile, which requires a fabric.

args

[]string

env

map[string]string

min_memory_gb

int

MinMemoryGB is the host RAM this profile needs to run.

The number that matters is the model's working set: the weights, plus the KV cache for the whole context pool, plus the engine's own overhead. For a GPU profile that lives in the cards and this is small; for a CPU profile it is the whole thing, and getting it wrong means an OOM on a node that reported itself healthy.

Zero means unconstrained, which is every profile that existed before this. A node that does not report its memory is not excluded either — an older agent must not have its nodes quietly emptied by an upgrade of the control plane.

min_disk_gb

int

MinDiskGB is the free space a node needs before it can hold these weights.

Only checked when the node does not already have them: a node holding the file needs no room for it, and requiring the space twice would keep a model off the one node best placed to run it.

Declared rather than derived, because the size of a model that has never been fetched anywhere is not knowable from here — and a wrong guess fails late, halfway through a download, which is the failure this exists to prevent. Zero means unconstrained, and a node that reports no storage is not excluded: unknown is not full.

node_selector

map[string]string

NodeSelector restricts this profile to nodes carrying every one of these labels, with these values.

A CPU profile otherwise fits everywhere, because there is no card to be free — which is right for a 0.5B model and wrong for a 14B one. Nothing in the inventory says how much memory a node has, so the operator says which nodes are meant for a model instead of the scheduler inferring it. Agents have reported labels since the beginning; this is what reads them.

weight

float64

Weight is the router's capacity weight. It should come from ctl bench rather than a guess (plan §6.2.1).

priority

int

Priority orders profiles during placement; highest that fits wins.

startup_timeout

time.Duration

StartupTimeout bounds model loading. A 70B model across two nodes takes tens of minutes, so this is per profile rather than global.

Router

activation_wait

time.Duration

ActivationWait is how long a request waits for an on-demand model to load before the client is told to come back. Zero takes the default.

The one real decision in on-demand loading. Refusing outright makes the feature useless, since the first request for every model would fail; waiting without a bound turns a request into a job, and a client that budgeted thirty seconds ends up holding a connection for ten minutes.

scrape_interval

time.Duration

scrape_timeout

time.Duration

stale_after

time.Duration

StaleAfter is how long an endpoint may go unscraped before being scored down (plan §7.6).

retries

int

Retries is how many additional endpoints may be tried after a failure, and only while nothing has been written to the client (plan §7.7).

cooldown

time.Duration

Cooldown is how long a failing endpoint leaves the candidate pool.

failure_threshold

int

FailureThreshold is the consecutive failure count that opens a breaker. 1 turns the breaker into a pure cooldown.

default_profile

string

DefaultProfile names the routing profile used when a model picks none.

prefix

Prefix

Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the router uses session affinity alone, which is tier 0 and already most of the win for multi-turn traffic.

Takes a Prefix block.

session_header

string

SessionHeader carries the session id for affinity (plan §7.5 tier 0).

inject_usage

bool

InjectUsage asks engines for usage even when the client did not, so token accounting works regardless of client behaviour.

max_body_bytes

int64

MaxBodyBytes bounds the request body the router will read.

timeouts

Timeouts

Takes a Timeouts block.

scoring

Scoring

Takes a Scoring block.

admission

Admission

Takes a Admission block.

SGLangConfig

python

string

Python is the interpreter of the SGLang virtualenv. SGLang is launched as a module rather than a console script, which is the documented entry point and avoids depending on the script being on PATH.

SLO

ttft_p99

time.Duration

tpot_p99

time.Duration

Scaling

min

int

max

int

idle_timeout

time.Duration

IdleTimeout is how long an on-demand model may go unasked-for before its replicas are stopped. Zero takes the cluster default.

Minutes rather than seconds, always. Stopping a replica throws away the KV cache prefix routing spent its effort filling, and a model that flaps in and out loses that entirely — so the saving has to be worth more than the cache, which it only is over a real idle period.

Scoring

target_waiting

float64

target_inflight

float64

kv_ceiling

float64

session_load_factor

float64

Server

listen

string

Listen is the client-facing OpenAI API address.

admin_listen

string

AdminListen serves the dashboard, the management API, health and pprof.

Loopback by default, and it should stay there: pprof can dump the process heap, and API keys are in that heap in plaintext. Every comment beside this setting has always said not to expose it, while the default bound every interface — a default that contradicts its own documentation is a footgun, so the default now matches the advice.

metrics_listen

string

MetricsListen serves Prometheus metrics and nothing else.

Separate from AdminListen because the reason that one is loopback is pprof, not metrics — and metrics are the one thing on it that something else genuinely needs to reach. Sharing a listener meant a fleet could either be monitored or keep its heap private, which is not a choice anybody should have to make.

Empty means off, and metrics remain available on AdminListen either way, so nothing that scrapes it today has to change.

agent_listen

string

AgentListen is where node agents connect. Separate from the client port so the fleet-facing surface can be firewalled independently.

auth

Auth

Auth turns on API keys on the client-facing port.

Takes a Auth block.

threads

Threads

Threads keeps conversations and agent memory without API keys, for a gateway on one trusted machine. With auth on they live in the auth store, per project, and this block is ignored.

Takes a Threads block.

landing

Landing

Landing configures the page served at "/".

Takes a Landing block.

agent_tls

tlsconf.Config

AgentTLS authenticates the fleet-facing port. It deliberately does not cover the client-facing one: that surface is fronted by a publicly trusted certificate, while this one is mutual and private.

Takes a Config block.

router

Router

Takes a Router block.

control

Control

Takes a Control block.

models

[]catalog.Model

endpoints

[]Endpoint

Endpoints is the phase-1 static replica list. It is only consulted when no model declares a profile, which is what makes a router-only smoke test against sim-engines possible without a control plane.

SimConfig

bin

string

Bin is the llmgw binary itself; the engine is one of its subcommands. Empty means whatever started this agent, which is right almost always.

Threads

store

string

Store is the SQLite file. Empty means nothing is kept.

retain_hours

int

RetainHours is how long idle conversations and notes are kept. Required with Store: a store with no expiry keeps everything anyone typed, by omission.

Timeouts

dial

time.Duration

response_header

time.Duration

ResponseHeader bounds the wait for headers on a streaming request. It is a liveness check there, because headers arrive before generation starts.

generation

time.Duration

Generation bounds a non-streaming request, where the engine sends no headers until the completion is done. Setting response_header alone used to cap both, so 60s read as a liveness guard and behaved as a limit of about a hundred tokens on a slow model.

first_token

time.Duration

idle

time.Duration

total

time.Duration

VLLMConfig

bin

string

Bin is the vllm executable.

launch_mode

string

LaunchMode selects the multi-node mechanism: "ray" or "mp".

Which one works depends on the vLLM version, and the flags have changed more than once (plan §14). Both paths exist so a cluster can pick the one its version supports instead of waiting for a code change.

ray_bin

string

RayBin is used when LaunchMode is "ray".