# Configuration reference Every setting llmgw reads from a file, with the type it takes and what it is for. **Generated from the source.** The descriptions are the comments on the fields themselves, so this cannot describe a setting that does not exist or miss one that does — and a test fails when it has drifted. Do not edit it; edit the comment and run `go generate ./...`. Where a setting takes another block, the block is listed under its own heading rather than repeated. ## Every key | Key | Type | In | |---|---|---| | `name` | `string` | [Adapter](#adapter) | | `source` | `string` | [Adapter](#adapter) | | `scale` | `float64` | [Adapter](#adapter) | | `owner` | `string` | [Adapter](#adapter) | | `enabled` | `bool` | [Admission](#admission) | | `bands` | `[]Band` | [Admission](#admission) | | `default_band` | `string` | [Admission](#admission) | | `band_header` | `string` | [Admission](#admission) | | `queue_ttl` | `time.Duration` | [Admission](#admission) | | `queue_capacity_per_band` | `int` | [Admission](#admission) | | `queue_capacity_total` | `int` | [Admission](#admission) | | `interval` | `time.Duration` | [Admission](#admission) | | `default` | `map[string]any` | [AgentSettings](#agentsettings) | | `nodes` | `map[string]map[string]any` | [AgentSettings](#agentsettings) | | `enabled` | `bool` | [Auth](#auth) | | `store` | `string` | [Auth](#auth) | | `cache_ttl` | `time.Duration` | [Auth](#auth) | | `negative_ttl` | `time.Duration` | [Auth](#auth) | | `default_rpm` | `int` | [Auth](#auth) | | `default_tpm` | `int` | [Auth](#auth) | | `signup_open` | `bool` | [Auth](#auth) | | `oidc` | `OIDC` | [Auth](#auth) | | `scim` | `SCIM` | [Auth](#auth) | | `max_key_ttl` | `time.Duration` | [Auth](#auth) | | `personal` | `Personal` | [Auth](#auth) | | `pricing` | `map[string]Price` | [Auth](#auth) | | `default_price` | `Price` | [Auth](#auth) | | `flush_interval` | `time.Duration` | [Auth](#auth) | | `batch_size` | `int` | [Auth](#auth) | | `buffer_size` | `int` | [Auth](#auth) | | `usage_retention` | `time.Duration` | [Auth](#auth) | | `max_file_bytes` | `int64` | [Batch](#batch) | | `max_lines` | `int` | [Batch](#batch) | | `concurrency` | `int` | [Batch](#batch) | | `retain_days` | `int` | [Batch](#batch) | | `max_bytes` | `int64` | [Cache](#cache) | | `max_entry_bytes` | `int64` | [Cache](#cache) | | `model` | `string` | [Canary](#canary) | | `percent` | `int` | [Canary](#canary) | | `min_requests` | `int` | [Canary](#canary) | | `max_error_increase` | `float64` | [Canary](#canary) | | `max_ttft_ratio` | `float64` | [Canary](#canary) | | `tools` | `bool` | [Capabilities](#capabilities) | | `vision` | `bool` | [Capabilities](#capabilities) | | `embeddings` | `bool` | [Capabilities](#capabilities) | | `images` | `bool` | [Capabilities](#capabilities) | | `meshes` | `bool` | [Capabilities](#capabilities) | | `workflow` | `string` | [ComfyUI](#comfyui) | | `checkpoint` | `string` | [ComfyUI](#comfyui) | | `output` | `string` | [ComfyUI](#comfyui) | | `inputs` | `[]Input` | [ComfyUI](#comfyui) | | `nodes` | `[]CustomNode` | [ComfyUI](#comfyui) | | `environment` | `string` | [ComfyUI](#comfyui) | | `dir` | `string` | [ComfyUIConfig](#comfyuiconfig) | | `python` | `string` | [ComfyUIConfig](#comfyuiconfig) | | `interval` | `time.Duration` | [Control](#control) | | `base_backoff` | `time.Duration` | [Control](#control) | | `max_backoff` | `time.Duration` | [Control](#control) | | `unreachable_after` | `time.Duration` | [Control](#control) | | `lost_after` | `time.Duration` | [Control](#control) | | `port_range` | `[]int` | [Control](#control) | | `mgmt_interface` | `string` | [Control](#control) | | `timezone` | `string` | [Control](#control) | | `eval_dir` | `string` | [Control](#control) | | `enabled` | `bool` | [Control](#control) | | `lease_ttl` | `time.Duration` | [Control](#control) | | `repo` | `string` | [CustomNode](#customnode) | | `commit` | `string` | [CustomNode](#customnode) | | `requirements` | `[]string` | [CustomNode](#customnode) | | `id` | `string` | [Endpoint](#endpoint) | | `url` | `string` | [Endpoint](#endpoint) | | `api_key` | `string` | [Endpoint](#endpoint) | | `models` | `[]string` | [Endpoint](#endpoint) | | `engine` | `string` | [Endpoint](#endpoint) | | `workflow` | `string` | [Endpoint](#endpoint) | | `pool` | `string` | [Endpoint](#endpoint) | | `weight` | `float64` | [Endpoint](#endpoint) | | `priority` | `int` | [Endpoint](#endpoint) | | `gpu_class` | `string` | [EngineOverride](#engineoverride) | | `nodes` | `[]string` | [EngineOverride](#engineoverride) | | `labels` | `map[string]string` | [EngineOverride](#engineoverride) | | `engines` | `map[string]string` | [EngineOverride](#engineoverride) | | `suite` | `string` | [EvalGate](#evalgate) | | `min_score` | `float64` | [EvalGate](#evalgate) | | `judge` | `string` | [EvalGate](#evalgate) | | `url` | `string` | [External](#external) | | `model` | `string` | [External](#external) | | `api_key_file` | `string` | [External](#external) | | `peer` | `string` | [External](#external) | | `api` | `string` | [External](#external) | | `workflow` | `string` | [External](#external) | | `output` | `string` | [External](#external) | | `budgets` | `[]FacilityBudget` | [Facility](#facility) | | `margin` | `float64` | [Facility](#facility) | | `gpu_watts` | `float64` | [Facility](#facility) | | `cpu_core_watts` | `float64` | [Facility](#facility) | | `node_watts` | `float64` | [Facility](#facility) | | `measured_fresh` | `time.Duration` | [Facility](#facility) | | `domain` | `string` | [FacilityBudget](#facilitybudget) | | `cap_watts` | `float64` | [FacilityBudget](#facilitybudget) | | `other_watts` | `float64` | [FacilityBudget](#facilitybudget) | | `name` | `string` | [Federation](#federation) | | `interval` | `time.Duration` | [Federation](#federation) | | `enabled` | `bool` | [FirewallSettings](#firewallsettings) | | `min_compute_capability` | `float64` | [GPUNeed](#gpuneed) | | `min_memory_gb` | `int` | [GPUNeed](#gpuneed) | | `format` | `string` | [GuardModel](#guardmodel) | | `categories` | `[]string` | [GuardModel](#guardmodel) | | `name` | `string` | [Input](#input) | | `type` | `string` | [Input](#input) | | `required` | `bool` | [Input](#input) | | `default` | `any` | [Input](#input) | | `min` | `*float64` | [Input](#input) | | `max` | `*float64` | [Input](#input) | | `choices` | `[]string` | [Input](#input) | | `max_mb` | `int` | [Input](#input) | | `doc` | `string` | [Input](#input) | | `model` | `string` | [JudgePolicy](#judgepolicy) | | `models` | `[]string` | [JudgePolicy](#judgepolicy) | | `input` | `string` | [JudgePolicy](#judgepolicy) | | `output` | `string` | [JudgePolicy](#judgepolicy) | | `categories` | `[]string` | [JudgePolicy](#judgepolicy) | | `controversial` | `string` | [JudgePolicy](#judgepolicy) | | `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) | | `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) | | `min_blocks` | `int` | [KVIndex](#kvindex) | | `max_blocks` | `int` | [KVIndex](#kvindex) | | `contact` | `string` | [Landing](#landing) | | `notice` | `string` | [Landing](#landing) | | `language` | `string` | [Landing](#landing) | | `docs_sign_in` | `bool` | [Landing](#landing) | | `id` | `string` | [License](#license) | | `name` | `string` | [License](#license) | | `url` | `string` | [License](#license) | | `notice` | `string` | [License](#license) | | `commercial` | `*bool` | [License](#license) | | `excluded_regions` | `[]string` | [License](#license) | | `revenue_cap_usd` | `int64` | [License](#license) | | `attribution` | `string` | [License](#license) | | `bin` | `string` | [LlamaCppConfig](#llamacppconfig) | | `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) | | `threads` | `int` | [LlamaCppConfig](#llamacppconfig) | | `api` | `string` | [Mirror](#mirror) | | `tls` | `tlsconf.Config` | [Mirror](#mirror) | | `registry` | `string` | [Mirror](#mirror) | | `timeout` | `time.Duration` | [Mirror](#mirror) | | `update_check` | `time.Duration` | [Mirror](#mirror) | | `name` | `string` | [Model](#model) | | `aliases` | `[]string` | [Model](#model) | | `description` | `string` | [Model](#model) | | `source` | `string` | [Model](#model) | | `profiles` | `[]Profile` | [Model](#model) | | `scaling` | `Scaling` | [Model](#model) | | `class` | `string` | [Model](#model) | | `owner` | `string` | [Model](#model) | | `external` | `*External` | [Model](#model) | | `comfyui` | `*ComfyUI` | [Model](#model) | | `license` | `*License` | [Model](#model) | | `adapters` | `[]Adapter` | [Model](#model) | | `max_loaded_adapters` | `int` | [Model](#model) | | `fallbacks` | `[]string` | [Model](#model) | | `canary` | `*Canary` | [Model](#model) | | `eval` | `*EvalGate` | [Model](#model) | | `routing_profile` | `string` | [Model](#model) | | `slo` | `SLO` | [Model](#model) | | `context_length` | `int` | [Model](#model) | | `max_output` | `int` | [Model](#model) | | `capabilities` | `Capabilities` | [Model](#model) | | `query_prefix` | `string` | [Model](#model) | | `role` | `string` | [Model](#model) | | `guard` | `*GuardModel` | [Model](#model) | | `origin` | `*Origin` | [Model](#model) | | `prometheus` | `string` | [Monitoring](#monitoring) | | `window` | `time.Duration` | [Monitoring](#monitoring) | | `timeout` | `time.Duration` | [Monitoring](#monitoring) | | `issuer` | `string` | [OIDC](#oidc) | | `client_id` | `string` | [OIDC](#oidc) | | `client_secret_file` | `string` | [OIDC](#oidc) | | `redirect_url` | `string` | [OIDC](#oidc) | | `label` | `string` | [OIDC](#oidc) | | `scopes` | `[]string` | [OIDC](#oidc) | | `groups_claim` | `string` | [OIDC](#oidc) | | `create_accounts` | `bool` | [OIDC](#oidc) | | `roles` | `map[string][]string` | [OIDC](#oidc) | | `memberships` | `map[string]string` | [OIDC](#oidc) | | `no_passwords` | `bool` | [OIDC](#oidc) | | `mirror` | `string` | [Offline](#offline) | | `checksums` | `string` | [Offline](#offline) | | `nvidia_smi` | `string` | [Options](#options) | | `ib_root` | `string` | [Options](#options) | | `prefer_nvml` | `bool` | [Options](#options) | | `skip_gpus` | `bool` | [Options](#options) | | `repo` | `string` | [Origin](#origin) | | `revision` | `string` | [Origin](#origin) | | `files` | `[]OriginFile` | [Origin](#origin) | | `previous` | `*PreviousOrigin` | [Origin](#origin) | | `path` | `string` | [OriginFile](#originfile) | | `sha256` | `string` | [OriginFile](#originfile) | | `bytes` | `int64` | [OriginFile](#originfile) | | `name` | `string` | [Pattern](#pattern) | | `regex` | `string` | [Pattern](#pattern) | | `action` | `string` | [Pattern](#pattern) | | `disabled` | `bool` | [Personal](#personal) | | `rpm` | `int` | [Personal](#personal) | | `tpm` | `int` | [Personal](#personal) | | `budget` | `float64` | [Personal](#personal) | | `period` | `string` | [Personal](#personal) | | `mode` | `string` | [Policy](#policy) | | `fail` | `string` | [Policy](#policy) | | `no_store` | `bool` | [Policy](#policy) | | `input` | `Rules` | [Policy](#policy) | | `output` | `Rules` | [Policy](#policy) | | `tools` | `Tools` | [Policy](#policy) | | `max_output_tokens` | `int` | [Policy](#policy) | | `system_prompt` | `string` | [Policy](#policy) | | `drop_client_system` | `bool` | [Policy](#policy) | | `require_capabilities` | `bool` | [Policy](#policy) | | `validate_structured` | `bool` | [Policy](#policy) | | `judge` | `*JudgePolicy` | [Policy](#policy) | | `stream` | `string` | [Policy](#policy) | | `enabled` | `bool` | [Prefix](#prefix) | | `max_nodes` | `int` | [Prefix](#prefix) | | `min_match` | `int` | [Prefix](#prefix) | | `ttl` | `time.Duration` | [Prefix](#prefix) | | `match_ratio` | `float64` | [Prefix](#prefix) | | `balance_abs` | `float64` | [Prefix](#prefix) | | `balance_rel` | `float64` | [Prefix](#prefix) | | `source` | `string` | [PreviousOrigin](#previousorigin) | | `revision` | `string` | [PreviousOrigin](#previousorigin) | | `files` | `[]OriginFile` | [PreviousOrigin](#previousorigin) | | `name` | `string` | [Profile](#profile) | | `engine` | `string` | [Profile](#profile) | | `engine_version` | `string` | [Profile](#profile) | | `gpu_class` | `string` | [Profile](#profile) | | `source` | `string` | [Profile](#profile) | | `tp` | `int` | [Profile](#profile) | | `pp` | `int` | [Profile](#profile) | | `gpus_per_node` | `int` | [Profile](#profile) | | `nodes` | `int` | [Profile](#profile) | | `args` | `[]string` | [Profile](#profile) | | `env` | `map[string]string` | [Profile](#profile) | | `min_memory_gb` | `int` | [Profile](#profile) | | `cpu_cores` | `int` | [Profile](#profile) | | `min_disk_gb` | `int` | [Profile](#profile) | | `node_selector` | `map[string]string` | [Profile](#profile) | | `power_watts` | `float64` | [Profile](#profile) | | `keep_within` | `string` | [Profile](#profile) | | `weight` | `float64` | [Profile](#profile) | | `priority` | `int` | [Profile](#profile) | | `startup_timeout` | `time.Duration` | [Profile](#profile) | | `kv_events` | `bool` | [Profile](#profile) | | `gpu` | `*GPUNeed` | [Profile](#profile) | | `comfyui` | `*ProfileComfyUI` | [Profile](#profile) | | `workflow` | `string` | [ProfileComfyUI](#profilecomfyui) | | `checkpoint` | `string` | [ProfileComfyUI](#profilecomfyui) | | `nodes` | `[]CustomNode` | [ProfileComfyUI](#profilecomfyui) | | `activation_wait` | `time.Duration` | [Router](#router) | | `scrape_interval` | `time.Duration` | [Router](#router) | | `scrape_timeout` | `time.Duration` | [Router](#router) | | `stale_after` | `time.Duration` | [Router](#router) | | `retries` | `int` | [Router](#router) | | `cooldown` | `time.Duration` | [Router](#router) | | `failure_threshold` | `int` | [Router](#router) | | `default_profile` | `string` | [Router](#router) | | `prefix` | `Prefix` | [Router](#router) | | `kv_index` | `KVIndex` | [Router](#router) | | `session_header` | `string` | [Router](#router) | | `inject_usage` | `bool` | [Router](#router) | | `tenant_label` | `string` | [Router](#router) | | `tag_labels` | `[]string` | [Router](#router) | | `max_body_bytes` | `int64` | [Router](#router) | | `batch` | `Batch` | [Router](#router) | | `cache` | `Cache` | [Router](#router) | | `trusted_proxies` | `[]string` | [Router](#router) | | `timeouts` | `Timeouts` | [Router](#router) | | `scoring` | `Scoring` | [Router](#router) | | `admission` | `Admission` | [Router](#router) | | `pii` | `map[string]string` | [Rules](#rules) | | `injection` | `string` | [Rules](#rules) | | `deny` | `[]Pattern` | [Rules](#rules) | | `max_chars` | `int` | [Rules](#rules) | | `token_file` | `string` | [SCIM](#scim) | | `python` | `string` | [SGLangConfig](#sglangconfig) | | `ttft_p99` | `time.Duration` | [SLO](#slo) | | `tpot_p99` | `time.Duration` | [SLO](#slo) | | `days` | `[]string` | [ScaleWindow](#scalewindow) | | `from` | `string` | [ScaleWindow](#scalewindow) | | `to` | `string` | [ScaleWindow](#scalewindow) | | `min` | `int` | [ScaleWindow](#scalewindow) | | `max` | `int` | [ScaleWindow](#scalewindow) | | `min` | `int` | [Scaling](#scaling) | | `max` | `int` | [Scaling](#scaling) | | `idle_timeout` | `time.Duration` | [Scaling](#scaling) | | `schedule` | `[]ScaleWindow` | [Scaling](#scaling) | | `prewarm` | `time.Duration` | [Scaling](#scaling) | | `target_waiting` | `float64` | [Scoring](#scoring) | | `target_inflight` | `float64` | [Scoring](#scoring) | | `kv_ceiling` | `float64` | [Scoring](#scoring) | | `session_load_factor` | `float64` | [Scoring](#scoring) | | `vault` | `secrets.Vault` | [Secrets](#secrets) | | `master_key` | `string` | [Secrets](#secrets) | | `listen` | `string` | [Server](#server) | | `admin_listen` | `string` | [Server](#server) | | `metrics_listen` | `string` | [Server](#server) | | `agent_listen` | `string` | [Server](#server) | | `auth` | `Auth` | [Server](#server) | | `threads` | `Threads` | [Server](#server) | | `landing` | `Landing` | [Server](#server) | | `agent_tls` | `tlsconf.Config` | [Server](#server) | | `max_connections` | `int` | [Server](#server) | | `admin_token_file` | `string` | [Server](#server) | | `admin_remote` | `bool` | [Server](#server) | | `debug_pprof` | `bool` | [Server](#server) | | `agent_insecure` | `bool` | [Server](#server) | | `monitoring` | `Monitoring` | [Server](#server) | | `tracing` | `obs.Tracing` | [Server](#server) | | `mirror` | `Mirror` | [Server](#server) | | `offline` | `Offline` | [Server](#server) | | `engines` | `map[string]string` | [Server](#server) | | `engine_overrides` | `[]EngineOverride` | [Server](#server) | | `backup_dir` | `string` | [Server](#server) | | `guard` | `guard.Config` | [Server](#server) | | `secrets` | `Secrets` | [Server](#server) | | `notify` | `notify.Config` | [Server](#server) | | `mail` | `mail.Config` | [Server](#server) | | `serving` | `ServingSettings` | [Server](#server) | | `firewall` | `FirewallSettings` | [Server](#server) | | `agents` | `AgentSettings` | [Server](#server) | | `firewall_networks` | `[]string` | [Server](#server) | | `backup_recipients` | `[]string` | [Server](#server) | | `backup_copy_to` | `[]string` | [Server](#server) | | `backup_copy_key` | `string` | [Server](#server) | | `releases` | `string` | [Server](#server) | | `router` | `Router` | [Server](#server) | | `control` | `Control` | [Server](#server) | | `models` | `[]catalog.Model` | [Server](#server) | | `facility` | `Facility` | [Server](#server) | | `federation` | `Federation` | [Server](#server) | | `endpoints` | `[]Endpoint` | [Server](#server) | | `stopped` | `bool` | [ServingSettings](#servingsettings) | | `bin` | `string` | [SimConfig](#simconfig) | | `python` | `string` | [TRTLLMConfig](#trtllmconfig) | | `store` | `string` | [Threads](#threads) | | `retain_hours` | `int` | [Threads](#threads) | | `dial` | `time.Duration` | [Timeouts](#timeouts) | | `response_header` | `time.Duration` | [Timeouts](#timeouts) | | `generation` | `time.Duration` | [Timeouts](#timeouts) | | `first_token` | `time.Duration` | [Timeouts](#timeouts) | | `idle` | `time.Duration` | [Timeouts](#timeouts) | | `total` | `time.Duration` | [Timeouts](#timeouts) | | `allow` | `[]string` | [Tools](#tools) | | `deny` | `[]string` | [Tools](#tools) | | `on_denied` | `string` | [Tools](#tools) | | `max_calls_per_turn` | `int` | [Tools](#tools) | | `validate_arguments` | `bool` | [Tools](#tools) | | `endpoint` | `string` | [Tracing](#tracing) | | `sample_ratio` | `float64` | [Tracing](#tracing) | | `bin` | `string` | [VLLMConfig](#vllmconfig) | | `launch_mode` | `string` | [VLLMConfig](#vllmconfig) | | `ray_bin` | `string` | [VLLMConfig](#vllmconfig) | | `addr` | `string` | [Vault](#vault) | | `token_file` | `string` | [Vault](#vault) | | `role_id_file` | `string` | [Vault](#vault) | | `secret_id_file` | `string` | [Vault](#vault) | | `ca_file` | `string` | [Vault](#vault) | | `namespace` | `string` | [Vault](#vault) | | `url` | `string` | [Webhook](#webhook) | | `format` | `string` | [Webhook](#webhook) | | `events` | `[]string` | [Webhook](#webhook) | | `secret_file` | `string` | [Webhook](#webhook) | | `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) | | `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) | | `per_image` | `int64` | [accounting.Price](#accountingprice) | | `per_mesh` | `int64` | [accounting.Price](#accountingprice) | | `name` | `string` | [admission.Band](#admissionband) | | `priority` | `int` | [admission.Band](#admissionband) | | `saturation_ceiling` | `float64` | [admission.Band](#admissionband) | | `queue_ttl` | `time.Duration` | [admission.Band](#admissionband) | | `bands` | `[]Band` | [admission.Config](#admissionconfig) | | `default_band` | `string` | [admission.Config](#admissionconfig) | | `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) | | `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) | | `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) | | `interval` | `time.Duration` | [admission.Config](#admissionconfig) | | `node_id` | `string` | [agent.Config](#agentconfig) | | `server` | `string` | [agent.Config](#agentconfig) | | `servers` | `[]string` | [agent.Config](#agentconfig) | | `hostname` | `string` | [agent.Config](#agentconfig) | | `mgmt_addr` | `string` | [agent.Config](#agentconfig) | | `labels` | `map[string]string` | [agent.Config](#agentconfig) | | `mgmt_interface` | `string` | [agent.Config](#agentconfig) | | `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) | | `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) | | `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) | | `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) | | `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) | | `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) | | `model_dirs` | `[]string` | [agent.Config](#agentconfig) | | `engines_dir` | `string` | [agent.Config](#agentconfig) | | `weights_cache` | `string` | [agent.Config](#agentconfig) | | `discovery` | `discovery.Options` | [agent.Config](#agentconfig) | | `engines` | `engines.Config` | [agent.Config](#agentconfig) | | `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) | | `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) | | `name` | `string` | [config.Band](#configband) | | `priority` | `int` | [config.Band](#configband) | | `saturation_ceiling` | `float64` | [config.Band](#configband) | | `queue_ttl` | `time.Duration` | [config.Band](#configband) | | `prompt_per_1k` | `int64` | [config.Price](#configprice) | | `completion_per_1k` | `int64` | [config.Price](#configprice) | | `per_image` | `int64` | [config.Price](#configprice) | | `per_mesh` | `int64` | [config.Price](#configprice) | | `disabled` | `[]string` | [engines.Config](#enginesconfig) | | `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) | | `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) | | `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) | | `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) | | `sim` | `SimConfig` | [engines.Config](#enginesconfig) | | `comfyui` | `ComfyUIConfig` | [engines.Config](#enginesconfig) | | `enabled` | `*bool` | [guard.Config](#guardconfig) | | `default` | `[]string` | [guard.Config](#guardconfig) | | `policies` | `map[string]Policy` | [guard.Config](#guardconfig) | | `smtp` | `string` | [mail.Config](#mailconfig) | | `from` | `string` | [mail.Config](#mailconfig) | | `username` | `string` | [mail.Config](#mailconfig) | | `password_file` | `string` | [mail.Config](#mailconfig) | | `base_url` | `string` | [mail.Config](#mailconfig) | | `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) | | `host` | `string` | [notify.Config](#notifyconfig) | | `work_dir` | `string` | [supervisor.Config](#supervisorconfig) | | `log_lines` | `int` | [supervisor.Config](#supervisorconfig) | | `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) | | `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) | | `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) | | `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) | | `run_as` | `string` | [supervisor.Config](#supervisorconfig) | | `ca` | `string` | [tlsconf.Config](#tlsconfconfig) | | `cert` | `string` | [tlsconf.Config](#tlsconfconfig) | | `key` | `string` | [tlsconf.Config](#tlsconfconfig) | | `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) | ## Adapter ### `name` `string` Name is what clients ask for, like a model name. ### `source` `string` Source is the adapter: a GGUF file for llama.cpp; for vLLM a PEFT adapter directory (a mirror version ending in "/", a path on the node, or a Hugging Face repository id). Fetched by each node as the model's weights are. ### `scale` `float64` Scale is how strongly it is applied; zero takes 1. ### `owner` `string` Owner is the part of the tenancy tree the adapter belongs to — org:ID, team:ID, project:ID, user:ID or key:ID — and only callers in it see it: to anyone else it is a model that does not exist. An adapter trained on an organisation's documents carries some of what they say in its weights. Empty is every caller the model allows. ## Admission ### `enabled` `bool` Enabled turns the layer on. Off, every request goes straight through, which is the v0.2.0 behaviour. ### `bands` `[]Band` ### `default_band` `string` ### `band_header` `string` BandHeader lets a client declare its band. Until authentication lands in v1.0.0 this is trusted, so the port must not be exposed to callers who should not be able to promote themselves. ### `queue_ttl` `time.Duration` ### `queue_capacity_per_band` `int` ### `queue_capacity_total` `int` QueueCapacityTotal bounds everything queued across bands. Reached, a request is not refused while a lower band has any queued: the newest of the lowest band is turned away instead, with a 503 and Retry-After. Zero takes half of all the bands' capacities together. ### `interval` `time.Duration` ## AgentSettings ### `default` `map[string]any` Default is agent.yaml's keys for every node. ### `nodes` `map[string]map[string]any` Nodes are agent.yaml's keys for one node each, by node id, over Default. ## Auth ### `enabled` `bool` ### `store` `string` Store is the SQLite file holding keys, tenancy and usage. ### `cache_ttl` `time.Duration` CacheTTL bounds how long a revoked key keeps working, traded against a database read per key per TTL. ### `negative_ttl` `time.Duration` NegativeTTL is how long a rejected key is remembered as rejected, so a client looping with a bad key cannot become a database read per request. ### `default_rpm` `int` DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to. Unlimited used to be the default, and a project created without limits is the easy path — the "default" project on this cluster still is one. The first sign of that is the bill or the outage. Zero keeps the old behaviour for anyone who wants it, but it should be a choice rather than an omission. A fallback, not a ceiling: an explicit limit at any scope wins, above or below. ### `default_tpm` `int` ### `signup_open` `bool` SignupOpen serves the signup form. Accounts it creates are pending until an operator approves them, so this is not a way in — but turning it off means accounts are made with `llmgw users create` instead. ### `oidc` `OIDC` OIDC is sign-in through an identity provider. Takes a [OIDC](#oidc) block. ### `scim` `SCIM` SCIM is account provisioning by the identity provider. Takes a [SCIM](#scim) block. ### `max_key_ttl` `time.Duration` MaxKeyTTL is how long any API key may live; a key asked for with no expiry is given this. Organizations, teams and projects can set shorter ones (llmgw keys ttl). Zero is no limit. ### `personal` `Personal` Personal is what a person may do on their own, outside any project: make personal keys and use the chat page personally, within these limits. Takes a [Personal](#personal) block. ### `pricing` `map[string]Price` Pricing is what each model costs, for budgets. A model absent from the table costs whatever DefaultPrice says, which is nothing unless set. ### `default_price` `Price` Takes a [config.Price](#configprice) block. ### `flush_interval` `time.Duration` FlushInterval and BatchSize decide when usage records are written. ### `batch_size` `int` ### `buffer_size` `int` BufferSize bounds the queue. Beyond it records are dropped rather than blocking a request: losing accounting is bad, turning a stalled database into an outage is worse. ### `usage_retention` `time.Duration` UsageRetention is how long the record of each request is kept: who sent it, which replica answered, how long it queued — what `ctl request` reads. Zero keeps it for ever. Totals, charts and budgets are read from five-minute sums kept beside the records, which are not pruned, so this bounds the table that grows by a row a request without changing any figure a page shows. At least a day when set. ## Batch ### `max_file_bytes` `int64` MaxFileBytes bounds one uploaded file; 100 MB unless set. ### `max_lines` `int` MaxLines bounds the requests in one batch; 50 000 unless set. ### `concurrency` `int` Concurrency is how many of one batch's requests are in flight at once; 4 unless set. They queue in the lowest admission band, behind every interactive request. ### `retain_days` `int` RetainDays is how long a batch's answer and error files are kept; 30 unless set. Uploaded files are kept until deleted. ## Cache ### `max_bytes` `int64` MaxBytes bounds every project's answers together; 128 MB unless set. Past it, the answers used longest ago go first. ### `max_entry_bytes` `int64` MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB unless set. ## Canary ### `model` `string` Model is the candidate: a catalog model, usually this one at newer weights, which clients cannot name themselves. ### `percent` `int` Percent of requests go to it, 0 to 100. ### `min_requests` `int` The candidate is taken out — rolled back — when, over at least MinRequests of its own, its error rate is MaxErrorIncrease above this model's (0.05 is five points), or its mean time to the first token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5. ### `max_error_increase` `float64` ### `max_ttft_ratio` `float64` ## Capabilities ### `tools` `bool` ### `vision` `bool` ### `embeddings` `bool` ### `images` `bool` Images is a model that makes pictures, asked for at /v1/images/generations. Implied by an external model whose API is ComfyUI's. ### `meshes` `bool` Meshes is a model that makes 3D models from a picture, asked for as a job at /v1/3d/generations. Implied by a ComfyUI model whose output is meshes. ## ComfyUI ### `workflow` `string` Workflow is a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane, in ComfyUI's API format. ### `checkpoint` `string` Checkpoint is the file the workflow's {{checkpoint}} names: the source's own name unless set, which a source that is a set of files must. ### `output` `string` Output is what the workflow makes: pictures (the default), asked for at /v1/images/generations; meshes, asked for as a job at /v1/3d/generations from a picture; or video, audio or files. Every kind is asked for as a job at /v1/workflows/runs too. ### `inputs` `[]Input` Inputs are what the workflow takes from a request at /v1/workflows/runs, each the placeholder of its name. ### `nodes` `[]CustomNode` Nodes are custom nodes the workflow needs, put in each replica's own custom_nodes at the commit named. ### `environment` `string` Environment names the models that may share one ComfyUI replica: every model of an environment agrees on the custom nodes they both name, so one replica can carry all of them and ComfyUI moves between their weights itself. Empty is an environment of the model alone. ## ComfyUIConfig ### `dir` `string` Dir is ComfyUI's directory, where main.py is. Readable by the user engines run as. ### `python` `string` Python is the interpreter of its environment; Dir's .venv when unset. ## Control ### `interval` `time.Duration` Interval is how often desired and actual state are compared. ### `base_backoff` `time.Duration` BaseBackoff and MaxBackoff bound how fast a failing model is retried. ### `max_backoff` `time.Duration` ### `unreachable_after` `time.Duration` UnreachableAfter and LostAfter stage node expiry. They are deliberately far apart: a network blip must not tear down replicas that still serve, and rebuilding a multi-node replica costs tens of minutes of loading. ### `lost_after` `time.Duration` ### `port_range` `[]int` PortRange is the per-node range for engine HTTP and dist-init ports. ### `mgmt_interface` `string` MgmtInterface is the ethernet interface NCCL bootstraps over. ### `timezone` `string` Timezone is where models' scaling schedules are read, an IANA name like Europe/Istanbul; empty is the control plane's own. ### `eval_dir` `string` EvalDir is where quality suites are read from by name, for `llmgw ctl eval MODEL --suite NAME`, a model's eval gate and the console. The suites the binary carries (arithmetic, json-format) are found when it has none of the name; a suite here of the same name replaces one. ### `enabled` `bool` Enabled runs the control plane even when configuration declares no models. Without it, a deployment whose catalog lives entirely in the database cannot start: control plane mode is inferred from the config file, so an empty models block means "router only", the process refuses to start for want of anything to serve, and the models it would have loaded are never read. That is a cluster that cannot be managed at runtime because it is managed at runtime. AgentListen cannot carry this signal — it has a default, so every deployment would become a control plane. ### `lease_ttl` `time.Duration` LeaseTTL is how long this control plane's claim to be the one placing work stands without being renewed. Only meaningful with a shared store, where a second control plane can see the claim. Long enough to survive a slow query or a brief network stall, short enough that a standby takes over before anyone has finished reading the alert. Zero takes the default. ## CustomNode ### `repo` `string` Repo is the repository, https only. ### `commit` `string` Commit is the full commit the node is used at: a branch moves, and a node that changed under a model is a model that changed unasked. ### `requirements` `[]string` Requirements are the PyPI packages the node needs, by name and version range only. They are installed beside the replica, never into ComfyUI's environment: a package named here at the range named, and every other one at the version ComfyUI already runs. One named here is first on ComfyUI's path, so its range must suit ComfyUI too. ## Endpoint ### `id` `string` ### `url` `string` ### `api_key` `string` APIKey is sent to the engine as a bearer token, for an engine started with one. ### `models` `[]string` ### `engine` `string` Engine names the inference engine, which decides what its metrics are called. Empty means vLLM's names. ### `workflow` `string` Workflow is the ComfyUI workflow for engine: comfyui, a built-in's name or its JSON; flux-schnell unless set. ### `pool` `string` ### `weight` `float64` ### `priority` `int` ## EngineOverride ### `gpu_class` `string` GPUClass picks nodes with a GPU of this class ("v100-32g"). ### `nodes` `[]string` Nodes picks nodes by id. ### `labels` `map[string]string` Labels picks nodes carrying all of these. ### `engines` `map[string]string` Engines are laid over the cluster's: a version, or "" for none. ## EvalGate ### `suite` `string` Suite is a suite's name in control.eval_dir, or one the binary carries (arithmetic, json-format). ### `min_score` `float64` MinScore is the score, 0 to 1, to pass at; zero is the suite's own threshold. ### `judge` `string` Judge grades the suite's rubric cases, instead of the judge it names. ## External ### `url` `string` URL is the provider's base address, without /v1: https://api.openai.com. ### `model` `string` Model is what the provider calls it; requests are rewritten to it. ### `api_key_file` `string` APIKeyFile holds the provider's key, on the control plane. A file rather than the key itself, so the key is in neither the catalog, the store nor the audit log; empty for a provider that asks for none. ### `peer` `string` Peer names the federated llmgw cluster this model is offered by, for a model that federation added (`llmgw ctl federation add`) rather than one written by hand. Such a model is named PEER/MODEL, comes and goes with the peer's own list, and is sent only while the peer is up. ### `api` `string` API is what the provider speaks: OpenAI's, unless this says comfyui — a ComfyUI server, asked for pictures at /v1/images/generations and sent the workflow below with the request's prompt in it. Model is then the checkpoint file the workflow loads, as ComfyUI lists it. ### `workflow` `string` Workflow is the ComfyUI workflow, in its API format: a built-in's name (flux-schnell, the default, or flux-dev) or a JSON file on the control plane. ### `output` `string` Output is what it makes: pictures, or meshes (see ComfyUI.Output). ## Facility ### `budgets` `[]FacilityBudget` Budgets are the domains with a power cap. ### `margin` `float64` Margin is the share of every cap kept free, 0.1 for ten percent. Zero keeps none, for caps already given derated. ### `gpu_watts` `float64` GPUWatts is a card's draw when it does not report its power limit: an agent without nvidia-smi's power fields, or a node that has not sent a heartbeat yet. Zero counts such a card as drawing nothing. ### `cpu_core_watts` `float64` CPUCoreWatts is the draw of a core reserved by a replica with no GPU (cpu_cores). Zero counts such replicas as drawing nothing. ### `node_watts` `float64` NodeWatts is what a node draws besides its cards and reserved cores — idle CPUs, memory, fans, disks — counted for every node in a capped domain, busy or not. ### `measured_fresh` `time.Duration` MeasuredFresh is how old a node's power reading may be and still be counted against its domains. Zero takes two minutes. ## FacilityBudget ### `domain` `string` Domain is level:name, as the nodes' labels give it: rack:r12-07, pdu:r12-07-a, row:12. ### `cap_watts` `float64` CapWatts is what the domain may draw. For a dual-fed rack it is what one feed carries alone, derated as the facility derates it (usually 80%), since that is what the rack has to survive. ### `other_watts` `float64` OtherWatts is drawn in the domain by what llmgw does not place on: switches, storage, machines outside the cluster. ## Federation ### `name` `string` Name is this cluster as a peer reads it, in the federated_from tag sent with a request to a peer set to receive tags (`ctl federation add --send-tags`). Empty is this machine's host name. ### `interval` `time.Duration` Interval is how often each peer is asked whether it is ready, timed, and what it offers. A peer that does not answer is taken out of routing until it does. Zero is 10s. ## FirewallSettings ### `enabled` `bool` Enabled keeps llmgw's internal ports — engines, node exporter, the agents' stream, the mirror, the store — to the cluster's networks on every machine. False removes llmgw's nftables table everywhere, for a site whose own firewall does that job; the rules are kept, and true puts them back. Changed in the cluster's settings, it is in force on every node and control plane within seconds. ## GPUNeed ### `min_compute_capability` `float64` MinComputeCapability is the oldest architecture that runs it: 7.5 is Turing, 8.0 Ampere (bf16, flash-attention 2), 8.9 Ada (fp8), 9.0 Hopper. ### `min_memory_gb` `int` MinMemoryGB is the card's own memory it needs. ## GuardModel ### `format` `string` Format is qwen3guard (Qwen3Guard-Gen: "Safety: Safe", "Unsafe" or "Controversial", and the categories), llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and the categories), shieldgemma (Google's ShieldGemma: Yes or No), or json ({"safe": false, "categories": ["..."]}). ### `categories` `[]string` Categories the model is asked about, for shieldgemma — each a policy sentence — and the names reported for llamaguard3's S1, S2, …. ## Input ### `name` `string` Name is the placeholder's, and the request's key. ### `type` `string` Type is one of text, int, float, bool, choice, image, mask, audio, video, file. The last five are files, sent as base64, a data URL, or the id of a file uploaded to /v1/files, and put in ComfyUI's input directory; the placeholder gets the name they were given there. ### `required` `bool` Required inputs must be in every request; the others take Default, or the workflow's own {{name|default}}. ### `default` `any` ### `min` `*float64` Min and Max bound a number; for text, Max bounds its length. ### `max` `*float64` ### `choices` `[]string` Choices are what a choice may be. ### `max_mb` `int` MaxMB bounds a file input; zero is the type's own bound. ### `doc` `string` Doc says what the input is for, to whoever fills it in. ## JudgePolicy ### `model` `string` Model is a catalog model with role: guard. ### `models` `[]string` Models are judges asked in turn, instead of Model: the first that answers gives the verdict. A judge on a GPU first and one on CPUs after it is the GPU's speed while it is there, and the CPUs' when it is not — busy, failed, or the node down. ### `input` `string` Input and Output are what an unsafe verdict does: flag or block. Empty does not ask the judge about that direction. ### `output` `string` ### `categories` `[]string` Categories, when set, are the only ones that count: Llama Guard's codes (S1, S10) or the names the model reports. ### `controversial` `string` Controversial is what Qwen3Guard's third verdict counts as: safe (the default) or unsafe. ### `latency_budget` `time.Duration` LatencyBudget is how long each judge is waited for before the next is asked, or, after the last, the policy's fail applies. Zero is 250ms. ## KVIndex ### `tokenize_timeout` `time.Duration` TokenizeTimeout bounds the call that turns a request into token ids, made to one of the model's replicas before the request is placed. Past it the request is routed as tier 1 would. Zero takes 250ms. ### `min_blocks` `int` MinBlocks is the fewest leading blocks a replica must hold to be preferred for them; fewer is coincidence — a chat template's opening tokens — rather than a prefix worth following. Zero takes 4, which is 64 tokens at vLLM's usual block size of 16. ### `max_blocks` `int` MaxBlocks bounds how many of a prompt's blocks are hashed and looked up. Beyond it a longer match still helps, but the difference no longer decides anything. Zero takes 4096. ## Landing ### `contact` `string` Contact is where someone without a key should go — an address, a URL, a channel name. Rendered as a link when it is one, as text otherwise. ### `notice` `string` Notice is an optional line of your own, for whatever the reader needs to know that this page cannot guess. ### `language` `string` Language is what the web portal speaks to someone who has not chosen a language with the switcher at the foot of every page and whose browser asks for none it speaks: "en" or "tr". Empty is English. ### `docs_sign_in` `bool` DocsSignIn serves the docs portal at /docs only to someone signed in. Off, the portal is public like this page: its client guide is for the person who has no account yet, and the operator documents describe llmgw in general — no address, model or key of this installation — which the project's repository publishes anyway. On, a visitor is sent to sign in, and without accounts the portal needs an API key. ## License ### `id` `string` ID names the licence for a machine: an SPDX identifier where there is one (MIT, Apache-2.0, CC-BY-NC-4.0), else a lowercase name of the licence's own (stability-community, tencent-hunyuan-community). What a person's or a site's choice of licences matches on. Name is for people. ### `name` `string` Name is the licence's own: MIT, Apache-2.0. ### `url` `string` ### `notice` `string` Notice is what must accompany what was made: attribution, terms on the output, what the weights were trained on. ### `commercial` `*bool` Commercial says whether what is made may be used commercially: nil is not said, which is not the same as yes. ### `excluded_regions` `[]string` ExcludedRegions are where the licence grants nothing: ISO 3166 codes, and EU for the union (Hunyuan's: EU, UK, KR). ### `revenue_cap_usd` `int64` RevenueCapUSD is the yearly revenue above which the licence asks for another agreement (Stability's community licence: 1000000). Zero is none. ### `attribution` `string` Attribution is what a product using the output must show: "Powered by Stability AI". ## LlamaCppConfig ### `bin` `string` Bin is the llama-server binary. It is the server, not the CLI: the gateway talks OpenAI over HTTP and needs the one that serves it. Empty means the agent installs llama.cpp itself, at the version the control plane asks for. Setting it opts out of that. ### `parallel` `int` Parallel is how many requests a server handles at once, when a profile does not say. llama.cpp defaults to one, which would make every second request wait behind the first no matter what the router decided. ### `threads` `int` Threads is how many CPU threads a shard with no GPU computes on. Zero takes half this machine's CPUs. Not all of them, which is llama.cpp's own default: two busy servers that each spin on every CPU fight for them, and on an 8-vCPU node that made 7B generation fall from 10 tokens a second to under 1 each. At half, one server alone is about a third slower and two together keep 5.5 each. A node that only ever runs one CPU replica does better with all of them; say so here. ## Mirror ### `api` `string` API is the mirror's management address, scheme included, for example https://models.internal:8082. Empty means no mirror is configured. The management port, not the one nodes fetch bytes from. Those are two listeners on the mirror precisely so this one can be locked down without re-certifying every agent's fetcher. ### `tls` `tlsconf.Config` TLS authenticates to the mirror's management API, with the same CA the agents use. Empty leaves the connection in plain HTTP, which is only acceptable on a network where nothing untrusted can reach that port — the mirror says as much in its own log when it starts that way. Takes a [tlsconf.Config](#tlsconfconfig) block. ### `registry` `string` Registry is the model registry the mirror page browses, empty for the public Hugging Face. Set it to an internal mirror of the registry where the control plane cannot reach the internet — or leave the whole mirror block out, which turns the explorer off rather than giving it a search box that can only time out. ### `timeout` `time.Duration` Timeout bounds a management call. It does not bound a download: the mirror answers a fetch as soon as it has started one, and the bytes keep arriving long after this deadline has passed. ### `update_check` `time.Duration` UpdateCheck is how often the registry is asked whether the files of the models served from it have changed there: a repository can publish new weights under the same name. Only told, never applied — moving a model to them is `llmgw ctl model update`. 0 never asks. ## Model ### `name` `string` ### `aliases` `[]string` ### `description` `string` Description is what this model is for, in a line. Written by whoever added it, because nobody else can say it. A registry offers a pipeline tag and a wall of README, and neither answers the question a person picking from a list is asking: which of these nine names should I send this request to. It reaches clients through /v1/models, which is the only view most of them ever see. A gateway serving nine models by name alone makes that choice by guesswork. ### `source` `string` Source is the weights path or a HuggingFace id. ### `profiles` `[]Profile` ### `scaling` `Scaling` Takes a [Scaling](#scaling) block. ### `class` `string` Class is how much this model's capacity is worth against another's, when there is not enough for both: critical, standard or opportunistic. Empty is standard, or opportunistic for a model loaded on demand. See EffectiveClass. ### `owner` `string` Owner is the part of the tenancy tree whose capacity quota this model counts against, written org:ID, team:ID or project:ID. Empty is shared: held by the cluster, counted against nobody. ### `external` `*External` External is a model served by someone else — an OpenAI-compatible API outside the cluster — rather than by replicas placed here. Such a model has no profiles and nothing is placed for it; the router sends its requests to the provider. Most useful named in another model's fallbacks: used only when the cluster has no capacity left. Its prompts leave the cluster, so which keys may name it is worth an allowlist. Takes a [External](#external) block. ### `comfyui` `*ComfyUI` ComfyUI is how a model whose profiles run engine comfyui makes its pictures: the workflow, and the file it loads. Takes a [ComfyUI](#comfyui) block. ### `license` `*License` License is the terms of what the model makes, for a model whose output somebody ships: shown with every result and written into a mesh. Takes a [License](#license) block. ### `adapters` `[]Adapter` Adapters are LoRA adapters served on this model's replicas, each asked for by its own name as though it were a model. Every replica loads every adapter, applying one only to a request that names it — unless MaxLoadedAdapters is set. ### `max_loaded_adapters` `int` MaxLoadedAdapters, above zero, has each replica hold at most this many adapters at once, loaded when a request first asks for one and unloaded least recently used first to make room. The router sends a request for an adapter to a replica that already holds it while that one has room to take it. vLLM profiles only: llama-server loads adapters at start and cannot load one afterwards. Zero loads every adapter on every replica at start. ### `fallbacks` `[]string` ### `canary` `*Canary` Canary sends a share of this model's requests to another catalog model — the same model at new weights — to compare the two before moving to it (`llmgw ctl model canary`). Takes a [Canary](#canary) block. ### `eval` `*EvalGate` Eval is a quality suite new weights must pass before they replace this model's: `llmgw ctl model update` and `model promote` run it against them first, and stop below its score. Takes a [EvalGate](#evalgate) block. ### `routing_profile` `string` RoutingProfile selects how the router scores endpoints for this model. ### `slo` `SLO` Takes a [SLO](#slo) block. ### `context_length` `int` ContextLength is how many tokens one request may use, prompt and completion together, and MaxOutput bounds the completion alone. Zero means unknown, which is what every model was until now. The gateway knew neither, so an over-long prompt was not refused: it was admitted, given a replica, and failed at the engine, and a client sizing its prompts from /v1/models had nothing to read. Worse, the number is not the one an operator writes in the engine's arguments — llama.cpp divides --ctx-size across its parallel slots, so a pool of 8192 across four slots is 2048 a request, and nothing said so. ### `max_output` `int` ### `capabilities` `Capabilities` Capabilities is what this model can do beyond plain completion. Takes a [Capabilities](#capabilities) block. ### `query_prefix` `string` QueryPrefix is put before a question an embedding model embeds for a knowledge-base search; documents are embedded without it. Models trained with an instruction on the query side want it, Qwen3-Embedding "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: " for one. ### `role` `string` Role "guard" is a model that judges the gateway's traffic for the guardrails rather than serving clients: hidden from /v1/models and the chat page, refused to any key that names it, critical by default, and always running. Guard says how its answers are read. ### `guard` `*GuardModel` Takes a [GuardModel](#guardmodel) block. ### `origin` `*Origin` Origin is the registry repository the weights came from, at which commit, and each file's checksum there. Written when a model is served from the registry; for one that was not, worked out from the mirror's record of where it fetched the file. It is what lets the control plane say that the repository has published different weights since, and move the model to them. Takes a [Origin](#origin) block. ## Monitoring ### `prometheus` `string` Prometheus is the server's address, for example http://monitoring.internal:9090. Empty leaves the charts off the dashboard rather than showing an error where one should be. ### `window` `time.Duration` Window is how far back the charts look. Zero takes an hour. ### `timeout` `time.Duration` Timeout bounds one query. It runs while somebody waits for a page, so a monitoring system that is slow today is skipped rather than waited for. ## OIDC ### `issuer` `string` Issuer is the provider's address; its /.well-known/openid-configuration says the rest. ### `client_id` `string` ClientID and the file holding its secret, as registered with the provider. ### `client_secret_file` `string` ### `redirect_url` `string` RedirectURL is where the provider sends people back: https:///login/oidc/callback. Empty works it out from the request, which is right unless a proxy changes the address. ### `label` `string` Label is what the sign-in button says: "Sign in with