Ilker ManapandClaude Opus 5.5
40e9fd2ab1
ctl federation add/list/remove/sync keeps peers in the store (migration 0029); their /v1/models are offered here as external models named PEER/MODEL and kept in step, a model still named by a fallback kept withdrawn. Peers are checked at /ready, timed, and taken out of routing while down. Requests to a peer carry x-llmgw-hop and are refused (508) when they come back to a cluster they passed through; residency and no_store keep requests off peers; X-Request-Id and, with --send-tags, attribution tags go along. ctl model fallbacks sets overflow to a peer. Console view, metrics, admin API routes and docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2909 lines
70 KiB
Markdown
2909 lines
70 KiB
Markdown
# Configuration reference
|
|
|
|
Every setting llmgw reads from a file, with the type it takes and what it is
|
|
for.
|
|
|
|
**Generated from the source.** The descriptions are the comments on the fields
|
|
themselves, so this cannot describe a setting that does not exist or miss one
|
|
that does — and a test fails when it has drifted. Do not edit it; edit the
|
|
comment and run `go generate ./...`.
|
|
|
|
Where a setting takes another block, the block is listed under its own heading
|
|
rather than repeated.
|
|
|
|
## Every key
|
|
|
|
| Key | Type | In |
|
|
|---|---|---|
|
|
| `name` | `string` | [Adapter](#adapter) |
|
|
| `source` | `string` | [Adapter](#adapter) |
|
|
| `scale` | `float64` | [Adapter](#adapter) |
|
|
| `enabled` | `bool` | [Admission](#admission) |
|
|
| `bands` | `[]Band` | [Admission](#admission) |
|
|
| `default_band` | `string` | [Admission](#admission) |
|
|
| `band_header` | `string` | [Admission](#admission) |
|
|
| `queue_ttl` | `time.Duration` | [Admission](#admission) |
|
|
| `queue_capacity_per_band` | `int` | [Admission](#admission) |
|
|
| `queue_capacity_total` | `int` | [Admission](#admission) |
|
|
| `interval` | `time.Duration` | [Admission](#admission) |
|
|
| `enabled` | `bool` | [Auth](#auth) |
|
|
| `store` | `string` | [Auth](#auth) |
|
|
| `cache_ttl` | `time.Duration` | [Auth](#auth) |
|
|
| `negative_ttl` | `time.Duration` | [Auth](#auth) |
|
|
| `default_rpm` | `int` | [Auth](#auth) |
|
|
| `default_tpm` | `int` | [Auth](#auth) |
|
|
| `signup_open` | `bool` | [Auth](#auth) |
|
|
| `oidc` | `OIDC` | [Auth](#auth) |
|
|
| `scim` | `SCIM` | [Auth](#auth) |
|
|
| `max_key_ttl` | `time.Duration` | [Auth](#auth) |
|
|
| `personal` | `Personal` | [Auth](#auth) |
|
|
| `pricing` | `map[string]Price` | [Auth](#auth) |
|
|
| `default_price` | `Price` | [Auth](#auth) |
|
|
| `flush_interval` | `time.Duration` | [Auth](#auth) |
|
|
| `batch_size` | `int` | [Auth](#auth) |
|
|
| `buffer_size` | `int` | [Auth](#auth) |
|
|
| `usage_retention` | `time.Duration` | [Auth](#auth) |
|
|
| `max_file_bytes` | `int64` | [Batch](#batch) |
|
|
| `max_lines` | `int` | [Batch](#batch) |
|
|
| `concurrency` | `int` | [Batch](#batch) |
|
|
| `retain_days` | `int` | [Batch](#batch) |
|
|
| `max_bytes` | `int64` | [Cache](#cache) |
|
|
| `max_entry_bytes` | `int64` | [Cache](#cache) |
|
|
| `model` | `string` | [Canary](#canary) |
|
|
| `percent` | `int` | [Canary](#canary) |
|
|
| `min_requests` | `int` | [Canary](#canary) |
|
|
| `max_error_increase` | `float64` | [Canary](#canary) |
|
|
| `max_ttft_ratio` | `float64` | [Canary](#canary) |
|
|
| `tools` | `bool` | [Capabilities](#capabilities) |
|
|
| `vision` | `bool` | [Capabilities](#capabilities) |
|
|
| `embeddings` | `bool` | [Capabilities](#capabilities) |
|
|
| `interval` | `time.Duration` | [Control](#control) |
|
|
| `base_backoff` | `time.Duration` | [Control](#control) |
|
|
| `max_backoff` | `time.Duration` | [Control](#control) |
|
|
| `unreachable_after` | `time.Duration` | [Control](#control) |
|
|
| `lost_after` | `time.Duration` | [Control](#control) |
|
|
| `port_range` | `[]int` | [Control](#control) |
|
|
| `mgmt_interface` | `string` | [Control](#control) |
|
|
| `timezone` | `string` | [Control](#control) |
|
|
| `eval_dir` | `string` | [Control](#control) |
|
|
| `enabled` | `bool` | [Control](#control) |
|
|
| `lease_ttl` | `time.Duration` | [Control](#control) |
|
|
| `id` | `string` | [Endpoint](#endpoint) |
|
|
| `url` | `string` | [Endpoint](#endpoint) |
|
|
| `api_key` | `string` | [Endpoint](#endpoint) |
|
|
| `models` | `[]string` | [Endpoint](#endpoint) |
|
|
| `engine` | `string` | [Endpoint](#endpoint) |
|
|
| `pool` | `string` | [Endpoint](#endpoint) |
|
|
| `weight` | `float64` | [Endpoint](#endpoint) |
|
|
| `priority` | `int` | [Endpoint](#endpoint) |
|
|
| `gpu_class` | `string` | [EngineOverride](#engineoverride) |
|
|
| `nodes` | `[]string` | [EngineOverride](#engineoverride) |
|
|
| `labels` | `map[string]string` | [EngineOverride](#engineoverride) |
|
|
| `engines` | `map[string]string` | [EngineOverride](#engineoverride) |
|
|
| `suite` | `string` | [EvalGate](#evalgate) |
|
|
| `min_score` | `float64` | [EvalGate](#evalgate) |
|
|
| `judge` | `string` | [EvalGate](#evalgate) |
|
|
| `url` | `string` | [External](#external) |
|
|
| `model` | `string` | [External](#external) |
|
|
| `api_key_file` | `string` | [External](#external) |
|
|
| `peer` | `string` | [External](#external) |
|
|
| `name` | `string` | [Federation](#federation) |
|
|
| `interval` | `time.Duration` | [Federation](#federation) |
|
|
| `format` | `string` | [GuardModel](#guardmodel) |
|
|
| `categories` | `[]string` | [GuardModel](#guardmodel) |
|
|
| `model` | `string` | [JudgePolicy](#judgepolicy) |
|
|
| `input` | `string` | [JudgePolicy](#judgepolicy) |
|
|
| `output` | `string` | [JudgePolicy](#judgepolicy) |
|
|
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
|
|
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
|
|
| `contact` | `string` | [Landing](#landing) |
|
|
| `notice` | `string` | [Landing](#landing) |
|
|
| `docs_sign_in` | `bool` | [Landing](#landing) |
|
|
| `bin` | `string` | [LlamaCppConfig](#llamacppconfig) |
|
|
| `parallel` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
|
| `threads` | `int` | [LlamaCppConfig](#llamacppconfig) |
|
|
| `api` | `string` | [Mirror](#mirror) |
|
|
| `tls` | `tlsconf.Config` | [Mirror](#mirror) |
|
|
| `registry` | `string` | [Mirror](#mirror) |
|
|
| `timeout` | `time.Duration` | [Mirror](#mirror) |
|
|
| `update_check` | `time.Duration` | [Mirror](#mirror) |
|
|
| `name` | `string` | [Model](#model) |
|
|
| `aliases` | `[]string` | [Model](#model) |
|
|
| `description` | `string` | [Model](#model) |
|
|
| `source` | `string` | [Model](#model) |
|
|
| `profiles` | `[]Profile` | [Model](#model) |
|
|
| `scaling` | `Scaling` | [Model](#model) |
|
|
| `class` | `string` | [Model](#model) |
|
|
| `owner` | `string` | [Model](#model) |
|
|
| `external` | `*External` | [Model](#model) |
|
|
| `adapters` | `[]Adapter` | [Model](#model) |
|
|
| `fallbacks` | `[]string` | [Model](#model) |
|
|
| `canary` | `*Canary` | [Model](#model) |
|
|
| `eval` | `*EvalGate` | [Model](#model) |
|
|
| `routing_profile` | `string` | [Model](#model) |
|
|
| `slo` | `SLO` | [Model](#model) |
|
|
| `context_length` | `int` | [Model](#model) |
|
|
| `max_output` | `int` | [Model](#model) |
|
|
| `capabilities` | `Capabilities` | [Model](#model) |
|
|
| `role` | `string` | [Model](#model) |
|
|
| `guard` | `*GuardModel` | [Model](#model) |
|
|
| `origin` | `*Origin` | [Model](#model) |
|
|
| `prometheus` | `string` | [Monitoring](#monitoring) |
|
|
| `window` | `time.Duration` | [Monitoring](#monitoring) |
|
|
| `timeout` | `time.Duration` | [Monitoring](#monitoring) |
|
|
| `issuer` | `string` | [OIDC](#oidc) |
|
|
| `client_id` | `string` | [OIDC](#oidc) |
|
|
| `client_secret_file` | `string` | [OIDC](#oidc) |
|
|
| `redirect_url` | `string` | [OIDC](#oidc) |
|
|
| `label` | `string` | [OIDC](#oidc) |
|
|
| `scopes` | `[]string` | [OIDC](#oidc) |
|
|
| `groups_claim` | `string` | [OIDC](#oidc) |
|
|
| `create_accounts` | `bool` | [OIDC](#oidc) |
|
|
| `roles` | `map[string][]string` | [OIDC](#oidc) |
|
|
| `memberships` | `map[string]string` | [OIDC](#oidc) |
|
|
| `no_passwords` | `bool` | [OIDC](#oidc) |
|
|
| `nvidia_smi` | `string` | [Options](#options) |
|
|
| `ib_root` | `string` | [Options](#options) |
|
|
| `prefer_nvml` | `bool` | [Options](#options) |
|
|
| `skip_gpus` | `bool` | [Options](#options) |
|
|
| `repo` | `string` | [Origin](#origin) |
|
|
| `revision` | `string` | [Origin](#origin) |
|
|
| `files` | `[]OriginFile` | [Origin](#origin) |
|
|
| `previous` | `*PreviousOrigin` | [Origin](#origin) |
|
|
| `path` | `string` | [OriginFile](#originfile) |
|
|
| `sha256` | `string` | [OriginFile](#originfile) |
|
|
| `bytes` | `int64` | [OriginFile](#originfile) |
|
|
| `name` | `string` | [Pattern](#pattern) |
|
|
| `regex` | `string` | [Pattern](#pattern) |
|
|
| `action` | `string` | [Pattern](#pattern) |
|
|
| `disabled` | `bool` | [Personal](#personal) |
|
|
| `rpm` | `int` | [Personal](#personal) |
|
|
| `tpm` | `int` | [Personal](#personal) |
|
|
| `budget` | `float64` | [Personal](#personal) |
|
|
| `period` | `string` | [Personal](#personal) |
|
|
| `mode` | `string` | [Policy](#policy) |
|
|
| `fail` | `string` | [Policy](#policy) |
|
|
| `no_store` | `bool` | [Policy](#policy) |
|
|
| `input` | `Rules` | [Policy](#policy) |
|
|
| `output` | `Rules` | [Policy](#policy) |
|
|
| `tools` | `Tools` | [Policy](#policy) |
|
|
| `max_output_tokens` | `int` | [Policy](#policy) |
|
|
| `system_prompt` | `string` | [Policy](#policy) |
|
|
| `drop_client_system` | `bool` | [Policy](#policy) |
|
|
| `require_capabilities` | `bool` | [Policy](#policy) |
|
|
| `validate_structured` | `bool` | [Policy](#policy) |
|
|
| `judge` | `*JudgePolicy` | [Policy](#policy) |
|
|
| `stream` | `string` | [Policy](#policy) |
|
|
| `enabled` | `bool` | [Prefix](#prefix) |
|
|
| `max_nodes` | `int` | [Prefix](#prefix) |
|
|
| `min_match` | `int` | [Prefix](#prefix) |
|
|
| `ttl` | `time.Duration` | [Prefix](#prefix) |
|
|
| `match_ratio` | `float64` | [Prefix](#prefix) |
|
|
| `balance_abs` | `float64` | [Prefix](#prefix) |
|
|
| `balance_rel` | `float64` | [Prefix](#prefix) |
|
|
| `source` | `string` | [PreviousOrigin](#previousorigin) |
|
|
| `revision` | `string` | [PreviousOrigin](#previousorigin) |
|
|
| `files` | `[]OriginFile` | [PreviousOrigin](#previousorigin) |
|
|
| `name` | `string` | [Profile](#profile) |
|
|
| `engine` | `string` | [Profile](#profile) |
|
|
| `gpu_class` | `string` | [Profile](#profile) |
|
|
| `tp` | `int` | [Profile](#profile) |
|
|
| `pp` | `int` | [Profile](#profile) |
|
|
| `gpus_per_node` | `int` | [Profile](#profile) |
|
|
| `nodes` | `int` | [Profile](#profile) |
|
|
| `args` | `[]string` | [Profile](#profile) |
|
|
| `env` | `map[string]string` | [Profile](#profile) |
|
|
| `min_memory_gb` | `int` | [Profile](#profile) |
|
|
| `cpu_cores` | `int` | [Profile](#profile) |
|
|
| `min_disk_gb` | `int` | [Profile](#profile) |
|
|
| `node_selector` | `map[string]string` | [Profile](#profile) |
|
|
| `weight` | `float64` | [Profile](#profile) |
|
|
| `priority` | `int` | [Profile](#profile) |
|
|
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
|
| `activation_wait` | `time.Duration` | [Router](#router) |
|
|
| `scrape_interval` | `time.Duration` | [Router](#router) |
|
|
| `scrape_timeout` | `time.Duration` | [Router](#router) |
|
|
| `stale_after` | `time.Duration` | [Router](#router) |
|
|
| `retries` | `int` | [Router](#router) |
|
|
| `cooldown` | `time.Duration` | [Router](#router) |
|
|
| `failure_threshold` | `int` | [Router](#router) |
|
|
| `default_profile` | `string` | [Router](#router) |
|
|
| `prefix` | `Prefix` | [Router](#router) |
|
|
| `session_header` | `string` | [Router](#router) |
|
|
| `inject_usage` | `bool` | [Router](#router) |
|
|
| `tenant_label` | `string` | [Router](#router) |
|
|
| `tag_labels` | `[]string` | [Router](#router) |
|
|
| `max_body_bytes` | `int64` | [Router](#router) |
|
|
| `batch` | `Batch` | [Router](#router) |
|
|
| `cache` | `Cache` | [Router](#router) |
|
|
| `trusted_proxies` | `[]string` | [Router](#router) |
|
|
| `timeouts` | `Timeouts` | [Router](#router) |
|
|
| `scoring` | `Scoring` | [Router](#router) |
|
|
| `admission` | `Admission` | [Router](#router) |
|
|
| `pii` | `map[string]string` | [Rules](#rules) |
|
|
| `injection` | `string` | [Rules](#rules) |
|
|
| `deny` | `[]Pattern` | [Rules](#rules) |
|
|
| `max_chars` | `int` | [Rules](#rules) |
|
|
| `token_file` | `string` | [SCIM](#scim) |
|
|
| `python` | `string` | [SGLangConfig](#sglangconfig) |
|
|
| `ttft_p99` | `time.Duration` | [SLO](#slo) |
|
|
| `tpot_p99` | `time.Duration` | [SLO](#slo) |
|
|
| `days` | `[]string` | [ScaleWindow](#scalewindow) |
|
|
| `from` | `string` | [ScaleWindow](#scalewindow) |
|
|
| `to` | `string` | [ScaleWindow](#scalewindow) |
|
|
| `min` | `int` | [ScaleWindow](#scalewindow) |
|
|
| `max` | `int` | [ScaleWindow](#scalewindow) |
|
|
| `min` | `int` | [Scaling](#scaling) |
|
|
| `max` | `int` | [Scaling](#scaling) |
|
|
| `idle_timeout` | `time.Duration` | [Scaling](#scaling) |
|
|
| `schedule` | `[]ScaleWindow` | [Scaling](#scaling) |
|
|
| `prewarm` | `time.Duration` | [Scaling](#scaling) |
|
|
| `target_waiting` | `float64` | [Scoring](#scoring) |
|
|
| `target_inflight` | `float64` | [Scoring](#scoring) |
|
|
| `kv_ceiling` | `float64` | [Scoring](#scoring) |
|
|
| `session_load_factor` | `float64` | [Scoring](#scoring) |
|
|
| `vault` | `secrets.Vault` | [Secrets](#secrets) |
|
|
| `master_key` | `string` | [Secrets](#secrets) |
|
|
| `listen` | `string` | [Server](#server) |
|
|
| `admin_listen` | `string` | [Server](#server) |
|
|
| `metrics_listen` | `string` | [Server](#server) |
|
|
| `agent_listen` | `string` | [Server](#server) |
|
|
| `auth` | `Auth` | [Server](#server) |
|
|
| `threads` | `Threads` | [Server](#server) |
|
|
| `landing` | `Landing` | [Server](#server) |
|
|
| `agent_tls` | `tlsconf.Config` | [Server](#server) |
|
|
| `max_connections` | `int` | [Server](#server) |
|
|
| `admin_token_file` | `string` | [Server](#server) |
|
|
| `admin_remote` | `bool` | [Server](#server) |
|
|
| `debug_pprof` | `bool` | [Server](#server) |
|
|
| `agent_insecure` | `bool` | [Server](#server) |
|
|
| `monitoring` | `Monitoring` | [Server](#server) |
|
|
| `tracing` | `obs.Tracing` | [Server](#server) |
|
|
| `mirror` | `Mirror` | [Server](#server) |
|
|
| `engines` | `map[string]string` | [Server](#server) |
|
|
| `engine_overrides` | `[]EngineOverride` | [Server](#server) |
|
|
| `backup_dir` | `string` | [Server](#server) |
|
|
| `guard` | `guard.Config` | [Server](#server) |
|
|
| `secrets` | `Secrets` | [Server](#server) |
|
|
| `notify` | `notify.Config` | [Server](#server) |
|
|
| `firewall_networks` | `[]string` | [Server](#server) |
|
|
| `backup_recipients` | `[]string` | [Server](#server) |
|
|
| `backup_copy_to` | `[]string` | [Server](#server) |
|
|
| `backup_copy_key` | `string` | [Server](#server) |
|
|
| `releases` | `string` | [Server](#server) |
|
|
| `router` | `Router` | [Server](#server) |
|
|
| `control` | `Control` | [Server](#server) |
|
|
| `models` | `[]catalog.Model` | [Server](#server) |
|
|
| `federation` | `Federation` | [Server](#server) |
|
|
| `endpoints` | `[]Endpoint` | [Server](#server) |
|
|
| `bin` | `string` | [SimConfig](#simconfig) |
|
|
| `python` | `string` | [TRTLLMConfig](#trtllmconfig) |
|
|
| `store` | `string` | [Threads](#threads) |
|
|
| `retain_hours` | `int` | [Threads](#threads) |
|
|
| `dial` | `time.Duration` | [Timeouts](#timeouts) |
|
|
| `response_header` | `time.Duration` | [Timeouts](#timeouts) |
|
|
| `generation` | `time.Duration` | [Timeouts](#timeouts) |
|
|
| `first_token` | `time.Duration` | [Timeouts](#timeouts) |
|
|
| `idle` | `time.Duration` | [Timeouts](#timeouts) |
|
|
| `total` | `time.Duration` | [Timeouts](#timeouts) |
|
|
| `allow` | `[]string` | [Tools](#tools) |
|
|
| `deny` | `[]string` | [Tools](#tools) |
|
|
| `on_denied` | `string` | [Tools](#tools) |
|
|
| `max_calls_per_turn` | `int` | [Tools](#tools) |
|
|
| `validate_arguments` | `bool` | [Tools](#tools) |
|
|
| `endpoint` | `string` | [Tracing](#tracing) |
|
|
| `sample_ratio` | `float64` | [Tracing](#tracing) |
|
|
| `bin` | `string` | [VLLMConfig](#vllmconfig) |
|
|
| `launch_mode` | `string` | [VLLMConfig](#vllmconfig) |
|
|
| `ray_bin` | `string` | [VLLMConfig](#vllmconfig) |
|
|
| `addr` | `string` | [Vault](#vault) |
|
|
| `token_file` | `string` | [Vault](#vault) |
|
|
| `role_id_file` | `string` | [Vault](#vault) |
|
|
| `secret_id_file` | `string` | [Vault](#vault) |
|
|
| `ca_file` | `string` | [Vault](#vault) |
|
|
| `namespace` | `string` | [Vault](#vault) |
|
|
| `url` | `string` | [Webhook](#webhook) |
|
|
| `format` | `string` | [Webhook](#webhook) |
|
|
| `events` | `[]string` | [Webhook](#webhook) |
|
|
| `secret_file` | `string` | [Webhook](#webhook) |
|
|
| `prompt_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
|
| `completion_per_1k` | `int64` | [accounting.Price](#accountingprice) |
|
|
| `name` | `string` | [admission.Band](#admissionband) |
|
|
| `priority` | `int` | [admission.Band](#admissionband) |
|
|
| `saturation_ceiling` | `float64` | [admission.Band](#admissionband) |
|
|
| `bands` | `[]Band` | [admission.Config](#admissionconfig) |
|
|
| `default_band` | `string` | [admission.Config](#admissionconfig) |
|
|
| `queue_ttl` | `time.Duration` | [admission.Config](#admissionconfig) |
|
|
| `queue_capacity_per_band` | `int` | [admission.Config](#admissionconfig) |
|
|
| `queue_capacity_total` | `int` | [admission.Config](#admissionconfig) |
|
|
| `interval` | `time.Duration` | [admission.Config](#admissionconfig) |
|
|
| `node_id` | `string` | [agent.Config](#agentconfig) |
|
|
| `server` | `string` | [agent.Config](#agentconfig) |
|
|
| `servers` | `[]string` | [agent.Config](#agentconfig) |
|
|
| `hostname` | `string` | [agent.Config](#agentconfig) |
|
|
| `mgmt_addr` | `string` | [agent.Config](#agentconfig) |
|
|
| `labels` | `map[string]string` | [agent.Config](#agentconfig) |
|
|
| `mgmt_interface` | `string` | [agent.Config](#agentconfig) |
|
|
| `fabrics` | `[]inventory.FabricAttachment` | [agent.Config](#agentconfig) |
|
|
| `heartbeat_interval` | `time.Duration` | [agent.Config](#agentconfig) |
|
|
| `reconnect_base` | `time.Duration` | [agent.Config](#agentconfig) |
|
|
| `reconnect_max` | `time.Duration` | [agent.Config](#agentconfig) |
|
|
| `adopt_ready_timeout` | `time.Duration` | [agent.Config](#agentconfig) |
|
|
| `tls` | `tlsconf.Config` | [agent.Config](#agentconfig) |
|
|
| `model_dirs` | `[]string` | [agent.Config](#agentconfig) |
|
|
| `engines_dir` | `string` | [agent.Config](#agentconfig) |
|
|
| `weights_cache` | `string` | [agent.Config](#agentconfig) |
|
|
| `discovery` | `discovery.Options` | [agent.Config](#agentconfig) |
|
|
| `engines` | `engines.Config` | [agent.Config](#agentconfig) |
|
|
| `supervisor` | `supervisor.Config` | [agent.Config](#agentconfig) |
|
|
| `tracing` | `obs.Tracing` | [agent.Config](#agentconfig) |
|
|
| `name` | `string` | [config.Band](#configband) |
|
|
| `priority` | `int` | [config.Band](#configband) |
|
|
| `saturation_ceiling` | `float64` | [config.Band](#configband) |
|
|
| `prompt_per_1k` | `int64` | [config.Price](#configprice) |
|
|
| `completion_per_1k` | `int64` | [config.Price](#configprice) |
|
|
| `disabled` | `[]string` | [engines.Config](#enginesconfig) |
|
|
| `vllm` | `VLLMConfig` | [engines.Config](#enginesconfig) |
|
|
| `sglang` | `SGLangConfig` | [engines.Config](#enginesconfig) |
|
|
| `llamacpp` | `LlamaCppConfig` | [engines.Config](#enginesconfig) |
|
|
| `trtllm` | `TRTLLMConfig` | [engines.Config](#enginesconfig) |
|
|
| `sim` | `SimConfig` | [engines.Config](#enginesconfig) |
|
|
| `default` | `[]string` | [guard.Config](#guardconfig) |
|
|
| `policies` | `map[string]Policy` | [guard.Config](#guardconfig) |
|
|
| `webhooks` | `[]Webhook` | [notify.Config](#notifyconfig) |
|
|
| `host` | `string` | [notify.Config](#notifyconfig) |
|
|
| `work_dir` | `string` | [supervisor.Config](#supervisorconfig) |
|
|
| `log_lines` | `int` | [supervisor.Config](#supervisorconfig) |
|
|
| `stop_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
|
| `health_interval` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
|
| `rendezvous_timeout` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
|
| `alive_grace` | `time.Duration` | [supervisor.Config](#supervisorconfig) |
|
|
| `run_as` | `string` | [supervisor.Config](#supervisorconfig) |
|
|
| `ca` | `string` | [tlsconf.Config](#tlsconfconfig) |
|
|
| `cert` | `string` | [tlsconf.Config](#tlsconfconfig) |
|
|
| `key` | `string` | [tlsconf.Config](#tlsconfconfig) |
|
|
| `insecure` | `bool` | [tlsconf.Config](#tlsconfconfig) |
|
|
|
|
## Adapter
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
Name is what clients ask for, like a model name.
|
|
|
|
### `source`
|
|
|
|
`string`
|
|
|
|
Source is the adapter file (GGUF for llama.cpp), fetched by each node
|
|
as the model's weights are.
|
|
|
|
### `scale`
|
|
|
|
`float64`
|
|
|
|
Scale is how strongly it is applied; zero takes 1.
|
|
|
|
## Admission
|
|
|
|
### `enabled`
|
|
|
|
`bool`
|
|
|
|
Enabled turns the layer on. Off, every request goes straight through,
|
|
which is the v0.2.0 behaviour.
|
|
|
|
### `bands`
|
|
|
|
`[]Band`
|
|
|
|
### `default_band`
|
|
|
|
`string`
|
|
|
|
### `band_header`
|
|
|
|
`string`
|
|
|
|
BandHeader lets a client declare its band. Until authentication lands in
|
|
v1.0.0 this is trusted, so the port must not be exposed to callers who
|
|
should not be able to promote themselves.
|
|
|
|
### `queue_ttl`
|
|
|
|
`time.Duration`
|
|
|
|
### `queue_capacity_per_band`
|
|
|
|
`int`
|
|
|
|
### `queue_capacity_total`
|
|
|
|
`int`
|
|
|
|
QueueCapacityTotal bounds everything queued across bands. Reached, a
|
|
request is not refused while a lower band has any queued: the newest of
|
|
the lowest band is turned away instead, with a 503 and Retry-After.
|
|
Zero takes half of all the bands' capacities together.
|
|
|
|
### `interval`
|
|
|
|
`time.Duration`
|
|
|
|
## Auth
|
|
|
|
### `enabled`
|
|
|
|
`bool`
|
|
|
|
### `store`
|
|
|
|
`string`
|
|
|
|
Store is the SQLite file holding keys, tenancy and usage.
|
|
|
|
### `cache_ttl`
|
|
|
|
`time.Duration`
|
|
|
|
CacheTTL bounds how long a revoked key keeps working, traded against a
|
|
database read per key per TTL.
|
|
|
|
### `negative_ttl`
|
|
|
|
`time.Duration`
|
|
|
|
NegativeTTL is how long a rejected key is remembered as rejected, so a
|
|
client looping with a bad key cannot become a database read per request.
|
|
|
|
### `default_rpm`
|
|
|
|
`int`
|
|
|
|
DefaultRPM and DefaultTPM apply to a key that no scope gives a limit to.
|
|
|
|
Unlimited used to be the default, and a project created without limits is
|
|
the easy path — the "default" project on this cluster still is one. The
|
|
first sign of that is the bill or the outage. Zero keeps the old
|
|
behaviour for anyone who wants it, but it should be a choice rather than
|
|
an omission. A fallback, not a ceiling: an explicit limit at any scope
|
|
wins, above or below.
|
|
|
|
### `default_tpm`
|
|
|
|
`int`
|
|
|
|
### `signup_open`
|
|
|
|
`bool`
|
|
|
|
SignupOpen serves the signup form. Accounts it creates are pending until
|
|
an operator approves them, so this is not a way in — but turning it off
|
|
means accounts are made with `llmgw users create` instead.
|
|
|
|
### `oidc`
|
|
|
|
`OIDC`
|
|
|
|
OIDC is sign-in through an identity provider.
|
|
|
|
Takes a [OIDC](#oidc) block.
|
|
|
|
### `scim`
|
|
|
|
`SCIM`
|
|
|
|
SCIM is account provisioning by the identity provider.
|
|
|
|
Takes a [SCIM](#scim) block.
|
|
|
|
### `max_key_ttl`
|
|
|
|
`time.Duration`
|
|
|
|
MaxKeyTTL is how long any API key may live; a key asked for with no
|
|
expiry is given this. Organizations, teams and projects can set
|
|
shorter ones (llmgw keys ttl). Zero is no limit.
|
|
|
|
### `personal`
|
|
|
|
`Personal`
|
|
|
|
Personal is what a person may do on their own, outside any project: make
|
|
personal keys and use the chat page personally, within these limits.
|
|
|
|
Takes a [Personal](#personal) block.
|
|
|
|
### `pricing`
|
|
|
|
`map[string]Price`
|
|
|
|
Pricing is what each model costs, for budgets. A model absent from the
|
|
table costs whatever DefaultPrice says, which is nothing unless set.
|
|
|
|
### `default_price`
|
|
|
|
`Price`
|
|
|
|
Takes a [config.Price](#configprice) block.
|
|
|
|
### `flush_interval`
|
|
|
|
`time.Duration`
|
|
|
|
FlushInterval and BatchSize decide when usage records are written.
|
|
|
|
### `batch_size`
|
|
|
|
`int`
|
|
|
|
### `buffer_size`
|
|
|
|
`int`
|
|
|
|
BufferSize bounds the queue. Beyond it records are dropped rather than
|
|
blocking a request: losing accounting is bad, turning a stalled database
|
|
into an outage is worse.
|
|
|
|
### `usage_retention`
|
|
|
|
`time.Duration`
|
|
|
|
UsageRetention is how long the record of each request is kept: who sent
|
|
it, which replica answered, how long it queued — what `ctl request`
|
|
reads. Zero keeps it for ever. Totals, charts and budgets are read from
|
|
five-minute sums kept beside the records, which are not pruned, so this
|
|
bounds the table that grows by a row a request without changing any
|
|
figure a page shows. At least a day when set.
|
|
|
|
## Batch
|
|
|
|
### `max_file_bytes`
|
|
|
|
`int64`
|
|
|
|
MaxFileBytes bounds one uploaded file; 100 MB unless set.
|
|
|
|
### `max_lines`
|
|
|
|
`int`
|
|
|
|
MaxLines bounds the requests in one batch; 50 000 unless set.
|
|
|
|
### `concurrency`
|
|
|
|
`int`
|
|
|
|
Concurrency is how many of one batch's requests are in flight at
|
|
once; 4 unless set. They queue in the lowest admission band, behind
|
|
every interactive request.
|
|
|
|
### `retain_days`
|
|
|
|
`int`
|
|
|
|
RetainDays is how long a batch's answer and error files are kept;
|
|
30 unless set. Uploaded files are kept until deleted.
|
|
|
|
## Cache
|
|
|
|
### `max_bytes`
|
|
|
|
`int64`
|
|
|
|
MaxBytes bounds every project's answers together; 128 MB unless set.
|
|
Past it, the answers used longest ago go first.
|
|
|
|
### `max_entry_bytes`
|
|
|
|
`int64`
|
|
|
|
MaxEntryBytes bounds one answer; a longer one is not kept. 1 MB
|
|
unless set.
|
|
|
|
## Canary
|
|
|
|
### `model`
|
|
|
|
`string`
|
|
|
|
Model is the candidate: a catalog model, usually this one at newer
|
|
weights, which clients cannot name themselves.
|
|
|
|
### `percent`
|
|
|
|
`int`
|
|
|
|
Percent of requests go to it, 0 to 100.
|
|
|
|
### `min_requests`
|
|
|
|
`int`
|
|
|
|
The candidate is taken out — rolled back — when, over at least
|
|
MinRequests of its own, its error rate is MaxErrorIncrease above
|
|
this model's (0.05 is five points), or its mean time to the first
|
|
token is MaxTTFTRatio times this model's. Zero is 50, 0.05 and 1.5.
|
|
|
|
### `max_error_increase`
|
|
|
|
`float64`
|
|
|
|
### `max_ttft_ratio`
|
|
|
|
`float64`
|
|
|
|
## Capabilities
|
|
|
|
### `tools`
|
|
|
|
`bool`
|
|
|
|
### `vision`
|
|
|
|
`bool`
|
|
|
|
### `embeddings`
|
|
|
|
`bool`
|
|
|
|
## Control
|
|
|
|
### `interval`
|
|
|
|
`time.Duration`
|
|
|
|
Interval is how often desired and actual state are compared.
|
|
|
|
### `base_backoff`
|
|
|
|
`time.Duration`
|
|
|
|
BaseBackoff and MaxBackoff bound how fast a failing model is retried.
|
|
|
|
### `max_backoff`
|
|
|
|
`time.Duration`
|
|
|
|
### `unreachable_after`
|
|
|
|
`time.Duration`
|
|
|
|
UnreachableAfter and LostAfter stage node expiry. They are deliberately
|
|
far apart: a network blip must not tear down replicas that still serve,
|
|
and rebuilding a multi-node replica costs tens of minutes of loading.
|
|
|
|
### `lost_after`
|
|
|
|
`time.Duration`
|
|
|
|
### `port_range`
|
|
|
|
`[]int`
|
|
|
|
PortRange is the per-node range for engine HTTP and dist-init ports.
|
|
|
|
### `mgmt_interface`
|
|
|
|
`string`
|
|
|
|
MgmtInterface is the ethernet interface NCCL bootstraps over.
|
|
|
|
### `timezone`
|
|
|
|
`string`
|
|
|
|
Timezone is where models' scaling schedules are read, an IANA name
|
|
like Europe/Istanbul; empty is the control plane's own.
|
|
|
|
### `eval_dir`
|
|
|
|
`string`
|
|
|
|
EvalDir is where quality suites are read from by name, for `llmgw ctl
|
|
eval MODEL --suite NAME`, a model's eval gate and the console. The
|
|
suites the binary carries (arithmetic, json-format) are found when it
|
|
has none of the name; a suite here of the same name replaces one.
|
|
|
|
### `enabled`
|
|
|
|
`bool`
|
|
|
|
Enabled runs the control plane even when configuration declares no
|
|
models.
|
|
|
|
Without it, a deployment whose catalog lives entirely in the database
|
|
cannot start: control plane mode is inferred from the config file, so an
|
|
empty models block means "router only", the process refuses to start for
|
|
want of anything to serve, and the models it would have loaded are never
|
|
read. That is a cluster that cannot be managed at runtime because it is
|
|
managed at runtime. AgentListen cannot carry this signal — it has a
|
|
default, so every deployment would become a control plane.
|
|
|
|
### `lease_ttl`
|
|
|
|
`time.Duration`
|
|
|
|
LeaseTTL is how long this control plane's claim to be the one placing
|
|
work stands without being renewed.
|
|
|
|
Only meaningful with a shared store, where a second control plane can
|
|
see the claim. Long enough to survive a slow query or a brief network
|
|
stall, short enough that a standby takes over before anyone has finished
|
|
reading the alert. Zero takes the default.
|
|
|
|
## Endpoint
|
|
|
|
### `id`
|
|
|
|
`string`
|
|
|
|
### `url`
|
|
|
|
`string`
|
|
|
|
### `api_key`
|
|
|
|
`string`
|
|
|
|
APIKey is sent to the engine as a bearer token, for an engine started
|
|
with one.
|
|
|
|
### `models`
|
|
|
|
`[]string`
|
|
|
|
### `engine`
|
|
|
|
`string`
|
|
|
|
Engine names the inference engine, which decides what its metrics are
|
|
called. Empty means vLLM's names.
|
|
|
|
### `pool`
|
|
|
|
`string`
|
|
|
|
### `weight`
|
|
|
|
`float64`
|
|
|
|
### `priority`
|
|
|
|
`int`
|
|
|
|
## EngineOverride
|
|
|
|
### `gpu_class`
|
|
|
|
`string`
|
|
|
|
GPUClass picks nodes with a GPU of this class ("v100-32g").
|
|
|
|
### `nodes`
|
|
|
|
`[]string`
|
|
|
|
Nodes picks nodes by id.
|
|
|
|
### `labels`
|
|
|
|
`map[string]string`
|
|
|
|
Labels picks nodes carrying all of these.
|
|
|
|
### `engines`
|
|
|
|
`map[string]string`
|
|
|
|
Engines are laid over the cluster's: a version, or "" for none.
|
|
|
|
## EvalGate
|
|
|
|
### `suite`
|
|
|
|
`string`
|
|
|
|
Suite is a suite's name in control.eval_dir, or one the binary
|
|
carries (arithmetic, json-format).
|
|
|
|
### `min_score`
|
|
|
|
`float64`
|
|
|
|
MinScore is the score, 0 to 1, to pass at; zero is the suite's own
|
|
threshold.
|
|
|
|
### `judge`
|
|
|
|
`string`
|
|
|
|
Judge grades the suite's rubric cases, instead of the judge it names.
|
|
|
|
## External
|
|
|
|
### `url`
|
|
|
|
`string`
|
|
|
|
URL is the provider's base address, without /v1: https://api.openai.com.
|
|
|
|
### `model`
|
|
|
|
`string`
|
|
|
|
Model is what the provider calls it; requests are rewritten to it.
|
|
|
|
### `api_key_file`
|
|
|
|
`string`
|
|
|
|
APIKeyFile holds the provider's key, on the control plane. A file
|
|
rather than the key itself, so the key is in neither the catalog, the
|
|
store nor the audit log; empty for a provider that asks for none.
|
|
|
|
### `peer`
|
|
|
|
`string`
|
|
|
|
Peer names the federated llmgw cluster this model is offered by, for
|
|
a model that federation added (`llmgw ctl federation add`) rather
|
|
than one written by hand. Such a model is named PEER/MODEL, comes and
|
|
goes with the peer's own list, and is sent only while the peer is up.
|
|
|
|
## Federation
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
Name is this cluster as a peer reads it, in the federated_from tag
|
|
sent with a request to a peer set to receive tags (`ctl federation add
|
|
--send-tags`). Empty is this machine's host name.
|
|
|
|
### `interval`
|
|
|
|
`time.Duration`
|
|
|
|
Interval is how often each peer is asked whether it is ready, timed,
|
|
and what it offers. A peer that does not answer is taken out of
|
|
routing until it does. Zero is 10s.
|
|
|
|
## GuardModel
|
|
|
|
### `format`
|
|
|
|
`string`
|
|
|
|
Format is llamaguard3 (Meta's Llama Guard 3: "safe", or "unsafe" and
|
|
the categories), shieldgemma (Google's ShieldGemma: Yes or No), or
|
|
json ({"safe": false, "categories": ["..."]}).
|
|
|
|
### `categories`
|
|
|
|
`[]string`
|
|
|
|
Categories the model is asked about, for shieldgemma — each a policy
|
|
sentence — and the names reported for llamaguard3's S1, S2, ….
|
|
|
|
## JudgePolicy
|
|
|
|
### `model`
|
|
|
|
`string`
|
|
|
|
Model is a catalog model with role: guard.
|
|
|
|
### `input`
|
|
|
|
`string`
|
|
|
|
Input and Output are what an unsafe verdict does: flag or block.
|
|
Empty does not ask the judge about that direction.
|
|
|
|
### `output`
|
|
|
|
`string`
|
|
|
|
### `categories`
|
|
|
|
`[]string`
|
|
|
|
Categories, when set, are the only ones that count: Llama Guard's
|
|
codes (S1, S10) or the names the model reports.
|
|
|
|
### `latency_budget`
|
|
|
|
`time.Duration`
|
|
|
|
LatencyBudget is how long the judge is waited for before the policy's
|
|
fail applies. Zero is 250ms.
|
|
|
|
## Landing
|
|
|
|
### `contact`
|
|
|
|
`string`
|
|
|
|
Contact is where someone without a key should go — an address, a URL, a
|
|
channel name. Rendered as a link when it is one, as text otherwise.
|
|
|
|
### `notice`
|
|
|
|
`string`
|
|
|
|
Notice is an optional line of your own, for whatever the reader needs to
|
|
know that this page cannot guess.
|
|
|
|
### `docs_sign_in`
|
|
|
|
`bool`
|
|
|
|
DocsSignIn serves the docs portal at /docs only to someone signed in.
|
|
Off, the portal is public like this page: its client guide is for the
|
|
person who has no account yet, and the operator documents describe
|
|
llmgw in general — no address, model or key of this installation —
|
|
which the project's repository publishes anyway. On, a visitor is sent
|
|
to sign in, and without accounts the portal needs an API key.
|
|
|
|
## LlamaCppConfig
|
|
|
|
### `bin`
|
|
|
|
`string`
|
|
|
|
Bin is the llama-server binary. It is the server, not the CLI: the
|
|
gateway talks OpenAI over HTTP and needs the one that serves it.
|
|
Empty means the agent installs llama.cpp itself, at the version the
|
|
control plane asks for. Setting it opts out of that.
|
|
|
|
### `parallel`
|
|
|
|
`int`
|
|
|
|
Parallel is how many requests a server handles at once, when a profile
|
|
does not say. llama.cpp defaults to one, which would make every second
|
|
request wait behind the first no matter what the router decided.
|
|
|
|
### `threads`
|
|
|
|
`int`
|
|
|
|
Threads is how many CPU threads a shard with no GPU computes on. Zero
|
|
takes half this machine's CPUs.
|
|
|
|
Not all of them, which is llama.cpp's own default: two busy servers
|
|
that each spin on every CPU fight for them, and on an 8-vCPU node
|
|
that made 7B generation fall from 10 tokens a second to under 1 each.
|
|
At half, one server alone is about a third slower and two together
|
|
keep 5.5 each. A node that only ever runs one CPU replica does better
|
|
with all of them; say so here.
|
|
|
|
## Mirror
|
|
|
|
### `api`
|
|
|
|
`string`
|
|
|
|
API is the mirror's management address, scheme included, for example
|
|
https://models.internal:8082. Empty means no mirror is configured.
|
|
|
|
The management port, not the one nodes fetch bytes from. Those are two
|
|
listeners on the mirror precisely so this one can be locked down without
|
|
re-certifying every agent's fetcher.
|
|
|
|
### `tls`
|
|
|
|
`tlsconf.Config`
|
|
|
|
TLS authenticates to the mirror's management API, with the same CA the
|
|
agents use. Empty leaves the connection in plain HTTP, which is only
|
|
acceptable on a network where nothing untrusted can reach that port —
|
|
the mirror says as much in its own log when it starts that way.
|
|
|
|
Takes a [tlsconf.Config](#tlsconfconfig) block.
|
|
|
|
### `registry`
|
|
|
|
`string`
|
|
|
|
Registry is the model registry the mirror page browses, empty for the
|
|
public Hugging Face. Set it to an internal mirror of the registry where
|
|
the control plane cannot reach the internet — or leave the whole mirror
|
|
block out, which turns the explorer off rather than giving it a search
|
|
box that can only time out.
|
|
|
|
### `timeout`
|
|
|
|
`time.Duration`
|
|
|
|
Timeout bounds a management call. It does not bound a download: the
|
|
mirror answers a fetch as soon as it has started one, and the bytes keep
|
|
arriving long after this deadline has passed.
|
|
|
|
### `update_check`
|
|
|
|
`time.Duration`
|
|
|
|
UpdateCheck is how often the registry is asked whether the files of
|
|
the models served from it have changed there: a repository can publish
|
|
new weights under the same name. Only told, never applied — moving a
|
|
model to them is `llmgw ctl model update`. 0 never asks.
|
|
|
|
## Model
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
### `aliases`
|
|
|
|
`[]string`
|
|
|
|
### `description`
|
|
|
|
`string`
|
|
|
|
Description is what this model is for, in a line.
|
|
|
|
Written by whoever added it, because nobody else can say it. A registry
|
|
offers a pipeline tag and a wall of README, and neither answers the
|
|
question a person picking from a list is asking: which of these nine
|
|
names should I send this request to.
|
|
|
|
It reaches clients through /v1/models, which is the only view most of
|
|
them ever see. A gateway serving nine models by name alone makes that
|
|
choice by guesswork.
|
|
|
|
### `source`
|
|
|
|
`string`
|
|
|
|
Source is the weights path or a HuggingFace id.
|
|
|
|
### `profiles`
|
|
|
|
`[]Profile`
|
|
|
|
### `scaling`
|
|
|
|
`Scaling`
|
|
|
|
Takes a [Scaling](#scaling) block.
|
|
|
|
### `class`
|
|
|
|
`string`
|
|
|
|
Class is how much this model's capacity is worth against another's,
|
|
when there is not enough for both: critical, standard or opportunistic.
|
|
Empty is standard, or opportunistic for a model loaded on demand. See
|
|
EffectiveClass.
|
|
|
|
### `owner`
|
|
|
|
`string`
|
|
|
|
Owner is the part of the tenancy tree whose capacity quota this model
|
|
counts against, written org:ID, team:ID or project:ID. Empty is shared:
|
|
held by the cluster, counted against nobody.
|
|
|
|
### `external`
|
|
|
|
`*External`
|
|
|
|
External is a model served by someone else — an OpenAI-compatible API
|
|
outside the cluster — rather than by replicas placed here. Such a model
|
|
has no profiles and nothing is placed for it; the router sends its
|
|
requests to the provider. Most useful named in another model's
|
|
fallbacks: used only when the cluster has no capacity left. Its
|
|
prompts leave the cluster, so which keys may name it is worth an
|
|
allowlist.
|
|
|
|
Takes a [External](#external) block.
|
|
|
|
### `adapters`
|
|
|
|
`[]Adapter`
|
|
|
|
Adapters are LoRA adapters served on this model's replicas, each asked
|
|
for by its own name as though it were a model. Every replica loads
|
|
every adapter, applying one only to a request that names it.
|
|
|
|
### `fallbacks`
|
|
|
|
`[]string`
|
|
|
|
### `canary`
|
|
|
|
`*Canary`
|
|
|
|
Canary sends a share of this model's requests to another catalog
|
|
model — the same model at new weights — to compare the two before
|
|
moving to it (`llmgw ctl model canary`).
|
|
|
|
Takes a [Canary](#canary) block.
|
|
|
|
### `eval`
|
|
|
|
`*EvalGate`
|
|
|
|
Eval is a quality suite new weights must pass before they replace
|
|
this model's: `llmgw ctl model update` and `model promote` run it
|
|
against them first, and stop below its score.
|
|
|
|
Takes a [EvalGate](#evalgate) block.
|
|
|
|
### `routing_profile`
|
|
|
|
`string`
|
|
|
|
RoutingProfile selects how the router scores endpoints for this model.
|
|
|
|
### `slo`
|
|
|
|
`SLO`
|
|
|
|
Takes a [SLO](#slo) block.
|
|
|
|
### `context_length`
|
|
|
|
`int`
|
|
|
|
ContextLength is how many tokens one request may use, prompt and
|
|
completion together, and MaxOutput bounds the completion alone. Zero
|
|
means unknown, which is what every model was until now.
|
|
|
|
The gateway knew neither, so an over-long prompt was not refused: it was
|
|
admitted, given a replica, and failed at the engine, and a client sizing
|
|
its prompts from /v1/models had nothing to read. Worse, the number is not
|
|
the one an operator writes in the engine's arguments — llama.cpp divides
|
|
--ctx-size across its parallel slots, so a pool of 8192 across four slots
|
|
is 2048 a request, and nothing said so.
|
|
|
|
### `max_output`
|
|
|
|
`int`
|
|
|
|
### `capabilities`
|
|
|
|
`Capabilities`
|
|
|
|
Capabilities is what this model can do beyond plain completion.
|
|
|
|
Takes a [Capabilities](#capabilities) block.
|
|
|
|
### `role`
|
|
|
|
`string`
|
|
|
|
Role "guard" is a model that judges the gateway's traffic for the
|
|
guardrails rather than serving clients: hidden from /v1/models and the
|
|
chat page, refused to any key that names it, critical by default, and
|
|
always running. Guard says how its answers are read.
|
|
|
|
### `guard`
|
|
|
|
`*GuardModel`
|
|
|
|
Takes a [GuardModel](#guardmodel) block.
|
|
|
|
### `origin`
|
|
|
|
`*Origin`
|
|
|
|
Origin is the registry repository the weights came from, at which
|
|
commit, and each file's checksum there. Written when a model is served
|
|
from the registry; for one that was not, worked out from the mirror's
|
|
record of where it fetched the file. It is what lets the control plane
|
|
say that the repository has published different weights since, and
|
|
move the model to them.
|
|
|
|
Takes a [Origin](#origin) block.
|
|
|
|
## Monitoring
|
|
|
|
### `prometheus`
|
|
|
|
`string`
|
|
|
|
Prometheus is the server's address, for example
|
|
http://monitoring.internal:9090. Empty leaves the charts off the
|
|
dashboard rather than showing an error where one should be.
|
|
|
|
### `window`
|
|
|
|
`time.Duration`
|
|
|
|
Window is how far back the charts look. Zero takes an hour.
|
|
|
|
### `timeout`
|
|
|
|
`time.Duration`
|
|
|
|
Timeout bounds one query. It runs while somebody waits for a page, so a
|
|
monitoring system that is slow today is skipped rather than waited for.
|
|
|
|
## OIDC
|
|
|
|
### `issuer`
|
|
|
|
`string`
|
|
|
|
Issuer is the provider's address; its /.well-known/openid-configuration
|
|
says the rest.
|
|
|
|
### `client_id`
|
|
|
|
`string`
|
|
|
|
ClientID and the file holding its secret, as registered with the
|
|
provider.
|
|
|
|
### `client_secret_file`
|
|
|
|
`string`
|
|
|
|
### `redirect_url`
|
|
|
|
`string`
|
|
|
|
RedirectURL is where the provider sends people back:
|
|
https://<gateway>/login/oidc/callback. Empty works it out from the
|
|
request, which is right unless a proxy changes the address.
|
|
|
|
### `label`
|
|
|
|
`string`
|
|
|
|
Label is what the sign-in button says: "Sign in with <label>".
|
|
|
|
### `scopes`
|
|
|
|
`[]string`
|
|
|
|
Scopes asked for besides openid; empty is email and profile, and
|
|
groups when Roles or Memberships use them.
|
|
|
|
### `groups_claim`
|
|
|
|
`string`
|
|
|
|
GroupsClaim names the ID token claim holding groups; empty is
|
|
"groups".
|
|
|
|
### `create_accounts`
|
|
|
|
`bool`
|
|
|
|
CreateAccounts makes an account, active, for a person the provider
|
|
vouches for and the gateway does not know. Off, only existing accounts
|
|
sign in this way.
|
|
|
|
### `roles`
|
|
|
|
`map[string][]string`
|
|
|
|
Roles maps a staff role (viewer, operator, billing, admin) to the
|
|
groups that give it. Set at every sign-in: someone taken out of the
|
|
group loses the role the next time.
|
|
|
|
### `memberships`
|
|
|
|
`map[string]string`
|
|
|
|
Memberships maps a group to a place in the tree — org:NAME,
|
|
team:ORG/TEAM, project:ORG/TEAM/PROJECT — joined at sign-in.
|
|
|
|
### `no_passwords`
|
|
|
|
`bool`
|
|
|
|
PasswordSignIn off refuses passwords on the sign-in form: people come
|
|
in through the provider only. Administrators still can, so a broken
|
|
provider does not lock everybody out.
|
|
|
|
## Options
|
|
|
|
### `nvidia_smi`
|
|
|
|
`string`
|
|
|
|
NvidiaSMI is the path to nvidia-smi; empty means look it up on PATH.
|
|
Worth configuring on nodes where the driver lives outside the default
|
|
paths, which containerized driver installs often do.
|
|
|
|
### `ib_root`
|
|
|
|
`string`
|
|
|
|
IBRoot is the sysfs InfiniBand directory.
|
|
|
|
### `prefer_nvml`
|
|
|
|
`bool`
|
|
|
|
PreferNVML uses the NVML backend when this binary has one. NVLink
|
|
topology is read from nvidia-smi topo -m on every build; NVML, where it
|
|
is used, is the more exact source of the peers.
|
|
|
|
### `skip_gpus`
|
|
|
|
`bool`
|
|
|
|
SkipGPUs omits GPU discovery. A node without nvidia-smi is already
|
|
found to have none, so this is for one whose driver is broken.
|
|
|
|
## Origin
|
|
|
|
### `repo`
|
|
|
|
`string`
|
|
|
|
Repo is the repository, owner/name.
|
|
|
|
### `revision`
|
|
|
|
`string`
|
|
|
|
Revision is the commit the files were fetched at.
|
|
|
|
### `files`
|
|
|
|
`[]OriginFile`
|
|
|
|
Files are the weights fetched from it, with what the registry
|
|
published as each one's sha256. A new commit that leaves them alone —
|
|
a README edited — is not an update.
|
|
|
|
### `previous`
|
|
|
|
`*PreviousOrigin`
|
|
|
|
Previous is what the last update replaced, kept so it can be gone back
|
|
to while its files are still on the mirror.
|
|
|
|
Takes a [PreviousOrigin](#previousorigin) block.
|
|
|
|
## OriginFile
|
|
|
|
### `path`
|
|
|
|
`string`
|
|
|
|
### `sha256`
|
|
|
|
`string`
|
|
|
|
### `bytes`
|
|
|
|
`int64`
|
|
|
|
## Pattern
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
### `regex`
|
|
|
|
`string`
|
|
|
|
### `action`
|
|
|
|
`string`
|
|
|
|
## Personal
|
|
|
|
### `disabled`
|
|
|
|
`bool`
|
|
|
|
Disabled refuses personal keys and personal chat; people then use the
|
|
gateway only through projects they are members of.
|
|
|
|
### `rpm`
|
|
|
|
`int`
|
|
|
|
### `tpm`
|
|
|
|
`int`
|
|
|
|
### `budget`
|
|
|
|
`float64`
|
|
|
|
Budget is in whole units per Period ("day" or "month").
|
|
|
|
### `period`
|
|
|
|
`string`
|
|
|
|
## Policy
|
|
|
|
### `mode`
|
|
|
|
`string`
|
|
|
|
Mode: enforce acts, shadow only records what it would have done, off
|
|
does nothing. Empty is enforce.
|
|
|
|
### `fail`
|
|
|
|
`string`
|
|
|
|
Fail: what happens when a policy cannot judge — a judge model down or
|
|
over its latency budget. open serves, closed refuses. Empty is open.
|
|
|
|
### `no_store`
|
|
|
|
`bool`
|
|
|
|
NoStore keeps nothing of the request: no stored conversation, no
|
|
memory notes, no prefix routing, no fallback outside the cluster.
|
|
|
|
### `input`
|
|
|
|
`Rules`
|
|
|
|
Takes a [Rules](#rules) block.
|
|
|
|
### `output`
|
|
|
|
`Rules`
|
|
|
|
Takes a [Rules](#rules) block.
|
|
|
|
### `tools`
|
|
|
|
`Tools`
|
|
|
|
Takes a [Tools](#tools) block.
|
|
|
|
### `max_output_tokens`
|
|
|
|
`int`
|
|
|
|
MaxOutputTokens is put in a request that asks for no limit; one
|
|
asking for more is lowered to it.
|
|
|
|
### `system_prompt`
|
|
|
|
`string`
|
|
|
|
SystemPrompt is put before every request's messages.
|
|
|
|
### `drop_client_system`
|
|
|
|
`bool`
|
|
|
|
DropClientSystem removes the caller's own system and developer
|
|
messages, so only the pinned prompt instructs the model.
|
|
|
|
### `require_capabilities`
|
|
|
|
`bool`
|
|
|
|
RequireCapabilities refuses a request offering tools, or sending
|
|
images, to a model whose catalog entry does not say it takes them.
|
|
|
|
### `validate_structured`
|
|
|
|
`bool`
|
|
|
|
ValidateStructured checks an answer against the JSON schema the
|
|
request asked for in response_format: one that does not fit is
|
|
stopped — or, streamed a sentence at a time, flagged, having been
|
|
sent already.
|
|
|
|
### `judge`
|
|
|
|
`*JudgePolicy`
|
|
|
|
Judge has a judge model on the cluster read the text too.
|
|
|
|
Takes a [JudgePolicy](#judgepolicy) block.
|
|
|
|
### `stream`
|
|
|
|
`string`
|
|
|
|
Stream: sentence scans a streamed answer a sentence at a time,
|
|
releasing each once it passes; buffer_all holds the whole answer
|
|
until it is judged. Empty is sentence.
|
|
|
|
## Prefix
|
|
|
|
### `enabled`
|
|
|
|
`bool`
|
|
|
|
### `max_nodes`
|
|
|
|
`int`
|
|
|
|
MaxNodes bounds the tree, MinMatch is the shortest match worth acting on
|
|
in bytes, and TTL is how long an entry is believed — an engine evicts on
|
|
its own schedule, so an old record may describe a cache that is gone.
|
|
|
|
### `min_match`
|
|
|
|
`int`
|
|
|
|
### `ttl`
|
|
|
|
`time.Duration`
|
|
|
|
### `match_ratio`
|
|
|
|
`float64`
|
|
|
|
MatchRatio is the share of the prompt that must match to be worth acting
|
|
on. BalanceAbs and BalanceRel are the imbalance guard, and both must trip
|
|
before affinity is abandoned.
|
|
|
|
### `balance_abs`
|
|
|
|
`float64`
|
|
|
|
### `balance_rel`
|
|
|
|
`float64`
|
|
|
|
## PreviousOrigin
|
|
|
|
### `source`
|
|
|
|
`string`
|
|
|
|
### `revision`
|
|
|
|
`string`
|
|
|
|
### `files`
|
|
|
|
`[]OriginFile`
|
|
|
|
## Profile
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
### `engine`
|
|
|
|
`string`
|
|
|
|
### `gpu_class`
|
|
|
|
`string`
|
|
|
|
### `tp`
|
|
|
|
`int`
|
|
|
|
### `pp`
|
|
|
|
`int`
|
|
|
|
### `gpus_per_node`
|
|
|
|
`int`
|
|
|
|
GPUsPerNode defaults to TP for a single-node profile.
|
|
|
|
### `nodes`
|
|
|
|
`int`
|
|
|
|
Nodes above 1 makes this a multi-node profile, which requires a fabric.
|
|
|
|
### `args`
|
|
|
|
`[]string`
|
|
|
|
### `env`
|
|
|
|
`map[string]string`
|
|
|
|
### `min_memory_gb`
|
|
|
|
`int`
|
|
|
|
MinMemoryGB is the host RAM this profile needs to run.
|
|
|
|
The number that matters is the model's working set: the weights, plus the
|
|
KV cache for the whole context pool, plus the engine's own overhead. For
|
|
a GPU profile that lives in the cards and this is small; for a CPU
|
|
profile it is the whole thing, and getting it wrong means an OOM on a
|
|
node that reported itself healthy.
|
|
|
|
Zero means unconstrained, which is every profile that existed before
|
|
this. A node that does not report its memory is not excluded either —
|
|
an older agent must not have its nodes quietly emptied by an upgrade of
|
|
the control plane.
|
|
|
|
### `cpu_cores`
|
|
|
|
`int`
|
|
|
|
CPUCores is the cores a replica of a profile with no GPU is given:
|
|
reserved out of its node's CPUs by placement, as memory is, and what
|
|
its engine computes on (llama.cpp's --threads).
|
|
|
|
Reserved because an engine takes every core it is told about, and two
|
|
on one node that each think they have all of them spin against each
|
|
other: on an 8-vCPU node, two 7B replicas fell from 10 tokens a second
|
|
to under one each. Zero reserves nothing and leaves the thread count to
|
|
the node, which is every profile from before this.
|
|
|
|
### `min_disk_gb`
|
|
|
|
`int`
|
|
|
|
MinDiskGB is the free space a node needs before it can hold these
|
|
weights.
|
|
|
|
Only checked when the node does not already have them: a node holding
|
|
the file needs no room for it, and requiring the space twice would keep
|
|
a model off the one node best placed to run it.
|
|
|
|
Declared rather than derived, because the size of a model that has never
|
|
been fetched anywhere is not knowable from here — and a wrong guess
|
|
fails late, halfway through a download, which is the failure this exists
|
|
to prevent. Zero means unconstrained, and a node that reports no storage
|
|
is not excluded: unknown is not full.
|
|
|
|
### `node_selector`
|
|
|
|
`map[string]string`
|
|
|
|
NodeSelector restricts this profile to nodes carrying every one of these
|
|
labels, with these values.
|
|
|
|
A CPU profile otherwise fits everywhere, because there is no card to be
|
|
free — which is right for a 0.5B model and wrong for a 14B one. Nothing
|
|
in the inventory says how much memory a node has, so the operator says
|
|
which nodes are meant for a model instead of the scheduler inferring it.
|
|
Agents have reported labels since the beginning; this is what reads them.
|
|
|
|
### `weight`
|
|
|
|
`float64`
|
|
|
|
Weight is the router's capacity weight. It should come from ctl bench
|
|
rather than a guess (plan §6.2.1).
|
|
|
|
### `priority`
|
|
|
|
`int`
|
|
|
|
Priority orders profiles during placement; highest that fits wins.
|
|
|
|
### `startup_timeout`
|
|
|
|
`time.Duration`
|
|
|
|
StartupTimeout bounds model loading. A 70B model across two nodes takes
|
|
tens of minutes, so this is per profile rather than global.
|
|
|
|
## Router
|
|
|
|
### `activation_wait`
|
|
|
|
`time.Duration`
|
|
|
|
ActivationWait is how long a request waits for an on-demand model to
|
|
load before the client is told to come back. Zero takes the default.
|
|
|
|
The one real decision in on-demand loading. Refusing outright makes the
|
|
feature useless, since the first request for every model would fail;
|
|
waiting without a bound turns a request into a job, and a client that
|
|
budgeted thirty seconds ends up holding a connection for ten minutes.
|
|
|
|
### `scrape_interval`
|
|
|
|
`time.Duration`
|
|
|
|
### `scrape_timeout`
|
|
|
|
`time.Duration`
|
|
|
|
### `stale_after`
|
|
|
|
`time.Duration`
|
|
|
|
StaleAfter is how long an endpoint may go unscraped before being scored
|
|
down (plan §7.6).
|
|
|
|
### `retries`
|
|
|
|
`int`
|
|
|
|
Retries is how many *additional* endpoints may be tried after a failure,
|
|
and only while nothing has been written to the client (plan §7.7).
|
|
|
|
### `cooldown`
|
|
|
|
`time.Duration`
|
|
|
|
Cooldown is how long a failing endpoint leaves the candidate pool.
|
|
|
|
### `failure_threshold`
|
|
|
|
`int`
|
|
|
|
FailureThreshold is the consecutive failure count that opens a breaker.
|
|
1 turns the breaker into a pure cooldown.
|
|
|
|
### `default_profile`
|
|
|
|
`string`
|
|
|
|
DefaultProfile names the routing profile used when a model picks none.
|
|
|
|
### `prefix`
|
|
|
|
`Prefix`
|
|
|
|
Prefix turns on the approximate prefix tree (plan §7.5 tier 1). Off, the
|
|
router uses session affinity alone, which is tier 0 and already most of
|
|
the win for multi-turn traffic.
|
|
|
|
Takes a [Prefix](#prefix) block.
|
|
|
|
### `session_header`
|
|
|
|
`string`
|
|
|
|
SessionHeader carries the session id for affinity (plan §7.5 tier 0).
|
|
|
|
### `inject_usage`
|
|
|
|
`bool`
|
|
|
|
InjectUsage asks engines for usage even when the client did not, so token
|
|
accounting works regardless of client behaviour.
|
|
|
|
### `tenant_label`
|
|
|
|
`string`
|
|
|
|
TenantLabel is the level of the tree the per-tenant request series are
|
|
labelled by — llmgw_tenant_requests_total and its siblings: "org",
|
|
"team" or "project", or "none" for no such series. A label per project
|
|
on a cluster of thousands is more series than Prometheus wants, and a
|
|
team is what an alert is usually about; "team" is the default.
|
|
|
|
### `tag_labels`
|
|
|
|
`[]string`
|
|
|
|
TagLabels are the request tags (metadata, x-llmgw-tags) counted in
|
|
llmgw_tagged_requests_total and llmgw_tagged_tokens_total, by value.
|
|
Only these: a tag is the client's to set, and every value of one is
|
|
a series. The first 100 values of each are counted apart, the rest as
|
|
"other". Empty counts none; the usage records keep every tag anyway.
|
|
|
|
### `max_body_bytes`
|
|
|
|
`int64`
|
|
|
|
MaxBodyBytes bounds the request body the router will read.
|
|
|
|
### `batch`
|
|
|
|
`Batch`
|
|
|
|
Batch is the Files and Batch APIs (/v1/files, /v1/batches): a file of
|
|
requests served at the lowest priority, its answers written to
|
|
another. Needs the store.
|
|
|
|
Takes a [Batch](#batch) block.
|
|
|
|
### `cache`
|
|
|
|
`Cache`
|
|
|
|
Cache bounds the answers kept for the projects that reuse them
|
|
(llmgw keys cache). Which projects do, and for how long, is theirs
|
|
to set; this is only how much memory it may take.
|
|
|
|
Takes a [Cache](#cache) block.
|
|
|
|
### `trusted_proxies`
|
|
|
|
`[]string`
|
|
|
|
TrustedProxies are the reverse proxies in front of the client port,
|
|
as addresses or CIDR prefixes. Their X-Forwarded-For, -Proto and -Host
|
|
are believed; anyone else's are not, since they decide the address
|
|
written into the setup scripts and whose attempts are counted where.
|
|
|
|
### `timeouts`
|
|
|
|
`Timeouts`
|
|
|
|
Takes a [Timeouts](#timeouts) block.
|
|
|
|
### `scoring`
|
|
|
|
`Scoring`
|
|
|
|
Takes a [Scoring](#scoring) block.
|
|
|
|
### `admission`
|
|
|
|
`Admission`
|
|
|
|
Takes a [Admission](#admission) block.
|
|
|
|
## Rules
|
|
|
|
### `pii`
|
|
|
|
`map[string]string`
|
|
|
|
PII maps a kind (email, phone, iban, card, tckn, ip, secret) to what is
|
|
done when it is found: flag, redact or block.
|
|
|
|
### `injection`
|
|
|
|
`string`
|
|
|
|
Injection is what is done on signs of prompt injection: flag or block.
|
|
|
|
### `deny`
|
|
|
|
`[]Pattern`
|
|
|
|
Deny are patterns, matched against the text as a model reads it.
|
|
|
|
### `max_chars`
|
|
|
|
`int`
|
|
|
|
MaxChars refuses text longer than this; zero is no limit.
|
|
|
|
## SCIM
|
|
|
|
### `token_file`
|
|
|
|
`string`
|
|
|
|
TokenFile holds the bearer token the provider sends. Empty turns
|
|
provisioning off.
|
|
|
|
## SGLangConfig
|
|
|
|
### `python`
|
|
|
|
`string`
|
|
|
|
Python is the interpreter of the SGLang virtualenv. SGLang is launched as
|
|
a module rather than a console script, which is the documented entry
|
|
point and avoids depending on the script being on PATH.
|
|
|
|
## SLO
|
|
|
|
### `ttft_p99`
|
|
|
|
`time.Duration`
|
|
|
|
TTFTp99 is the time to the first token 99 requests in 100 are within.
|
|
|
|
### `tpot_p99`
|
|
|
|
`time.Duration`
|
|
|
|
TPOTp99 is the time per output token, after the first, 99 streamed
|
|
requests in 100 are within.
|
|
|
|
## ScaleWindow
|
|
|
|
### `days`
|
|
|
|
`[]string`
|
|
|
|
Days are mon … sun, or weekdays, weekends; empty is every day.
|
|
|
|
### `from`
|
|
|
|
`string`
|
|
|
|
From and To are HH:MM; a window whose To is before its From runs
|
|
past midnight.
|
|
|
|
### `to`
|
|
|
|
`string`
|
|
|
|
### `min`
|
|
|
|
`int`
|
|
|
|
### `max`
|
|
|
|
`int`
|
|
|
|
Max, when set, is the window's ceiling for load scaling; zero keeps
|
|
the model's own.
|
|
|
|
## Scaling
|
|
|
|
### `min`
|
|
|
|
`int`
|
|
|
|
### `max`
|
|
|
|
`int`
|
|
|
|
### `idle_timeout`
|
|
|
|
`time.Duration`
|
|
|
|
IdleTimeout is how long an on-demand model may go unasked-for before its
|
|
replicas are stopped. Zero takes the cluster default.
|
|
|
|
Minutes rather than seconds, always. Stopping a replica throws away the
|
|
KV cache prefix routing spent its effort filling, and a model that flaps
|
|
in and out loses that entirely — so the saving has to be worth more than
|
|
the cache, which it only is over a real idle period.
|
|
|
|
### `schedule`
|
|
|
|
`[]ScaleWindow`
|
|
|
|
Schedule changes min and max by the time of day, in the control
|
|
plane's control.timezone: more replicas kept through working hours,
|
|
fewer at night, an on-demand model kept loaded while people are at
|
|
work. The first window that holds now wins; outside them all, Min and
|
|
Max apply.
|
|
|
|
### `prewarm`
|
|
|
|
`time.Duration`
|
|
|
|
Prewarm starts a window this long before it opens, so its replicas
|
|
have loaded by then: loading a model is minutes, and the people who
|
|
arrive at nine are the reason for the window.
|
|
|
|
## Scoring
|
|
|
|
### `target_waiting`
|
|
|
|
`float64`
|
|
|
|
### `target_inflight`
|
|
|
|
`float64`
|
|
|
|
### `kv_ceiling`
|
|
|
|
`float64`
|
|
|
|
### `session_load_factor`
|
|
|
|
`float64`
|
|
|
|
## Secrets
|
|
|
|
### `vault`
|
|
|
|
`secrets.Vault`
|
|
|
|
Vault is how vault: references and vault-transit: are reached.
|
|
|
|
Takes a [Vault](#vault) block.
|
|
|
|
### `master_key`
|
|
|
|
`string`
|
|
|
|
MasterKey seals the secrets the store keeps for itself — what every
|
|
engine's key is derived from — so a copy of the store, or of a backup,
|
|
does not hold them: vault-transit:KEY seals with Vault's transit engine,
|
|
which never lets the key out; otherwise a reference to 32 random bytes
|
|
(openssl rand -hex 32), which then must be kept as safely as the store.
|
|
Values from before it was set are sealed the first time they are read.
|
|
|
|
## Server
|
|
|
|
### `listen`
|
|
|
|
`string`
|
|
|
|
Listen is the client-facing OpenAI API address.
|
|
|
|
### `admin_listen`
|
|
|
|
`string`
|
|
|
|
AdminListen serves the dashboard, the management API, health and pprof.
|
|
|
|
Loopback by default, and it should stay there: pprof can dump the process
|
|
heap, and API keys are in that heap in plaintext. Every comment beside
|
|
this setting has always said not to expose it, while the default bound
|
|
every interface — a default that contradicts its own documentation is a
|
|
footgun, so the default now matches the advice.
|
|
|
|
### `metrics_listen`
|
|
|
|
`string`
|
|
|
|
MetricsListen serves Prometheus metrics and nothing else.
|
|
|
|
Separate from AdminListen because the reason that one is loopback is
|
|
pprof, not metrics — and metrics are the one thing on it that something
|
|
else genuinely needs to reach. Sharing a listener meant a fleet could
|
|
either be monitored or keep its heap private, which is not a choice
|
|
anybody should have to make.
|
|
|
|
Empty means off, and metrics remain available on AdminListen either way,
|
|
so nothing that scrapes it today has to change.
|
|
|
|
### `agent_listen`
|
|
|
|
`string`
|
|
|
|
AgentListen is where node agents connect. Separate from the client port
|
|
so the fleet-facing surface can be firewalled independently.
|
|
|
|
### `auth`
|
|
|
|
`Auth`
|
|
|
|
Auth turns on API keys on the client-facing port.
|
|
|
|
Takes a [Auth](#auth) block.
|
|
|
|
### `threads`
|
|
|
|
`Threads`
|
|
|
|
Threads keeps conversations and agent memory without API keys, for a
|
|
gateway on one trusted machine. With auth on they live in the auth store,
|
|
per project, and this block is ignored.
|
|
|
|
Takes a [Threads](#threads) block.
|
|
|
|
### `landing`
|
|
|
|
`Landing`
|
|
|
|
Landing configures the page served at "/".
|
|
|
|
Takes a [Landing](#landing) block.
|
|
|
|
### `agent_tls`
|
|
|
|
`tlsconf.Config`
|
|
|
|
AgentTLS authenticates the fleet-facing port. It deliberately does not
|
|
cover the client-facing one: that surface is fronted by a publicly
|
|
trusted certificate, while this one is mutual and private.
|
|
|
|
Takes a [tlsconf.Config](#tlsconfconfig) block.
|
|
|
|
### `max_connections`
|
|
|
|
`int`
|
|
|
|
MaxConnections bounds the client port's open connections; past it new
|
|
ones wait in the kernel's backlog. 0 is no bound.
|
|
|
|
### `admin_token_file`
|
|
|
|
`string`
|
|
|
|
AdminTokenFile holds the token the admin API's /api and /debug paths
|
|
demand. Empty is admin-token in the systemd unit's state directory
|
|
(/var/lib/llmgw), or secrets/admin-token beside this file for a server
|
|
run by hand. Made, 0600, when the server starts and it does not exist;
|
|
`sudo llmgw ctl` and the console read it from there.
|
|
|
|
### `admin_remote`
|
|
|
|
`bool`
|
|
|
|
AdminRemote allows admin_listen on an address other than loopback. The
|
|
token still guards the API, but it then crosses the network in clear.
|
|
|
|
### `debug_pprof`
|
|
|
|
`bool`
|
|
|
|
DebugPprof serves Go's profiler on the admin listener, behind the token.
|
|
Off by default: a heap profile holds the API keys requests presented.
|
|
|
|
### `agent_insecure`
|
|
|
|
`bool`
|
|
|
|
AgentInsecure lets a control plane run its agent port without
|
|
agent_tls: plaintext, and any machine that reaches it may register as
|
|
any node — be given replicas, and so be sent users' prompts — or, in
|
|
the other direction, a machine in the path may tell every agent what
|
|
to run. Refused unless said here, and then the port listens on
|
|
loopback unless agent_listen names an address. For one machine, a
|
|
test, or a network nothing untrusted reaches.
|
|
|
|
### `monitoring`
|
|
|
|
`Monitoring`
|
|
|
|
Monitoring is where the dashboard reads its history from.
|
|
|
|
Takes a [Monitoring](#monitoring) block.
|
|
|
|
### `tracing`
|
|
|
|
`obs.Tracing`
|
|
|
|
Tracing sends traces to an OpenTelemetry collector: each inference
|
|
request, and on a control plane each placement, replica start and
|
|
model job. Off unless an endpoint is set here or in
|
|
OTEL_EXPORTER_OTLP_ENDPOINT.
|
|
|
|
Takes a [Tracing](#tracing) block.
|
|
|
|
### `mirror`
|
|
|
|
`Mirror`
|
|
|
|
Mirror points the control plane at a model mirror it can administer.
|
|
Empty means there is none, and /admin/mirror stays unmounted.
|
|
|
|
Takes a [Mirror](#mirror) block.
|
|
|
|
### `engines`
|
|
|
|
`map[string]string`
|
|
|
|
Engines are the engines every node installs, and at which version. The
|
|
agents install them themselves — the right build for their hardware,
|
|
checked against its publisher's checksum and seen to run — so a node is
|
|
never trusted to already have what somebody once put there.
|
|
|
|
Unset keeps DefaultEngines. A version replaces the default; an empty
|
|
one leaves that engine to be installed by hand.
|
|
|
|
### `engine_overrides`
|
|
|
|
`[]EngineOverride`
|
|
|
|
EngineOverrides give some nodes different engines from the rest: a
|
|
vLLM built for CUDA 12 on the Volta cards CUDA 13 dropped, or none at
|
|
all on nodes meant only for llama.cpp. Each picks nodes and lays its
|
|
engines over the cluster's, in order, later ones last; an empty version
|
|
removes that engine from the nodes it picks.
|
|
|
|
### `backup_dir`
|
|
|
|
`string`
|
|
|
|
BackupDir is where `llmgw server backup` writes, and where the newest
|
|
archive's age is read for llmgw_backup_last_success_timestamp_seconds.
|
|
Empty means /var/backups/llmgw-server.
|
|
|
|
### `guard`
|
|
|
|
`guard.Config`
|
|
|
|
Guard is the library of guardrail policies: what requests and answers
|
|
are checked for, and what is done. Applied to every request through
|
|
guard.default, and to parts of the tenancy tree by attaching them
|
|
(llmgw ctl guard attach). Empty checks nothing.
|
|
|
|
Takes a [guard.Config](#guardconfig) block.
|
|
|
|
### `secrets`
|
|
|
|
`Secrets`
|
|
|
|
Secrets is where secrets kept outside files are read from, and the key
|
|
the store's own secrets are sealed under.
|
|
|
|
Takes a [Secrets](#secrets) block.
|
|
|
|
### `notify`
|
|
|
|
`notify.Config`
|
|
|
|
Notify sends what needs a person — a model parked, a node lost, a
|
|
budget spent, new weights published — to webhooks, signed, without a
|
|
Prometheus and Alertmanager in between. Empty sends nothing.
|
|
|
|
Takes a [notify.Config](#notifyconfig) block.
|
|
|
|
### `firewall_networks`
|
|
|
|
`[]string`
|
|
|
|
FirewallNetworks are networks, besides those each machine reaches the
|
|
cluster on, that llmgw's internal ports answer: a monitoring host on
|
|
another subnet, nodes behind a router. Written into every machine's
|
|
firewall by `server init`, `node add` and `server add`.
|
|
|
|
### `backup_recipients`
|
|
|
|
`[]string`
|
|
|
|
BackupRecipients encrypt every archive `llmgw server backup` writes, to
|
|
whoever holds one of their keys: age recipients (age1…) or SSH public
|
|
keys (ssh-ed25519 …, ssh-rsa …), as `age` takes them. An archive holds
|
|
the certificate authority's key and every secret in the store, and is
|
|
copied off the machine; encrypted, the copy is safe wherever it lands.
|
|
Empty writes archives unencrypted, readable by root only.
|
|
|
|
### `backup_copy_to`
|
|
|
|
`[]string`
|
|
|
|
BackupCopyTo is where each archive is copied once written, so it
|
|
survives this machine: a directory (a mounted share, another disk) or
|
|
ssh://user@host[:port]/path, reached with BackupCopyKey and checked
|
|
against /etc/llmgw/backup_known_hosts, which learns a host the first
|
|
time. The same number of archives is kept there as here. Refused
|
|
without backup_recipients: an archive leaves the machine encrypted.
|
|
|
|
### `backup_copy_key`
|
|
|
|
`string`
|
|
|
|
BackupCopyKey is the SSH private key for ssh:// copies.
|
|
|
|
### `releases`
|
|
|
|
`string`
|
|
|
|
Releases is where `llmgw ctl upgrade` fetches release binaries from,
|
|
for this machine and — through their agents — every node. Empty means
|
|
the project's own releases.
|
|
|
|
### `router`
|
|
|
|
`Router`
|
|
|
|
Takes a [Router](#router) block.
|
|
|
|
### `control`
|
|
|
|
`Control`
|
|
|
|
Takes a [Control](#control) block.
|
|
|
|
### `models`
|
|
|
|
`[]catalog.Model`
|
|
|
|
### `federation`
|
|
|
|
`Federation`
|
|
|
|
Federation is how this cluster takes part in federation with other
|
|
llmgw clusters. The peers themselves are added at runtime, with
|
|
`llmgw ctl federation add`, and kept in the store.
|
|
|
|
Takes a [Federation](#federation) block.
|
|
|
|
### `endpoints`
|
|
|
|
`[]Endpoint`
|
|
|
|
Endpoints is the phase-1 static replica list. It is only consulted when
|
|
no model declares a profile, which is what makes a router-only smoke test
|
|
against sim-engines possible without a control plane.
|
|
|
|
## SimConfig
|
|
|
|
### `bin`
|
|
|
|
`string`
|
|
|
|
Bin is the llmgw binary itself; the engine is one of its subcommands.
|
|
Empty means whatever started this agent, which is right almost always.
|
|
|
|
## TRTLLMConfig
|
|
|
|
### `python`
|
|
|
|
`string`
|
|
|
|
Python is the interpreter of the environment TensorRT-LLM is installed
|
|
in: <venv>/bin/python for a virtual environment, python3 in NVIDIA's
|
|
container. Naming it is what makes this node offer the engine; empty
|
|
offers nothing.
|
|
|
|
The interpreter rather than trtllm-serve, because llmgw starts the
|
|
server through a few lines of its own that make it demand the
|
|
gateway's key, which trtllm-serve has no option for.
|
|
|
|
## Threads
|
|
|
|
### `store`
|
|
|
|
`string`
|
|
|
|
Store is the SQLite file. Empty means nothing is kept.
|
|
|
|
### `retain_hours`
|
|
|
|
`int`
|
|
|
|
RetainHours is how long idle conversations and notes are kept. Required
|
|
with Store: a store with no expiry keeps everything anyone typed, by
|
|
omission.
|
|
|
|
## Timeouts
|
|
|
|
### `dial`
|
|
|
|
`time.Duration`
|
|
|
|
### `response_header`
|
|
|
|
`time.Duration`
|
|
|
|
ResponseHeader bounds the wait for headers on a streaming request. It is
|
|
a liveness check there, because headers arrive before generation starts.
|
|
|
|
### `generation`
|
|
|
|
`time.Duration`
|
|
|
|
Generation bounds a non-streaming request, where the engine sends no
|
|
headers until the completion is done. Setting response_header alone used
|
|
to cap both, so 60s read as a liveness guard and behaved as a limit of
|
|
about a hundred tokens on a slow model.
|
|
|
|
### `first_token`
|
|
|
|
`time.Duration`
|
|
|
|
### `idle`
|
|
|
|
`time.Duration`
|
|
|
|
### `total`
|
|
|
|
`time.Duration`
|
|
|
|
## Tools
|
|
|
|
### `allow`
|
|
|
|
`[]string`
|
|
|
|
Allow, when set, are the only tool names a request may offer and a
|
|
model may call.
|
|
|
|
### `deny`
|
|
|
|
`[]string`
|
|
|
|
Deny are tool names never offered or called.
|
|
|
|
### `on_denied`
|
|
|
|
`string`
|
|
|
|
OnDenied is strip (the tool is removed from the request) or block.
|
|
Empty is strip.
|
|
|
|
### `max_calls_per_turn`
|
|
|
|
`int`
|
|
|
|
MaxCallsPerTurn stops an answer calling more tools than this.
|
|
|
|
### `validate_arguments`
|
|
|
|
`bool`
|
|
|
|
ValidateArguments checks a call's arguments against the parameters
|
|
schema the request declared for the tool.
|
|
|
|
## Tracing
|
|
|
|
### `endpoint`
|
|
|
|
`string`
|
|
|
|
Endpoint is the collector, for example http://otel.internal:4318.
|
|
Empty falls back to OTEL_EXPORTER_OTLP_ENDPOINT, and to no tracing.
|
|
|
|
### `sample_ratio`
|
|
|
|
`float64`
|
|
|
|
SampleRatio is the share of traces kept, 0 to 1, of those nobody
|
|
upstream already decided. Zero takes 1. A request with a sampled
|
|
traceparent of its own is kept, and an agent keeps every start the
|
|
control plane kept, whatever their own ratio says.
|
|
|
|
## VLLMConfig
|
|
|
|
### `bin`
|
|
|
|
`string`
|
|
|
|
Bin is the vllm executable.
|
|
|
|
### `launch_mode`
|
|
|
|
`string`
|
|
|
|
LaunchMode selects the multi-node mechanism: "mp", the default, or
|
|
"ray".
|
|
|
|
mp is what vLLM documents today — every node runs serve with its rank,
|
|
the workers headless — and needs nothing but vLLM. ray is the older way,
|
|
kept for a vLLM too old for mp's multi-node flags; it needs Ray on every
|
|
node and has not been checked against a real cluster.
|
|
|
|
### `ray_bin`
|
|
|
|
`string`
|
|
|
|
RayBin is used when LaunchMode is "ray".
|
|
|
|
## Vault
|
|
|
|
### `addr`
|
|
|
|
`string`
|
|
|
|
Addr is Vault's address, like https://vault.internal:8200; empty
|
|
takes VAULT_ADDR.
|
|
|
|
### `token_file`
|
|
|
|
`string`
|
|
|
|
TokenFile holds a token; empty takes VAULT_TOKEN, or logs in with
|
|
AppRole when RoleIDFile and SecretIDFile are set.
|
|
|
|
### `role_id_file`
|
|
|
|
`string`
|
|
|
|
### `secret_id_file`
|
|
|
|
`string`
|
|
|
|
### `ca_file`
|
|
|
|
`string`
|
|
|
|
CAFile verifies Vault's certificate; empty uses the system's.
|
|
|
|
### `namespace`
|
|
|
|
`string`
|
|
|
|
Namespace is Vault Enterprise's or OpenBao's namespace.
|
|
|
|
## Webhook
|
|
|
|
### `url`
|
|
|
|
`string`
|
|
|
|
### `format`
|
|
|
|
`string`
|
|
|
|
Format is json (the event, with X-Llmgw-Signature when SecretFile is
|
|
set) or slack ({"text": …}, which Slack, Mattermost and Rocket.Chat
|
|
incoming webhooks take).
|
|
|
|
### `events`
|
|
|
|
`[]string`
|
|
|
|
Events are the kinds sent here; empty is all.
|
|
|
|
### `secret_file`
|
|
|
|
`string`
|
|
|
|
SecretFile holds the key the body is signed with: X-Llmgw-Signature
|
|
is sha256=<hex HMAC of the body>.
|
|
|
|
## accounting.Price
|
|
|
|
### `prompt_per_1k`
|
|
|
|
`int64`
|
|
|
|
### `completion_per_1k`
|
|
|
|
`int64`
|
|
|
|
## admission.Band
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
### `priority`
|
|
|
|
`int`
|
|
|
|
Priority orders bands; higher is served first.
|
|
|
|
### `saturation_ceiling`
|
|
|
|
`float64`
|
|
|
|
SaturationCeiling is the pool saturation above which this band waits.
|
|
1.0 means "never hold this band back".
|
|
|
|
## admission.Config
|
|
|
|
### `bands`
|
|
|
|
`[]Band`
|
|
|
|
### `default_band`
|
|
|
|
`string`
|
|
|
|
### `queue_ttl`
|
|
|
|
`time.Duration`
|
|
|
|
QueueTTL drops a request that has waited this long. Sending the GPU work
|
|
the client has already given up on helps nobody.
|
|
|
|
### `queue_capacity_per_band`
|
|
|
|
`int`
|
|
|
|
QueueCapacityPerBand bounds each band's queue. Rejecting immediately
|
|
beats accumulating silently.
|
|
|
|
### `queue_capacity_total`
|
|
|
|
`int`
|
|
|
|
QueueCapacityTotal bounds everything queued, across bands. When it is
|
|
reached, a request is not refused while a lower band has requests
|
|
queued: the newest of the lowest band is turned away instead
|
|
(displacement), since the oldest have waited longest and are nearest
|
|
their TTL. Zero takes half of every band's capacity together.
|
|
|
|
### `interval`
|
|
|
|
`time.Duration`
|
|
|
|
Interval is how often the dispatcher re-checks saturation. Releases also
|
|
wake it, so this is only the backstop for saturation falling on its own.
|
|
|
|
## agent.Config
|
|
|
|
### `node_id`
|
|
|
|
`string`
|
|
|
|
NodeID defaults to the hostname. It must equal the common name of the
|
|
agent's certificate, so set it only where the two differ.
|
|
|
|
### `server`
|
|
|
|
`string`
|
|
|
|
Server is the control plane, and Servers is the same thing when there is
|
|
more than one. Both are accepted: every configuration written before
|
|
there could be a second control plane names one address, and none of
|
|
them should have to change.
|
|
|
|
The order is tried in order, and a refusal moves to the next. Only the
|
|
leader accepts an agent, so a standby is skipped in the time it takes to
|
|
be told so — which is what makes failover a reconnect rather than an
|
|
edit to every node's configuration.
|
|
|
|
### `servers`
|
|
|
|
`[]string`
|
|
|
|
### `hostname`
|
|
|
|
`string`
|
|
|
|
Hostname defaults to the machine's.
|
|
|
|
### `mgmt_addr`
|
|
|
|
`string`
|
|
|
|
MgmtAddr is the address engines are reached on. It defaults to this
|
|
machine's address on the route to the control plane, which is the one
|
|
the control plane and the router can reach.
|
|
|
|
### `labels`
|
|
|
|
`map[string]string`
|
|
|
|
Labels are free-form facts about a node for a profile's node_selector.
|
|
Nothing needs them: memory, GPUs, engines and disk are measured, and a
|
|
label restating one of those goes stale the day the machine changes.
|
|
|
|
### `mgmt_interface`
|
|
|
|
`string`
|
|
|
|
MgmtInterface is the ethernet interface NCCL bootstraps over. It
|
|
defaults to the interface holding MgmtAddr, then eth0.
|
|
|
|
### `fabrics`
|
|
|
|
`[]inventory.FabricAttachment`
|
|
|
|
Fabrics labels this node's HCAs. It is explicit on purpose: OpenSM's
|
|
default subnet prefix is the same on every fabric, so nothing can be
|
|
inferred from the wire (plan §5.2).
|
|
|
|
### `heartbeat_interval`
|
|
|
|
`time.Duration`
|
|
|
|
### `reconnect_base`
|
|
|
|
`time.Duration`
|
|
|
|
ReconnectBase is the first reconnect delay; it backs off from there.
|
|
|
|
### `reconnect_max`
|
|
|
|
`time.Duration`
|
|
|
|
### `adopt_ready_timeout`
|
|
|
|
`time.Duration`
|
|
|
|
AdoptReadyTimeout bounds how long an adopted engine is given to answer a
|
|
health check before the agent stops waiting on it.
|
|
|
|
Shorter than a startup timeout on purpose: this engine is already
|
|
running, so it is not loading a model, it is either serving or it is not.
|
|
|
|
### `tls`
|
|
|
|
`tlsconf.Config`
|
|
|
|
TLS authenticates both ends of the control-plane stream. Leaving it unset
|
|
keeps the plaintext behaviour, which only belongs on a trusted network.
|
|
|
|
Takes a [tlsconf.Config](#tlsconfconfig) block.
|
|
|
|
### `model_dirs`
|
|
|
|
`[]string`
|
|
|
|
ModelDirs are the directories this node keeps model weights in, beyond
|
|
WeightsCache, which is always looked in. Storage differs at every installation,
|
|
so llmgw observes it rather than owning it: a Lustre mount, an image
|
|
exported over the fabric, local NVMe and a copy made by hand all look
|
|
identical from here, and placement only needs to know whether the file
|
|
is present.
|
|
|
|
### `engines_dir`
|
|
|
|
`string`
|
|
|
|
EnginesDir is where the agent installs engines, one directory per
|
|
build. Defaults to /opt/llmgw/engines.
|
|
|
|
### `weights_cache`
|
|
|
|
`string`
|
|
|
|
WeightsCache is where a remote source is downloaded to. Defaults to the
|
|
first entry of ModelDirs, because a node that says where its models live
|
|
has already answered this question, and otherwise to
|
|
/var/lib/llmgw-weights.
|
|
|
|
Fetching is the simple arrangement: every node keeps its own copy. The
|
|
exotic ones need no code — weights on a shared filer or an image exported
|
|
over the fabric are simply present, and present means nothing is fetched.
|
|
|
|
### `discovery`
|
|
|
|
`discovery.Options`
|
|
|
|
Takes a [Options](#options) block.
|
|
|
|
### `engines`
|
|
|
|
`engines.Config`
|
|
|
|
Takes a [engines.Config](#enginesconfig) block.
|
|
|
|
### `supervisor`
|
|
|
|
`supervisor.Config`
|
|
|
|
Takes a [supervisor.Config](#supervisorconfig) block.
|
|
|
|
### `tracing`
|
|
|
|
`obs.Tracing`
|
|
|
|
Tracing sends this node's part of each replica start — the weights
|
|
fetched, the engine launched, the wait for it to answer — to an
|
|
OpenTelemetry collector, inside the control plane's trace of that
|
|
start. Give it the collector server.yaml's tracing names. Off unless
|
|
an endpoint is set here or in OTEL_EXPORTER_OTLP_ENDPOINT.
|
|
|
|
Takes a [Tracing](#tracing) block.
|
|
|
|
## config.Band
|
|
|
|
### `name`
|
|
|
|
`string`
|
|
|
|
### `priority`
|
|
|
|
`int`
|
|
|
|
### `saturation_ceiling`
|
|
|
|
`float64`
|
|
|
|
## config.Price
|
|
|
|
### `prompt_per_1k`
|
|
|
|
`int64`
|
|
|
|
### `completion_per_1k`
|
|
|
|
`int64`
|
|
|
|
## engines.Config
|
|
|
|
### `disabled`
|
|
|
|
`[]string`
|
|
|
|
Disabled are engines this node never installs or advertises, whatever
|
|
the control plane asks: vLLM on a node of cards no build of it runs on,
|
|
that is meant for llama.cpp alone.
|
|
|
|
### `vllm`
|
|
|
|
`VLLMConfig`
|
|
|
|
Takes a [VLLMConfig](#vllmconfig) block.
|
|
|
|
### `sglang`
|
|
|
|
`SGLangConfig`
|
|
|
|
Takes a [SGLangConfig](#sglangconfig) block.
|
|
|
|
### `llamacpp`
|
|
|
|
`LlamaCppConfig`
|
|
|
|
Takes a [LlamaCppConfig](#llamacppconfig) block.
|
|
|
|
### `trtllm`
|
|
|
|
`TRTLLMConfig`
|
|
|
|
TRTLLM is TensorRT-LLM, which the agent does not install: a node
|
|
offers it only when its interpreter is named here.
|
|
|
|
Takes a [TRTLLMConfig](#trtllmconfig) block.
|
|
|
|
### `sim`
|
|
|
|
`SimConfig`
|
|
|
|
Takes a [SimConfig](#simconfig) block.
|
|
|
|
## guard.Config
|
|
|
|
### `default`
|
|
|
|
`[]string`
|
|
|
|
### `policies`
|
|
|
|
`map[string]Policy`
|
|
|
|
## notify.Config
|
|
|
|
### `webhooks`
|
|
|
|
`[]Webhook`
|
|
|
|
### `host`
|
|
|
|
`string`
|
|
|
|
Host names the gateway in messages; empty is this machine's name.
|
|
|
|
## supervisor.Config
|
|
|
|
### `work_dir`
|
|
|
|
`string`
|
|
|
|
WorkDir holds pid files and per-shard scratch space.
|
|
|
|
### `log_lines`
|
|
|
|
`int`
|
|
|
|
LogLines is how many lines of engine output to retain per shard.
|
|
|
|
### `stop_timeout`
|
|
|
|
`time.Duration`
|
|
|
|
StopTimeout is how long a process gets to exit after SIGTERM before it is
|
|
killed. Engines need time to free GPU memory cleanly.
|
|
|
|
### `health_interval`
|
|
|
|
`time.Duration`
|
|
|
|
HealthInterval is how often a starting shard is polled.
|
|
|
|
### `rendezvous_timeout`
|
|
|
|
`time.Duration`
|
|
|
|
RendezvousTimeout bounds the wait for a multi-node rendezvous point to
|
|
start listening.
|
|
|
|
### `alive_grace`
|
|
|
|
`time.Duration`
|
|
|
|
AliveGrace is how long a non-serving rank must stay up before it counts
|
|
as ready. There is no external signal for a worker, so staying alive is
|
|
the only evidence available.
|
|
|
|
### `run_as`
|
|
|
|
`string`
|
|
|
|
RunAs is the user engines run as when the agent is root; empty is
|
|
llmgw-engine, "root" keeps them root.
|
|
|
|
## tlsconf.Config
|
|
|
|
### `ca`
|
|
|
|
`string`
|
|
|
|
CA is the bundle used to verify the other side. In a real deployment this
|
|
is whichever authority issued the fleet's certificates.
|
|
|
|
### `cert`
|
|
|
|
`string`
|
|
|
|
Cert and Key are this component's own identity.
|
|
|
|
### `key`
|
|
|
|
`string`
|
|
|
|
### `insecure`
|
|
|
|
`bool`
|
|
|
|
Insecure disables TLS entirely. It exists for local runs against
|
|
sim-engines; anything using it must stay on a trusted network.
|
|
|