Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.
Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.
The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.
Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.
Also formats two files that went in unformatted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
response_header sounds like a check that the endpoint is alive, and for a
streaming request it is: headers arrive before generation starts. For a
non-streaming request the engine sends nothing until the completion is
finished, so the same setting is the total generation budget wearing a
liveness timeout's name.
At 60s that was invisible on GPUs and a hard cap of roughly a hundred
tokens on qwen2.5-coder-14b at 1.7 tokens a second. A request for 120
tokens failed at exactly one minute, and neither the setting's name nor the
error said why.
Two transports now: the short header timeout for streaming, where it still
catches a sick endpoint, and a generation budget for non-streaming,
defaulting to ten minutes. One number could not mean both things.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw server is real now. It fronts a static set of engine replicas with an
OpenAI-compatible API, scoring endpoints through the pipeline added in the
previous commit.
- proxy: SSE relayed event by event with a flush after each, cancellation
propagated upstream, and separate deadlines for dial, headers, first token,
inter-token idle and total. The idle watchdog is what distinguishes a wedged
engine from a slow one.
- router: dispatch with retry and model fallback. Retry only happens while
nothing has reached the client; after that, splicing two responses together
would be worse than the short stream the client already sees.
- config: YAML via koanf overlaid on defaults, with LLMGW_ env overrides.
Validation rejects duplicate endpoint ids and aliases that shadow a model,
because both misroute silently at request time rather than failing loudly.
- metrics: requests, duration, TTFT, tokens, plus per-scorer values and
per-filter drops on a separate admin listener with health and pprof.
Phase 1 exit criterion, measured in test/e2e: 1000 concurrent streams across
3 sim-engines complete at ~6800 req/s; every client disconnect frees the KV
blocks on every engine; router overhead p99 is about 0.9 ms against the plan's
2 ms budget.
The handlers live in internal/router rather than an internal/router/openai
package as the plan's tree sketched: they need the dispatch loop, and splitting
them apart would have meant an interface with exactly one implementation. Plan
§4 updated to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ilkerandClaude Opus 5