An engine that answers at once and the gateway in front of it, in one
process on loopback, given the same load straight and through the gateway:
bare, with authentication and accounting, and with the guardrails' pattern
checks. It reports p50 and p99, what the gateway added, and a rough CPU
per thousand requests.
On an Apple M5 at 100 to 2,000 requests a second (10 s a step, 72,000
requests, none failed), the gateway added 0.05–0.42 ms at p50 and at most
0.49 ms at p99. That is now in the README, with how it was measured and
why the CPU figure is rough. CI runs a one-second version.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The sampler sent the key given for --target, which for the gateway is a
client's, so llama.cpp and TensorRT-LLM refused every read and the run
reported an idle engine. The engine key now comes from the admin API for the
replica at --metrics' address, or --metrics-key; unreadable metrics stop the
run before any load, and a step that read none shows why.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engine trtllm runs trtllm-serve on one node, through a launcher of
llmgw's that demands the replica's key and turns on the Prometheus
endpoint its queue and cache gauges are read from; the router reads it
at /prometheus/metrics. Brought by the operator (engines.trtllm.python),
not installed; checked by ctl engine-check trtllm. Written against the
documented CLI, server and metrics and tested with stand-ins: not yet
run on a datacenter GPU.
mirror import-ollama copies a model's GGUF, and a vision model's
projector, from Ollama's directory into the mirror, checked against the
digest Ollama named each blob by; mirror import does the same for any
local file. The console's Mirror view imports from Ollama with I.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Engines listen on every address their node has, with no key. Checked on the
live cluster: from a neighbouring node, a request straight to an engine's
port was answered — past the gateway's authentication, quotas and
accounting, by anyone who could reach the node.
Each replica's engine is now started with a key and refuses requests without
it. The key is derived from the replica id with an HMAC over a secret kept
in the store, rather than stored per replica: the router has to send the
right key to replicas this process never started — adopted after a restart,
or placed by the other control plane — and a function of a shared secret
gives that without a table to keep in step. A key taken from one node opens
that replica only. The secret is made on first use and copied with the rest
of the store when moving to PostgreSQL; a new one would lock the router out
of every running engine at the moment of the move.
The key reaches llama.cpp and vLLM through their environment
(LLAMA_API_KEY, VLLM_API_KEY) rather than their command line, which every
user on the node can read; SGLang only takes it as an argument. The agent's
readiness check, the router's proxy and its metrics scrape send it, the
client's own Authorization still never reaches the engine, and `ctl bench`
fetches the key of the replica it is pointed at from the admin API.
Engines started before this keep running without a key until their replica
is next started. With no store there is nowhere to keep the secret, and
engines start without one, with a warning.
Also formats two files that went in unformatted.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`ctl bench` has always been able to work out what one replica can serve — it
ramps concurrency until the SLO breaks and reports the last level that held.
It printed that and nothing kept it, so the one number saying what a replica is
worth lived in whoever ran it's scrollback. Everything downstream then had to
guess: admission applies a single global default for healthy queue depth, and
sizing a model against real demand had nothing to size against.
`ctl bench --save` records it; `ctl capacity` reads it back.
Keyed by model and profile. Capacity belongs to the hardware and the engine
arguments as much as to the weights, so the same model on CPU and on an H100
are two numbers, and one key would overwrite the other on every run.
The profile is looked up from the replica rather than asked for, because the
control plane already knows which profile answers on which endpoint and a
measurement filed under the wrong one is worse than none. A target that is not
a replica — the gateway, usually — is refused with that said: benchmarking the
whole path is worth doing and says nothing about the hardware.
The conclusion is stored, not the ramp. Every level and its latencies are worth
reading once while judging whether the run was sound; keeping them would grow
without bound as a model is re-benchmarked.
Re-measuring replaces. A second run is a better answer to the same question,
not another data point, and keeping both would leave a reader choosing between
numbers with no basis for the choice.
The SLO and the date are stored with it. Capacity means nothing without the
target it was measured against, and hardware and engine builds change — a
measurement is an observation with a date on it, not a fact about the model.
A failed save is reported and not fatal: the run has already cost the GPU time.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rows are printed as each concurrency level finishes, and a tabwriter lays out
only what has been flushed — so it aligned the header and the first row to
each other and every row after that to nothing. Fixed columns instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ctl bench` measures what a replica can actually serve, so weight,
target_waiting and kv_ceiling stop being numbers somebody typed. Concurrency
climbs until the SLO breaks and the last level that met it is the capacity:
peak throughput arrives long after latency has become unusable, so throughput
alone overstates what the hardware can serve. Queue depth and KV usage are
read from the engine while it is still meeting the target, which is exactly
what admission should treat as healthy.
Prompts are unique per request by default. With one repeated prompt the
engine's cache answers most of the prefill and the measurement describes the
cache rather than the hardware; --shared-prefix asks that question
deliberately, and it now produces byte-identical prompts, which the first
version did not — it dropped the random header and still picked words at
random, so the cache missed and the flag quietly measured the thing it exists
to avoid.
Running it against the cluster turned up two real defects:
A replica stuck in Starting never timed out. Every other way out of that state
needs the agent to report something, and when it never does the replica sits
there holding GPUs and counting towards the model's replica count while
serving nothing. Two of them had been like that for eight minutes past a
two-minute startup timeout, and nothing would ever have moved them.
They were stuck because adoption lost to a replica we had only placed. Replica
ids come from the model name and a counter, so a control plane that restarts
and places before its agents reconnect builds a different replica under the
same id on a freshly allocated port; the agent's report of the engine that was
actually serving was then discarded as already-known. The cluster pointed at
port 20004 while the engine answered on 20000. The fleet is authoritative
about what is running, so what we merely placed now loses to what an agent
reports — unless our copy has a shard that has actually come up, in which case
two things really are running and the stray is the one we did not ask for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
