Chat goes to the replica that has been answering soonest
CI / test (push) Failing after 7m18s

The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 17:01:30 +01:00
1 parent 0b37796067
commit 73e1bb6733
9 files changed
+106 -3

No files matched your search

+3 -2
View File
@@ -313,8 +313,9 @@ within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.
Routing profiles select how endpoints are scored: `chat` favours session
affinity, `batch` favours the shortest queue, `long-context` favours free KV
cache. Pick one per model in the config, or per request with the
affinity and the replica that has been answering soonest (its time to first
token, smoothed), `batch` favours the shortest queue, `long-context` favours
free KV cache. Pick one per model in the config, or per request with the
`x-llmgw-profile` header.
## sim-engine