Files
llmgw/internal/router/endpoint
Ilker ManapandClaude Opus 5.5 73e1bb6733
CI / test (push) Failing after 7m18s
Chat goes to the replica that has been answering soonest
The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-25 17:01:30 +01:00
..