Chat goes to the replica that has been answering soonest

The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-25 17:01:30 +01:00
1 parent 0b37796067
commit 73e1bb6733
9 files changed
+106 -3

No files matched your search

+4 -1
View File
@@ -1728,7 +1728,10 @@ templated HTTP call with an expression for "block".
what it says to, and a service that does not answer is treated as `fail`
says.
### L4. Routing by latency and by cost, and failover by priority (M)
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
what `fallbacks` already does (local, then an outside model when no room). Left:
routing by cost between several outside providers. The original item follows.
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
and KV cache. Kong also balances by measured latency, by tokens or cost
used, and in priority tiers.