Chat goes to the replica that has been answering soonest
The router keeps each endpoint's time to first token, smoothed over its last streamed answers, and the chat profile scores by it beside session, queue, KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet measured 1 so a new replica is tried. Load still sees a slow drain from the engine's own counters; this sees what answers actually took. BACKLOG L4. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
0b37796067
commit
73e1bb6733
9 files changed
+106
-3
No files matched your search
+4
-1
@@ -1728,7 +1728,10 @@ templated HTTP call with an expression for "block".
|
||||
what it says to, and a service that does not answer is treated as `fail`
|
||||
says.
|
||||
|
||||
### L4. Routing by latency and by cost, and failover by priority (M)
|
||||
### L4. Routing by latency and by cost, and failover by priority (M) — latency done 2026-10-25
|
||||
`endpoint.Registry.ObserveTTFT`, `scheduler.Latency()` in `chat`. Failover in tiers is
|
||||
what `fallbacks` already does (local, then an outside model when no room). Left:
|
||||
routing by cost between several outside providers. The original item follows.
|
||||
The scorers (`internal/router/scheduler/scorers.go`) know sessions, queues
|
||||
and KV cache. Kong also balances by measured latency, by tokens or cost
|
||||
used, and in priority tiers.
|
||||
|
||||
Reference in new issue
Block a user