Ilker ManapandClaude Opus 5.5 73e1bb6733 Chat goes to the replica that has been answering soonest ...

The router keeps each endpoint's time to first token, smoothed over its last
streamed answers, and the chat profile scores by it beside session, queue,
KV and inflight: the fastest endpoint 1, one twice as slow 0.5, one not yet
measured 1 so a new replica is tried. Load still sees a slow drain from the
engine's own counters; this sees what answers actually took. BACKLOG L4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-25 17:01:30 +01:00

..

adapters_test.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

adapters.go

LoRA adapters on vLLM, loaded on demand with replica affinity: max_loaded_adapters

2026-10-05 01:14:38 +02:00

filters.go

Spill a saturated pool's requests to the next pool

2026-09-30 23:23:32 +02:00

hashring.go

Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping

2026-09-16 01:15:21 +02:00

kv_test.go

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

latency_test.go

Chat goes to the replica that has been answering soonest

2026-10-25 17:01:30 +01:00

pickers.go

Phase 1 (1/2): scheduler pipeline, endpoint registry, metric scraping

2026-09-16 01:15:21 +02:00

prefix.go

Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held

2026-10-05 01:21:11 +02:00

profiles.go

Chat goes to the replica that has been answering soonest

2026-10-25 17:01:30 +01:00

scheduler_test.go

Spill a saturated pool's requests to the next pool

2026-09-30 23:23:32 +02:00

scheduler.go

Merge C24 KV-index routing from vLLM's cache events

2026-10-05 01:24:25 +02:00

scorers.go

Chat goes to the replica that has been answering soonest

2026-10-25 17:01:30 +01:00