Merge C24 KV-index routing from vLLM's cache events

Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:24:25 +02:00
commit 9e46ee83e4
43 files changed
+3498 -265

No files matched your search

+59
View File
@@ -102,6 +102,9 @@ rather than repeated.
| `output` | `string` | [JudgePolicy](#judgepolicy) |
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
| `min_blocks` | `int` | [KVIndex](#kvindex) |
| `max_blocks` | `int` | [KVIndex](#kvindex) |
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `language` | `string` | [Landing](#landing) |
@@ -212,6 +215,7 @@ rather than repeated.
| `weight` | `float64` | [Profile](#profile) |
| `priority` | `int` | [Profile](#profile) |
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
| `kv_events` | `bool` | [Profile](#profile) |
| `activation_wait` | `time.Duration` | [Router](#router) |
| `scrape_interval` | `time.Duration` | [Router](#router) |
| `scrape_timeout` | `time.Duration` | [Router](#router) |
@@ -221,6 +225,7 @@ rather than repeated.
| `failure_threshold` | `int` | [Router](#router) |
| `default_profile` | `string` | [Router](#router) |
| `prefix` | `Prefix` | [Router](#router) |
| `kv_index` | `KVIndex` | [Router](#router) |
| `session_header` | `string` | [Router](#router) |
| `inject_usage` | `bool` | [Router](#router) |
| `tenant_label` | `string` | [Router](#router) |
@@ -977,6 +982,33 @@ codes (S1, S10) or the names the model reports.
LatencyBudget is how long the judge is waited for before the policy's
fail applies. Zero is 250ms.
## KVIndex
### `tokenize_timeout`
`time.Duration`
TokenizeTimeout bounds the call that turns a request into token ids,
made to one of the model's replicas before the request is placed. Past
it the request is routed as tier 1 would. Zero takes 250ms.
### `min_blocks`
`int`
MinBlocks is the fewest leading blocks a replica must hold to be
preferred for them; fewer is coincidence — a chat template's opening
tokens — rather than a prefix worth following. Zero takes 4, which is
64 tokens at vLLM's usual block size of 16.
### `max_blocks`
`int`
MaxBlocks bounds how many of a prompt's blocks are hashed and looked
up. Beyond it a longer match still helps, but the difference no longer
decides anything. Zero takes 4096.
## Landing
### `contact`
@@ -1811,6 +1843,23 @@ Priority orders profiles during placement; highest that fits wins.
StartupTimeout bounds model loading. A 70B model across two nodes takes
tens of minutes, so this is per profile rather than global.
### `kv_events`
`bool`
KVEvents has vLLM publish what its prefix cache holds, and the router
route this model's requests by it: to the replica holding the most of
a prompt's leading blocks (plan §7.5 tier 2). vLLM only.
Off by default, because it costs something on every request: the
router asks a replica to tokenize the prompt before placing it, a
round trip and the chat template run twice. It pays where prompts
share long prefixes that the approximate tree guesses wrong — many
replicas, conversations that move between them, evictions. The engine
is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED,
which the router's hashing assumes; arguments overriding either leave
the index empty, and the agent says so in its log.
## Router
### `activation_wait`
@@ -1876,6 +1925,16 @@ the win for multi-turn traffic.
Takes a [Prefix](#prefix) block.
### `kv_index`
`KVIndex`
KVIndex tunes the precise KV index (plan §7.5 tier 2). Which models
use it is the catalog's to say, by kv_events on a vLLM profile; this
is only how.
Takes a [KVIndex](#kvindex) block.
### `session_header`
`string`