Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM publishing KV cache events to a loopback port it listens on (pure-Go zmq4, msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the existing stream, with a full copy on every new one and a reset on gaps, restarts and clears. The control plane hands them to the router, for the sending node's own replicas only. The router tokenizes through a replica's /tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib golden values), and scores replicas by the share of leading blocks held, falling back to the approximate tree. New metrics beside llmgw_prefix_decisions_total. Untested against a running vLLM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
117f417ef5
43 files changed
+3496
-263
No files matched your search
@@ -93,6 +93,9 @@ rather than repeated.
|
||||
| `output` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
|
||||
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
|
||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `max_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `docs_sign_in` | `bool` | [Landing](#landing) |
|
||||
@@ -197,6 +200,7 @@ rather than repeated.
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
| `priority` | `int` | [Profile](#profile) |
|
||||
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
||||
| `kv_events` | `bool` | [Profile](#profile) |
|
||||
| `activation_wait` | `time.Duration` | [Router](#router) |
|
||||
| `scrape_interval` | `time.Duration` | [Router](#router) |
|
||||
| `scrape_timeout` | `time.Duration` | [Router](#router) |
|
||||
@@ -206,6 +210,7 @@ rather than repeated.
|
||||
| `failure_threshold` | `int` | [Router](#router) |
|
||||
| `default_profile` | `string` | [Router](#router) |
|
||||
| `prefix` | `Prefix` | [Router](#router) |
|
||||
| `kv_index` | `KVIndex` | [Router](#router) |
|
||||
| `session_header` | `string` | [Router](#router) |
|
||||
| `inject_usage` | `bool` | [Router](#router) |
|
||||
| `tenant_label` | `string` | [Router](#router) |
|
||||
@@ -889,6 +894,33 @@ codes (S1, S10) or the names the model reports.
|
||||
LatencyBudget is how long the judge is waited for before the policy's
|
||||
fail applies. Zero is 250ms.
|
||||
|
||||
## KVIndex
|
||||
|
||||
### `tokenize_timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
TokenizeTimeout bounds the call that turns a request into token ids,
|
||||
made to one of the model's replicas before the request is placed. Past
|
||||
it the request is routed as tier 1 would. Zero takes 250ms.
|
||||
|
||||
### `min_blocks`
|
||||
|
||||
`int`
|
||||
|
||||
MinBlocks is the fewest leading blocks a replica must hold to be
|
||||
preferred for them; fewer is coincidence — a chat template's opening
|
||||
tokens — rather than a prefix worth following. Zero takes 4, which is
|
||||
64 tokens at vLLM's usual block size of 16.
|
||||
|
||||
### `max_blocks`
|
||||
|
||||
`int`
|
||||
|
||||
MaxBlocks bounds how many of a prompt's blocks are hashed and looked
|
||||
up. Beyond it a longer match still helps, but the difference no longer
|
||||
decides anything. Zero takes 4096.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -1658,6 +1690,23 @@ Priority orders profiles during placement; highest that fits wins.
|
||||
StartupTimeout bounds model loading. A 70B model across two nodes takes
|
||||
tens of minutes, so this is per profile rather than global.
|
||||
|
||||
### `kv_events`
|
||||
|
||||
`bool`
|
||||
|
||||
KVEvents has vLLM publish what its prefix cache holds, and the router
|
||||
route this model's requests by it: to the replica holding the most of
|
||||
a prompt's leading blocks (plan §7.5 tier 2). vLLM only.
|
||||
|
||||
Off by default, because it costs something on every request: the
|
||||
router asks a replica to tokenize the prompt before placing it, a
|
||||
round trip and the chat template run twice. It pays where prompts
|
||||
share long prefixes that the approximate tree guesses wrong — many
|
||||
replicas, conversations that move between them, evictions. The engine
|
||||
is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED,
|
||||
which the router's hashing assumes; arguments overriding either leave
|
||||
the index empty, and the agent says so in its log.
|
||||
|
||||
## Router
|
||||
|
||||
### `activation_wait`
|
||||
@@ -1723,6 +1772,16 @@ the win for multi-turn traffic.
|
||||
|
||||
Takes a [Prefix](#prefix) block.
|
||||
|
||||
### `kv_index`
|
||||
|
||||
`KVIndex`
|
||||
|
||||
KVIndex tunes the precise KV index (plan §7.5 tier 2). Which models
|
||||
use it is the catalog's to say, by kv_events on a vLLM profile; this
|
||||
is only how.
|
||||
|
||||
Takes a [KVIndex](#kvindex) block.
|
||||
|
||||
### `session_header`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user