Merge C24 KV-index routing from vLLM's cache events

Conflicts with the LoRA work: vLLM's arguments, the engine profile, router
metrics and the scheduler's request all keep both; agent.pb.go regenerated
from the merged proto; the two test fakes of vLLM given separate names.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5.5 committed 2026-10-05 01:24:25 +02:00
commit 9e46ee83e4
43 files changed
+3498 -265

No files matched your search

+4
View File
@@ -5618,6 +5618,10 @@
"type": "integer",
"format": "int64",
"description": "StartupTimeout bounds model loading. A 70B model across two nodes takes tens of minutes, so this is per profile rather than global.\n\nA duration, in nanoseconds."
},
"kv_events": {
"type": "boolean",
"description": "KVEvents has vLLM publish what its prefix cache holds, and the router route this model's requests by it: to the replica holding the most of a prompt's leading blocks (plan §7.5 tier 2). vLLM only.\n\nOff by default, because it costs something on every request: the router asks a replica to tokenize the prompt before placing it, a round trip and the chat template run twice. It pays where prompts share long prefixes that the approximate tree guesses wrong — many replicas, conversations that move between them, evictions. The engine is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED, which the router's hashing assumes; arguments overriding either leave the index empty, and the agent says so in its log."
}
}
},
+59
View File
@@ -1103,6 +1103,65 @@ joined a load, loaded it), `llmgw_adapter_loads_total` and
adapter at start: llama-server can change the scale of an adapter it started
with, not load a new one.
### Routing by what each replica's cache actually holds
With several replicas of a vLLM model, a request is fastest on the replica
that already holds its prompt's prefix in KV cache. The router guesses which
that is from what it has sent where (`router.prefix`). It can also be told:
vLLM publishes an event every time it stores or evicts a cache block, and with
`kv_events` on a profile llmgw routes that model by those events.
```yaml
- name: llama-3.1-8b-instruct
source: meta-llama/Llama-3.1-8B-Instruct
routing_profile: chat # chat or long-context; batch ignores prefixes
profiles:
- name: h100
gpu_class: h100-80g
kv_events: true
```
How it works, so you know what you are paying for:
- The node agent starts vLLM with `--kv-events-config` publishing to a port
on 127.0.0.1 that the agent listens on, `--prefix-caching-hash-algo
sha256_cbor`, and `PYTHONHASHSEED=1729`. The events never leave the node:
the agent keeps the set of blocks each engine holds and sends the changes,
four times a second, up the stream it already has to the control plane,
which hands them to the router. A new stream — a control plane restart, a
failover — starts with a full copy. Nothing new is opened in the firewall.
- For each request to such a model, the router asks one of its replicas to
tokenize the prompt (vLLM's `/tokenize`, with the request's messages and
tools), hashes the token ids into blocks the way vLLM does, and prefers the
replica holding the most leading blocks. Tokenizing at the engine is what
makes the hashes agree: the chat template is part of the tokens, and only
the engine runs it exactly. It costs a round trip before the request is
placed and the template run twice; `router.kv_index.tokenize_timeout`
(250ms) bounds it.
- When the index has nothing to say — no events have arrived yet, the
request has an image or a `cache_salt` or names a LoRA adapter, the
tokenize call failed — the router routes as it would without it: by the
approximate tree if `router.prefix.enabled`, else by load. The imbalance
guard of `router.prefix` applies to both.
Turning `kv_events` on changes how the engine is started, so replicas already
running keep running without it (and show as stale) until they are replaced.
Leave `--prefix-caching-hash-algo` out of the profile's `args`: an engine
hashing another way publishes blocks the router can never match. The agent
checks the first block of prompts against its own hashing, and if they never
agree it says so in its log and on `llmgw_kv_index_problem`, and the replica
is routed as though it published nothing.
Watch `llmgw_kv_index_decisions_total` beside `llmgw_prefix_decisions_total`:
`kv_hit` is the index deciding, `kv_miss_min_load` is it knowing no replica
holds enough (`router.kv_index.min_blocks`, 4 blocks), and the `fallback_*`
values say why it could not answer. `llmgw_kv_index_tokenize_seconds` is what
the tokenize call costs, and `llmgw_kv_index_blocks` what each replica holds.
This has been built and tested against vLLM's event format and its hashing as
its source defines them, with libzmq and msgspec publishing — not yet against
a running vLLM. Treat it as experimental until it has been.
### Models served outside the cluster
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
+59
View File
@@ -102,6 +102,9 @@ rather than repeated.
| `output` | `string` | [JudgePolicy](#judgepolicy) |
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
| `min_blocks` | `int` | [KVIndex](#kvindex) |
| `max_blocks` | `int` | [KVIndex](#kvindex) |
| `contact` | `string` | [Landing](#landing) |
| `notice` | `string` | [Landing](#landing) |
| `language` | `string` | [Landing](#landing) |
@@ -212,6 +215,7 @@ rather than repeated.
| `weight` | `float64` | [Profile](#profile) |
| `priority` | `int` | [Profile](#profile) |
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
| `kv_events` | `bool` | [Profile](#profile) |
| `activation_wait` | `time.Duration` | [Router](#router) |
| `scrape_interval` | `time.Duration` | [Router](#router) |
| `scrape_timeout` | `time.Duration` | [Router](#router) |
@@ -221,6 +225,7 @@ rather than repeated.
| `failure_threshold` | `int` | [Router](#router) |
| `default_profile` | `string` | [Router](#router) |
| `prefix` | `Prefix` | [Router](#router) |
| `kv_index` | `KVIndex` | [Router](#router) |
| `session_header` | `string` | [Router](#router) |
| `inject_usage` | `bool` | [Router](#router) |
| `tenant_label` | `string` | [Router](#router) |
@@ -977,6 +982,33 @@ codes (S1, S10) or the names the model reports.
LatencyBudget is how long the judge is waited for before the policy's
fail applies. Zero is 250ms.
## KVIndex
### `tokenize_timeout`
`time.Duration`
TokenizeTimeout bounds the call that turns a request into token ids,
made to one of the model's replicas before the request is placed. Past
it the request is routed as tier 1 would. Zero takes 250ms.
### `min_blocks`
`int`
MinBlocks is the fewest leading blocks a replica must hold to be
preferred for them; fewer is coincidence — a chat template's opening
tokens — rather than a prefix worth following. Zero takes 4, which is
64 tokens at vLLM's usual block size of 16.
### `max_blocks`
`int`
MaxBlocks bounds how many of a prompt's blocks are hashed and looked
up. Beyond it a longer match still helps, but the difference no longer
decides anything. Zero takes 4096.
## Landing
### `contact`
@@ -1811,6 +1843,23 @@ Priority orders profiles during placement; highest that fits wins.
StartupTimeout bounds model loading. A 70B model across two nodes takes
tens of minutes, so this is per profile rather than global.
### `kv_events`
`bool`
KVEvents has vLLM publish what its prefix cache holds, and the router
route this model's requests by it: to the replica holding the most of
a prompt's leading blocks (plan §7.5 tier 2). vLLM only.
Off by default, because it costs something on every request: the
router asks a replica to tokenize the prompt before placing it, a
round trip and the chat template run twice. It pays where prompts
share long prefixes that the approximate tree guesses wrong — many
replicas, conversations that move between them, evictions. The engine
is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED,
which the router's hashing assumes; arguments overriding either leave
the index empty, and the agent says so in its log.
## Router
### `activation_wait`
@@ -1876,6 +1925,16 @@ the win for multi-turn traffic.
Takes a [Prefix](#prefix) block.
### `kv_index`
`KVIndex`
KVIndex tunes the precise KV index (plan §7.5 tier 2). Which models
use it is the catalog's to say, by kv_events on a vLLM profile; this
is only how.
Takes a [KVIndex](#kvindex) block.
### `session_header`
`string`