Merge C24 KV-index routing from vLLM's cache events
Conflicts with the LoRA work: vLLM's arguments, the engine profile, router metrics and the scheduler's request all keep both; agent.pb.go regenerated from the merged proto; the two test fakes of vLLM given separate names. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
commit
9e46ee83e4
43 files changed
+3498
-265
No files matched your search
@@ -5618,6 +5618,10 @@
|
||||
"type": "integer",
|
||||
"format": "int64",
|
||||
"description": "StartupTimeout bounds model loading. A 70B model across two nodes takes tens of minutes, so this is per profile rather than global.\n\nA duration, in nanoseconds."
|
||||
},
|
||||
"kv_events": {
|
||||
"type": "boolean",
|
||||
"description": "KVEvents has vLLM publish what its prefix cache holds, and the router route this model's requests by it: to the replica holding the most of a prompt's leading blocks (plan §7.5 tier 2). vLLM only.\n\nOff by default, because it costs something on every request: the router asks a replica to tokenize the prompt before placing it, a round trip and the chat template run twice. It pays where prompts share long prefixes that the approximate tree guesses wrong — many replicas, conversations that move between them, evictions. The engine is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED, which the router's hashing assumes; arguments overriding either leave the index empty, and the agent says so in its log."
|
||||
}
|
||||
}
|
||||
},
|
||||
|
||||
@@ -1103,6 +1103,65 @@ joined a load, loaded it), `llmgw_adapter_loads_total` and
|
||||
adapter at start: llama-server can change the scale of an adapter it started
|
||||
with, not load a new one.
|
||||
|
||||
### Routing by what each replica's cache actually holds
|
||||
|
||||
With several replicas of a vLLM model, a request is fastest on the replica
|
||||
that already holds its prompt's prefix in KV cache. The router guesses which
|
||||
that is from what it has sent where (`router.prefix`). It can also be told:
|
||||
vLLM publishes an event every time it stores or evicts a cache block, and with
|
||||
`kv_events` on a profile llmgw routes that model by those events.
|
||||
|
||||
```yaml
|
||||
- name: llama-3.1-8b-instruct
|
||||
source: meta-llama/Llama-3.1-8B-Instruct
|
||||
routing_profile: chat # chat or long-context; batch ignores prefixes
|
||||
profiles:
|
||||
- name: h100
|
||||
gpu_class: h100-80g
|
||||
kv_events: true
|
||||
```
|
||||
|
||||
How it works, so you know what you are paying for:
|
||||
|
||||
- The node agent starts vLLM with `--kv-events-config` publishing to a port
|
||||
on 127.0.0.1 that the agent listens on, `--prefix-caching-hash-algo
|
||||
sha256_cbor`, and `PYTHONHASHSEED=1729`. The events never leave the node:
|
||||
the agent keeps the set of blocks each engine holds and sends the changes,
|
||||
four times a second, up the stream it already has to the control plane,
|
||||
which hands them to the router. A new stream — a control plane restart, a
|
||||
failover — starts with a full copy. Nothing new is opened in the firewall.
|
||||
- For each request to such a model, the router asks one of its replicas to
|
||||
tokenize the prompt (vLLM's `/tokenize`, with the request's messages and
|
||||
tools), hashes the token ids into blocks the way vLLM does, and prefers the
|
||||
replica holding the most leading blocks. Tokenizing at the engine is what
|
||||
makes the hashes agree: the chat template is part of the tokens, and only
|
||||
the engine runs it exactly. It costs a round trip before the request is
|
||||
placed and the template run twice; `router.kv_index.tokenize_timeout`
|
||||
(250ms) bounds it.
|
||||
- When the index has nothing to say — no events have arrived yet, the
|
||||
request has an image or a `cache_salt` or names a LoRA adapter, the
|
||||
tokenize call failed — the router routes as it would without it: by the
|
||||
approximate tree if `router.prefix.enabled`, else by load. The imbalance
|
||||
guard of `router.prefix` applies to both.
|
||||
|
||||
Turning `kv_events` on changes how the engine is started, so replicas already
|
||||
running keep running without it (and show as stale) until they are replaced.
|
||||
Leave `--prefix-caching-hash-algo` out of the profile's `args`: an engine
|
||||
hashing another way publishes blocks the router can never match. The agent
|
||||
checks the first block of prompts against its own hashing, and if they never
|
||||
agree it says so in its log and on `llmgw_kv_index_problem`, and the replica
|
||||
is routed as though it published nothing.
|
||||
|
||||
Watch `llmgw_kv_index_decisions_total` beside `llmgw_prefix_decisions_total`:
|
||||
`kv_hit` is the index deciding, `kv_miss_min_load` is it knowing no replica
|
||||
holds enough (`router.kv_index.min_blocks`, 4 blocks), and the `fallback_*`
|
||||
values say why it could not answer. `llmgw_kv_index_tokenize_seconds` is what
|
||||
the tokenize call costs, and `llmgw_kv_index_blocks` what each replica holds.
|
||||
|
||||
This has been built and tested against vLLM's event format and its hashing as
|
||||
its source defines them, with libzmq and msgspec publishing — not yet against
|
||||
a running vLLM. Treat it as experimental until it has been.
|
||||
|
||||
### Models served outside the cluster
|
||||
|
||||
A model can be an OpenAI-compatible API elsewhere — a hosted provider, or
|
||||
|
||||
@@ -102,6 +102,9 @@ rather than repeated.
|
||||
| `output` | `string` | [JudgePolicy](#judgepolicy) |
|
||||
| `categories` | `[]string` | [JudgePolicy](#judgepolicy) |
|
||||
| `latency_budget` | `time.Duration` | [JudgePolicy](#judgepolicy) |
|
||||
| `tokenize_timeout` | `time.Duration` | [KVIndex](#kvindex) |
|
||||
| `min_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `max_blocks` | `int` | [KVIndex](#kvindex) |
|
||||
| `contact` | `string` | [Landing](#landing) |
|
||||
| `notice` | `string` | [Landing](#landing) |
|
||||
| `language` | `string` | [Landing](#landing) |
|
||||
@@ -212,6 +215,7 @@ rather than repeated.
|
||||
| `weight` | `float64` | [Profile](#profile) |
|
||||
| `priority` | `int` | [Profile](#profile) |
|
||||
| `startup_timeout` | `time.Duration` | [Profile](#profile) |
|
||||
| `kv_events` | `bool` | [Profile](#profile) |
|
||||
| `activation_wait` | `time.Duration` | [Router](#router) |
|
||||
| `scrape_interval` | `time.Duration` | [Router](#router) |
|
||||
| `scrape_timeout` | `time.Duration` | [Router](#router) |
|
||||
@@ -221,6 +225,7 @@ rather than repeated.
|
||||
| `failure_threshold` | `int` | [Router](#router) |
|
||||
| `default_profile` | `string` | [Router](#router) |
|
||||
| `prefix` | `Prefix` | [Router](#router) |
|
||||
| `kv_index` | `KVIndex` | [Router](#router) |
|
||||
| `session_header` | `string` | [Router](#router) |
|
||||
| `inject_usage` | `bool` | [Router](#router) |
|
||||
| `tenant_label` | `string` | [Router](#router) |
|
||||
@@ -977,6 +982,33 @@ codes (S1, S10) or the names the model reports.
|
||||
LatencyBudget is how long the judge is waited for before the policy's
|
||||
fail applies. Zero is 250ms.
|
||||
|
||||
## KVIndex
|
||||
|
||||
### `tokenize_timeout`
|
||||
|
||||
`time.Duration`
|
||||
|
||||
TokenizeTimeout bounds the call that turns a request into token ids,
|
||||
made to one of the model's replicas before the request is placed. Past
|
||||
it the request is routed as tier 1 would. Zero takes 250ms.
|
||||
|
||||
### `min_blocks`
|
||||
|
||||
`int`
|
||||
|
||||
MinBlocks is the fewest leading blocks a replica must hold to be
|
||||
preferred for them; fewer is coincidence — a chat template's opening
|
||||
tokens — rather than a prefix worth following. Zero takes 4, which is
|
||||
64 tokens at vLLM's usual block size of 16.
|
||||
|
||||
### `max_blocks`
|
||||
|
||||
`int`
|
||||
|
||||
MaxBlocks bounds how many of a prompt's blocks are hashed and looked
|
||||
up. Beyond it a longer match still helps, but the difference no longer
|
||||
decides anything. Zero takes 4096.
|
||||
|
||||
## Landing
|
||||
|
||||
### `contact`
|
||||
@@ -1811,6 +1843,23 @@ Priority orders profiles during placement; highest that fits wins.
|
||||
StartupTimeout bounds model loading. A 70B model across two nodes takes
|
||||
tens of minutes, so this is per profile rather than global.
|
||||
|
||||
### `kv_events`
|
||||
|
||||
`bool`
|
||||
|
||||
KVEvents has vLLM publish what its prefix cache holds, and the router
|
||||
route this model's requests by it: to the replica holding the most of
|
||||
a prompt's leading blocks (plan §7.5 tier 2). vLLM only.
|
||||
|
||||
Off by default, because it costs something on every request: the
|
||||
router asks a replica to tokenize the prompt before placing it, a
|
||||
round trip and the chat template run twice. It pays where prompts
|
||||
share long prefixes that the approximate tree guesses wrong — many
|
||||
replicas, conversations that move between them, evictions. The engine
|
||||
is started with sha256_cbor block hashing and a fixed PYTHONHASHSEED,
|
||||
which the router's hashing assumes; arguments overriding either leave
|
||||
the index empty, and the agent says so in its log.
|
||||
|
||||
## Router
|
||||
|
||||
### `activation_wait`
|
||||
@@ -1876,6 +1925,16 @@ the win for multi-turn traffic.
|
||||
|
||||
Takes a [Prefix](#prefix) block.
|
||||
|
||||
### `kv_index`
|
||||
|
||||
`KVIndex`
|
||||
|
||||
KVIndex tunes the precise KV index (plan §7.5 tier 2). Which models
|
||||
use it is the catalog's to say, by kv_events on a vLLM profile; this
|
||||
is only how.
|
||||
|
||||
Takes a [KVIndex](#kvindex) block.
|
||||
|
||||
### `session_header`
|
||||
|
||||
`string`
|
||||
|
||||
Reference in new issue
Block a user