Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM publishing KV cache events to a loopback port it listens on (pure-Go zmq4, msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the existing stream, with a full copy on every new one and a reset on gaps, restarts and clears. The control plane hands them to the router, for the sending node's own replicas only. The router tokenizes through a replica's /tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib golden values), and scores replicas by the share of leading blocks held, falling back to the approximate tree. New metrics beside llmgw_prefix_decisions_total. Untested against a running vLLM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
d41104a452
commit
117f417ef5
43 files changed
+3496
-263
No files matched your search
@@ -0,0 +1,42 @@
|
||||
# C24 — precise KV-index routing (plan §7.5 tier 2)
|
||||
|
||||
## CHANGELOG
|
||||
|
||||
### Routing by what each vLLM replica's cache holds
|
||||
|
||||
- A vLLM profile can set `kv_events: true`. The agent then starts the engine
|
||||
publishing its KV cache events (`--kv-events-config`) to a port on
|
||||
127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo
|
||||
sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be
|
||||
computed outside Python. The events never leave the node.
|
||||
- The agent mirrors each engine's set of cached blocks and sends the net
|
||||
changes up its existing stream to the control plane four times a second
|
||||
(new `KVEvents` agent message), starting every new stream with a full
|
||||
copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or
|
||||
`AllBlocksCleared` start a replica's index again; blocks offloaded to CPU,
|
||||
remote ones and a hybrid model's second cache group are not indexed.
|
||||
- The control plane passes them to the router's index, only for replicas
|
||||
the sending node runs, and empties what a stream described when it closes.
|
||||
- For a model with `kv_events`, the router asks one of its replicas to
|
||||
tokenize the request (vLLM's `/tokenize`, with messages and tools, so the
|
||||
chat template is applied exactly), hashes the tokens into blocks as vLLM
|
||||
does, and scores replicas by the share of the prompt's leading blocks they
|
||||
hold. The imbalance guard still applies. With no answer — no events yet,
|
||||
an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls
|
||||
back to the approximate tree (`router.prefix`) or to load.
|
||||
- The agent checks first blocks against llmgw's hashing; an engine that
|
||||
hashes another way (a profile overriding the algorithm, a changed vLLM) is
|
||||
reported in the agent's log and `llmgw_kv_index_problem`, and routed as
|
||||
though it published nothing.
|
||||
- New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4)
|
||||
and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total`
|
||||
(`kv_hit`, `kv_miss_min_load`, `load_balance_override`,
|
||||
`fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`),
|
||||
`llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`.
|
||||
- Experimental: tested against vLLM's event format and hashing as its source
|
||||
defines them, and against libzmq and msgspec publishing — not yet against a
|
||||
running vLLM.
|
||||
|
||||
## BACKLOG
|
||||
|
||||
- [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.
|
||||
Reference in new issue
Block a user