# C24 — precise KV-index routing (plan §7.5 tier 2) ## CHANGELOG ### Routing by what each vLLM replica's cache holds - A vLLM profile can set `kv_events: true`. The agent then starts the engine publishing its KV cache events (`--kv-events-config`) to a port on 127.0.0.1 the agent listens on, with `--prefix-caching-hash-algo sha256_cbor` and `PYTHONHASHSEED=1729`, so its block hashes can be computed outside Python. The events never leave the node. - The agent mirrors each engine's set of cached blocks and sends the net changes up its existing stream to the control plane four times a second (new `KVEvents` agent message), starting every new stream with a full copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or `AllBlocksCleared` start a replica's index again; blocks offloaded to CPU, remote ones and a hybrid model's second cache group are not indexed. - The control plane passes them to the router's index, only for replicas the sending node runs, and empties what a stream described when it closes. - For a model with `kv_events`, the router asks one of its replicas to tokenize the request (vLLM's `/tokenize`, with messages and tools, so the chat template is applied exactly), hashes the tokens into blocks as vLLM does, and scores replicas by the share of the prompt's leading blocks they hold. The imbalance guard still applies. With no answer — no events yet, an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls back to the approximate tree (`router.prefix`) or to load. - The agent checks first blocks against llmgw's hashing; an engine that hashes another way (a profile overriding the algorithm, a changed vLLM) is reported in the agent's log and `llmgw_kv_index_problem`, and routed as though it published nothing. - New settings `router.kv_index.tokenize_timeout` (250ms), `min_blocks` (4) and `max_blocks` (4096). New metrics `llmgw_kv_index_decisions_total` (`kv_hit`, `kv_miss_min_load`, `load_balance_override`, `fallback_no_index`, `fallback_unsupported`, `fallback_tokenize_failed`), `llmgw_kv_index_tokenize_seconds`, `llmgw_kv_index_blocks`. - Experimental: tested against vLLM's event format and hashing as its source defines them, and against libzmq and msgspec publishing — not yet against a running vLLM. ## BACKLOG - [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's `/tokenize`, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in `kv_events` per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, `/tokenize` matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.