Files
llmgw/notes/c24-kvindex.md
T
Ilker ManapandClaude Opus 5.5 117f417ef5 Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:11 +02:00

3.3 KiB

C24 — precise KV-index routing (plan §7.5 tier 2)

CHANGELOG

Routing by what each vLLM replica's cache holds

  • A vLLM profile can set kv_events: true. The agent then starts the engine publishing its KV cache events (--kv-events-config) to a port on 127.0.0.1 the agent listens on, with --prefix-caching-hash-algo sha256_cbor and PYTHONHASHSEED=1729, so its block hashes can be computed outside Python. The events never leave the node.
  • The agent mirrors each engine's set of cached blocks and sends the net changes up its existing stream to the control plane four times a second (new KVEvents agent message), starting every new stream with a full copy. Lost events (a gap in vLLM's sequence numbers), an engine restart or AllBlocksCleared start a replica's index again; blocks offloaded to CPU, remote ones and a hybrid model's second cache group are not indexed.
  • The control plane passes them to the router's index, only for replicas the sending node runs, and empties what a stream described when it closes.
  • For a model with kv_events, the router asks one of its replicas to tokenize the request (vLLM's /tokenize, with messages and tools, so the chat template is applied exactly), hashes the tokens into blocks as vLLM does, and scores replicas by the share of the prompt's leading blocks they hold. The imbalance guard still applies. With no answer — no events yet, an image, a cache salt, a LoRA adapter, a failed or slow tokenize — it falls back to the approximate tree (router.prefix) or to load.
  • The agent checks first blocks against llmgw's hashing; an engine that hashes another way (a profile overriding the algorithm, a changed vLLM) is reported in the agent's log and llmgw_kv_index_problem, and routed as though it published nothing.
  • New settings router.kv_index.tokenize_timeout (250ms), min_blocks (4) and max_blocks (4096). New metrics llmgw_kv_index_decisions_total (kv_hit, kv_miss_min_load, load_balance_override, fallback_no_index, fallback_unsupported, fallback_tokenize_failed), llmgw_kv_index_tokenize_seconds, llmgw_kv_index_blocks.
  • Experimental: tested against vLLM's event format and hashing as its source defines them, and against libzmq and msgspec publishing — not yet against a running vLLM.

BACKLOG

  • [~] 24. Precise KV index (tier 2) — built: vLLM KV events via the agent (loopback ZMQ, pure-Go zmq4, msgpack in both of vLLM's encodings) forwarded as net block-hash changes over the agent stream; router index, tokenize via the replica's /tokenize, sha256_cbor block-hash chain checked against cbor2/hashlib golden values; scorer by leading-block share with tier 1 fallback; opt-in kv_events per profile; metrics; docs. Left: validation against a real vLLM (event shape, hash agreement, /tokenize matching the served prompt, cost of tokenizing); vLLM's replay endpoint instead of resetting on a gap; data-parallel ranks (one publisher each, ports offset); hybrid models' cache groups and CPU-offloaded blocks; LoRA/multimodal/cache_salt requests (extra keys); a router-only deployment (no agents) subscribing directly; standby control planes have no index until they lead; a per-replica fallback when only some candidates have an index; caching tokenizations of repeated prompts.