Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM publishing KV cache events to a loopback port it listens on (pure-Go zmq4, msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the existing stream, with a full copy on every new one and a reset on gaps, restarts and clears. The control plane hands them to the router, for the sending node's own replicas only. The router tokenizes through a replica's /tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib golden values), and scores replicas by the share of leading blocks held, falling back to the approximate tree. New metrics beside llmgw_prefix_decisions_total. Untested against a running vLLM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
34 lines
1.5 KiB
Go
34 lines
1.5 KiB
Go
package kvindex
|
|
|
|
// Decision is which branch a precise-index lookup took, counted in
|
|
// llmgw_kv_index_decisions_total beside tier 1's llmgw_prefix_decisions_total.
|
|
//
|
|
// The fallbacks are counted by why. "The index is not helping" has three
|
|
// causes that want different fixes — no events are arriving, the requests are
|
|
// of a kind it cannot index, the engine is too slow to tokenize — and one
|
|
// count would not say which.
|
|
type Decision string
|
|
|
|
// The branches.
|
|
const (
|
|
// Hit: some replica holds enough of the prompt's leading blocks, and
|
|
// replicas are scored by how many.
|
|
Hit Decision = "kv_hit"
|
|
// MissMinLoad: the index knows the replicas and none holds enough; load
|
|
// decides, and tier 1 is not asked — its guess cannot beat a fact.
|
|
MissMinLoad Decision = "kv_miss_min_load"
|
|
// LoadOverride: load is lopsided enough that affinity is ignored, as in
|
|
// tier 1.
|
|
LoadOverride Decision = "load_balance_override"
|
|
// FallbackNoIndex: no candidate's events have arrived (or their hashing
|
|
// was found wrong); tier 1 decides.
|
|
FallbackNoIndex Decision = "fallback_no_index"
|
|
// FallbackUnsupported: the request cannot be indexed — an image, a cache
|
|
// salt, an adapter, a list of prompts, an endpoint other than chat or
|
|
// completions; tier 1 decides.
|
|
FallbackUnsupported Decision = "fallback_unsupported"
|
|
// FallbackTokenizeFailed: the engine did not tokenize the request in
|
|
// time; tier 1 decides.
|
|
FallbackTokenizeFailed Decision = "fallback_tokenize_failed"
|
|
)
|