Files
llmgw/internal/router/kvindex/decision.go
T
Ilker ManapandClaude Opus 5.5 117f417ef5 Precise KV-index routing: vLLM KV cache events through the agent, routed by leading blocks held
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM
publishing KV cache events to a loopback port it listens on (pure-Go zmq4,
msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed
PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the
existing stream, with a full copy on every new one and a reset on gaps,
restarts and clears. The control plane hands them to the router, for the
sending node's own replicas only. The router tokenizes through a replica's
/tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib
golden values), and scores replicas by the share of leading blocks held,
falling back to the approximate tree. New metrics beside
llmgw_prefix_decisions_total. Untested against a running vLLM.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-05 01:21:11 +02:00

34 lines
1.5 KiB
Go

package kvindex
// Decision is which branch a precise-index lookup took, counted in
// llmgw_kv_index_decisions_total beside tier 1's llmgw_prefix_decisions_total.
//
// The fallbacks are counted by why. "The index is not helping" has three
// causes that want different fixes — no events are arriving, the requests are
// of a kind it cannot index, the engine is too slow to tokenize — and one
// count would not say which.
type Decision string
// The branches.
const (
// Hit: some replica holds enough of the prompt's leading blocks, and
// replicas are scored by how many.
Hit Decision = "kv_hit"
// MissMinLoad: the index knows the replicas and none holds enough; load
// decides, and tier 1 is not asked — its guess cannot beat a fact.
MissMinLoad Decision = "kv_miss_min_load"
// LoadOverride: load is lopsided enough that affinity is ignored, as in
// tier 1.
LoadOverride Decision = "load_balance_override"
// FallbackNoIndex: no candidate's events have arrived (or their hashing
// was found wrong); tier 1 decides.
FallbackNoIndex Decision = "fallback_no_index"
// FallbackUnsupported: the request cannot be indexed — an image, a cache
// salt, an adapter, a list of prompts, an endpoint other than chat or
// completions; tier 1 decides.
FallbackUnsupported Decision = "fallback_unsupported"
// FallbackTokenizeFailed: the engine did not tokenize the request in
// time; tier 1 decides.
FallbackTokenizeFailed Decision = "fallback_tokenize_failed"
)