Files
llmgw/internal/router/prefix/scorer.go
T
Ilker ManapandClaude Opus 5 a075ff4120
CI / test (push) Successful in 1m19s
Add approximate prefix routing, and find out it did nothing
Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which
replica has already prefilled one. Bytes, not tokens — needing the engine's
exact tokenizer is what makes tier 2 a separate project.

It went in inert, and the decision metric is what caught it: 144 lookups, zero
hits. The cause was a rule I had written and tested for. A node's label is the
first request that created it, so the second request for the same system
prompt diverges in its last few bytes — a new question after the same
preamble, the single most common shape there is. Requiring the whole label to
match threw away the 700 bytes they shared. What matters is the shared head,
because that is what the engine cached, and a partial label match now counts.
Two unit tests asserted the wrong rule and have been inverted.

With that fixed, measured end to end against engines whose caches are too
small for the working set: 22.5% cache hits without, 81.7% with, and the
breakdown shows the misses are exactly the first round, when the tree is
empty.

The scenario matters as much as the number. With a handful of prompts and a
large cache, scattering is free — every replica ends up holding everything —
and prefix routing can only add constraint. The first version of the test
measured that case and showed a loss, which was a true statement about a
situation the feature is not for.

Two things keep it from being the disaster prefix routing usually is. The
imbalance guard, which needs both an absolute and a relative gap before it
fires, because either alone misfires — absolute on an idle cluster, relative
on a busy one. And it is a scorer rather than a filter: affinity is a vote the
queue and inflight scorers can outvote, not a rule.

The tree records the replica the picker actually chose, not the one the scorer
proposed, and a node prefers the replica it has seen most rather than most
recently — a count is stable under a randomizing picker, where recency just
chases whatever the picker last did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:13:19 +02:00

127 lines
3.8 KiB
Go

package prefix
import (
"math"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
)
// Decision is which branch a prefix lookup took.
//
// Recorded as a metric because tuning prefix routing without it is guesswork:
// a cache hit rate says nothing about how often affinity was overridden, and
// "affinity is not helping" and "affinity never applied" look identical from
// the outside.
type Decision string
// The branches.
const (
Hit Decision = "cache_hit"
MissMinLoad Decision = "cache_miss_min_load"
LoadOverride Decision = "load_balance_override"
NoTree Decision = "no_tree"
NoCandidates Decision = "no_healthy_worker"
)
// Thresholds decide when the prefix is worth following.
type Thresholds struct {
// MatchRatio is the share of the prompt that must match. Below it, the
// overlap is not enough to save a meaningful prefill.
MatchRatio float64
// BalanceAbs and BalanceRel are the imbalance guard, and both must trip
// before it does. Without this the cache hit rate rises while p99 latency
// collapses: affinity keeps sending a hot prefix to one replica long after
// that replica has stopped being able to serve it quickly.
//
// Absolute alone would fire on an idle cluster where one replica has two
// requests and another none. Relative alone would fire on a busy one where
// 200 against 180 is a rounding error. Together they mean "meaningfully
// more loaded, and by enough to matter".
BalanceAbs float64
BalanceRel float64
}
// DefaultThresholds returns values that work without measurement.
func DefaultThresholds() Thresholds {
return Thresholds{MatchRatio: 0.3, BalanceAbs: 8, BalanceRel: 1.5}
}
func (t *Thresholds) normalize() {
d := DefaultThresholds()
if t.MatchRatio <= 0 {
t.MatchRatio = d.MatchRatio
}
if t.BalanceAbs <= 0 {
t.BalanceAbs = d.BalanceAbs
}
if t.BalanceRel <= 1 {
t.BalanceRel = d.BalanceRel
}
}
// Choose picks the replica a prompt should go to, and says why.
//
// It returns an index into eps, or -1 when the caller should decide on load
// alone — which is the common case and not a failure.
func Choose(t *Tree, prompt string, eps []endpoint.Snapshot, th Thresholds) (int, Decision) {
th.normalize()
if len(eps) == 0 {
return -1, NoCandidates
}
if t == nil || prompt == "" {
return -1, NoTree
}
// The guard runs before the lookup, not after. Checking afterwards would
// mean paying for a match only to discard it, and — worse — would invite
// the shape where a cache hit quietly wins because somebody moved the
// check.
if imbalanced(eps, th) {
return -1, LoadOverride
}
replica, matched := t.Match(prompt)
if replica == "" || float64(matched)/float64(len(prompt)) < th.MatchRatio {
return -1, MissMinLoad
}
for i, e := range eps {
if e.ID == replica {
return i, Hit
}
}
// The match names a replica that is no longer a candidate: gone, draining,
// or its breaker is open. That is a miss, not an error.
return -1, MissMinLoad
}
// imbalanced reports whether load is lopsided enough to ignore affinity.
func imbalanced(eps []endpoint.Snapshot, th Thresholds) bool {
if len(eps) < 2 {
return false
}
maxL, minL := math.Inf(-1), math.Inf(1)
for _, e := range eps {
l := load(e)
maxL = math.Max(maxL, l)
minL = math.Min(minL, l)
}
if maxL-minL <= th.BalanceAbs {
return false
}
// A ratio against zero is infinite, and an idle replica is exactly where
// the next request should go, so treat it as tripping the guard.
if minL <= 0 {
return true
}
return maxL/minL > th.BalanceRel
}
// load is what the guard compares.
//
// Queue depth plus what the router has dispatched but not seen finish: the
// metric alone lags by a scrape interval, which is long enough for affinity to
// pile a burst onto one replica before anything notices.
func load(e endpoint.Snapshot) float64 {
return float64(e.Stats.Waiting + e.Stats.Running + e.Inflight)
}