Add approximate prefix routing, and find out it did nothing
CI / test (push) Successful in 1m19s

Tier 1 of plan §7.5: a radix tree of request prefixes that estimates which
replica has already prefilled one. Bytes, not tokens — needing the engine's
exact tokenizer is what makes tier 2 a separate project.

It went in inert, and the decision metric is what caught it: 144 lookups, zero
hits. The cause was a rule I had written and tested for. A node's label is the
first request that created it, so the second request for the same system
prompt diverges in its last few bytes — a new question after the same
preamble, the single most common shape there is. Requiring the whole label to
match threw away the 700 bytes they shared. What matters is the shared head,
because that is what the engine cached, and a partial label match now counts.
Two unit tests asserted the wrong rule and have been inverted.

With that fixed, measured end to end against engines whose caches are too
small for the working set: 22.5% cache hits without, 81.7% with, and the
breakdown shows the misses are exactly the first round, when the tree is
empty.

The scenario matters as much as the number. With a handful of prompts and a
large cache, scattering is free — every replica ends up holding everything —
and prefix routing can only add constraint. The first version of the test
measured that case and showed a loss, which was a true statement about a
situation the feature is not for.

Two things keep it from being the disaster prefix routing usually is. The
imbalance guard, which needs both an absolute and a relative gap before it
fires, because either alone misfires — absolute on an idle cluster, relative
on a busy one. And it is a scorer rather than a filter: affinity is a vote the
queue and inflight scorers can outvote, not a rule.

The tree records the replica the picker actually chose, not the one the scorer
proposed, and a node prefers the replica it has seen most rather than most
recently — a count is stable under a randomizing picker, where recency just
chases whatever the picker last did.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-16 12:13:19 +02:00
1 parent de21649ad6
commit a075ff4120
13 files changed
+1207 -7

No files matched your search

+65
View File
@@ -0,0 +1,65 @@
package scheduler
import (
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
"gitea.mynodes.xyz/ilker/llmgw/internal/router/prefix"
)
// PrefixScorer steers a request towards the replica that has probably already
// prefilled its prefix (plan §7.5 tier 1).
//
// A scorer rather than a filter, deliberately. A filter would make affinity a
// rule, and a rule that sends every request carrying a popular system prompt to
// one replica is how prefix routing produces a beautiful cache hit rate and an
// unusable p99. As a scorer it is one voice among several, and the queue and
// inflight scorers can outvote it — on top of the imbalance guard inside the
// lookup itself, which stops it speaking at all when load has gone lopsided.
type PrefixScorer struct {
tree *prefix.Tree
thresholds prefix.Thresholds
// observe reports which branch the decision took, for the metric without
// which tuning this is guesswork.
observe func(prefix.Decision)
}
// NewPrefixScorer builds the scorer. A nil tree makes it inert, which is what
// a model with prefix routing turned off should get.
func NewPrefixScorer(t *prefix.Tree, th prefix.Thresholds, observe func(prefix.Decision)) *PrefixScorer {
if observe == nil {
observe = func(prefix.Decision) {}
}
return &PrefixScorer{tree: t, thresholds: th, observe: observe}
}
// Name identifies the scorer in weights and metrics.
func (*PrefixScorer) Name() string { return "prefix" }
// Score gives the matched replica 1 and everything else 0.
func (p *PrefixScorer) Score(c *Ctx, eps []endpoint.Snapshot) []float64 {
out := make([]float64, len(eps))
idx, decision := prefix.Choose(p.tree, c.Req.PromptPrefix, eps, p.thresholds)
p.observe(decision)
if idx >= 0 {
out[idx] = 1
}
return out
}
// Observe records where a request was actually sent.
//
// After the decision rather than during it: the scorer proposes, the picker
// disposes, and recording the proposal would fill the tree with replicas that
// were never sent anything — which is the one thing that would make the
// estimate worse than no estimate at all.
func (p *PrefixScorer) Observe(promptPrefix, replicaID string) {
if p.tree != nil {
p.tree.Observe(promptPrefix, replicaID)
}
}
// Forget drops a replica that has gone away.
func (p *PrefixScorer) Forget(replicaID string) {
if p.tree != nil {
p.tree.Forget(replicaID)
}
}
+24 -4
View File
@@ -10,6 +10,11 @@ type ProfileConfig struct {
KVCeiling float64
// SessionLoadFactor is the load ceiling multiplier for session affinity.
SessionLoadFactor float64
// Prefix, when set, adds the approximate prefix tree to the profiles that
// benefit from it. Nil leaves them exactly as they were — session affinity
// alone, which is tier 0 and already most of the win.
Prefix *PrefixScorer
}
// DefaultProfileConfig returns sensible starting values.
@@ -46,17 +51,30 @@ func Chat(c ProfileConfig) *Profile {
return &Profile{
Name: "chat",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Scorers: []WeightedScorer{
Scorers: c.withPrefix([]WeightedScorer{
{Session(c.SessionLoadFactor), 3.0},
{Queue(c.TargetWaiting), 1.0},
{KV(c.KVCeiling), 1.0},
{Inflight(c.TargetInflight), 1.0},
{Weight(), 0.5},
},
}, 2.0),
Picker: P2C(),
}
}
// withPrefix appends the prefix scorer when one is configured.
//
// Weighted below session affinity on purpose. A session id is a statement from
// the client that these requests belong together; a prefix match is the
// router's guess that they might. When the two disagree the client is more
// likely to be right.
func (c ProfileConfig) withPrefix(scorers []WeightedScorer, weight float64) []WeightedScorer {
if c.Prefix == nil {
return scorers
}
return append(scorers, WeightedScorer{c.Prefix, weight})
}
// Batch is the profile for bulk work: throughput matters, TTFT does not, so
// finding the shortest queue outranks stickiness.
func Batch(c ProfileConfig) *Profile {
@@ -82,12 +100,14 @@ func LongContext(c ProfileConfig) *Profile {
return &Profile{
Name: "long-context",
Filters: []Filter{BreakerOpen(), PoolPriority()},
Scorers: []WeightedScorer{
// Prefix matching earns more here than anywhere else: a long prompt is
// mostly prefill, and prefill is what a cache hit skips.
Scorers: c.withPrefix([]WeightedScorer{
{KV(c.KVCeiling), 3.0},
{Session(c.SessionLoadFactor), 2.0},
{Queue(c.TargetWaiting), 1.0},
{Inflight(c.TargetInflight), 1.0},
},
}, 3.0),
Picker: Softmax(0.2),
}
}