ilkerandClaude Opus 5
b7417e1a69
The code behind plan §7.3-7.6: - breaker: folds cooldown and circuit breaking into one component (FailureThreshold=1 gives LiteLLM-style cooldown), with exponential backoff and half-open probes. - endpoint: live state registry. Inflight counting, staleness detection, drain rate derived from counters. A scrape failure does not trip the breaker — the engine may be serving fine while only its metrics endpoint is down. - scrape: parses the §7.6 metric contract. Errors when any of the mandatory three is missing; the metric name mapping is overridable, since names drift between versions (plan §14). - scheduler: the filter -> score -> pick pipeline. A "strategy" is now just a set of scorer weights; chat, batch and long-context are the same code with different weights. Scorers: queue, kv (hard penalty above the ceiling), inflight, load, weight, session (bounded-load consistent hashing). Pickers: max-score, p2c, softmax, round-robin. Decision carries the rationale — which scorer produced what, which filter dropped how many. Tuning the weights without that is guesswork. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18 lines
474 B
AMPL
18 lines
474 B
AMPL
module gitea.mynodes.xyz/ilker/llmgw
|
|
|
|
go 1.26.3
|
|
|
|
require (
|
|
github.com/prometheus/common v0.71.0
|
|
github.com/spf13/cobra v1.10.2
|
|
)
|
|
|
|
require (
|
|
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
|
github.com/prometheus/client_model v0.6.3 // indirect
|
|
github.com/spf13/pflag v1.0.9 // indirect
|
|
google.golang.org/protobuf v1.36.12 // indirect
|
|
)
|