Ilker ManapandClaude Opus 5.5
117f417ef5
Plan §7.5 tier 2, opt-in per profile with kv_events. The agent starts vLLM publishing KV cache events to a loopback port it listens on (pure-Go zmq4, msgpack in both of vLLM's encodings), with sha256_cbor hashing and a fixed PYTHONHASHSEED; it mirrors each engine's blocks and sends net changes up the existing stream, with a full copy on every new one and a reset on gaps, restarts and clears. The control plane hands them to the router, for the sending node's own replicas only. The router tokenizes through a replica's /tokenize, hashes the tokens as vLLM does (checked against cbor2/hashlib golden values), and scores replicas by the share of leading blocks held, falling back to the approximate tree. New metrics beside llmgw_prefix_decisions_total. Untested against a running vLLM. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
97 lines
4.1 KiB
AMPL
97 lines
4.1 KiB
AMPL
module gitea.mynodes.xyz/ilker/llmgw
|
|
|
|
go 1.26.6
|
|
|
|
require (
|
|
connectrpc.com/connect v1.21.0
|
|
filippo.io/age v1.3.2
|
|
github.com/NVIDIA/go-nvml v0.13.4-0
|
|
github.com/cespare/xxhash/v2 v2.3.0
|
|
github.com/charmbracelet/bubbletea v1.3.10
|
|
github.com/charmbracelet/lipgloss v1.1.0
|
|
github.com/charmbracelet/x/ansi v0.11.6
|
|
github.com/coreos/go-oidc/v3 v3.21.0
|
|
github.com/go-jose/go-jose/v4 v4.1.4
|
|
github.com/go-zeromq/zmq4 v0.17.0
|
|
github.com/jackc/pgx/v5 v5.11.0
|
|
github.com/knadh/koanf/parsers/yaml v1.1.1
|
|
github.com/knadh/koanf/providers/env/v2 v2.0.1
|
|
github.com/knadh/koanf/providers/file v1.2.1
|
|
github.com/knadh/koanf/v2 v2.3.6
|
|
github.com/mattn/go-isatty v0.0.24
|
|
github.com/prometheus/client_golang v1.24.1
|
|
github.com/prometheus/client_model v0.6.3
|
|
github.com/prometheus/common v0.71.0
|
|
github.com/spf13/cobra v1.10.2
|
|
github.com/vmihailenco/msgpack/v5 v5.4.1
|
|
github.com/yuin/goldmark v1.8.6
|
|
go.opentelemetry.io/otel v1.46.0
|
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.46.0
|
|
go.opentelemetry.io/otel/sdk v1.46.0
|
|
go.opentelemetry.io/otel/trace v1.46.0
|
|
go.yaml.in/yaml/v3 v3.0.5
|
|
golang.org/x/crypto v0.57.0
|
|
golang.org/x/net v0.59.0
|
|
golang.org/x/oauth2 v0.37.0
|
|
golang.org/x/sync v0.23.0
|
|
golang.org/x/text v0.42.0
|
|
google.golang.org/protobuf v1.36.12
|
|
modernc.org/sqlite v1.59.0
|
|
)
|
|
|
|
require (
|
|
filippo.io/edwards25519 v1.2.0 // indirect
|
|
filippo.io/hpke v0.4.0 // indirect
|
|
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
|
github.com/beorn7/perks v1.0.1 // indirect
|
|
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
|
github.com/charmbracelet/colorprofile v0.4.1 // indirect
|
|
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
|
|
github.com/charmbracelet/x/term v0.2.2 // indirect
|
|
github.com/clipperhouse/displaywidth v0.9.0 // indirect
|
|
github.com/clipperhouse/stringish v0.1.1 // indirect
|
|
github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
|
|
github.com/dustin/go-humanize v1.0.1 // indirect
|
|
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
|
|
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
|
github.com/go-logr/logr v1.4.4 // indirect
|
|
github.com/go-logr/stdr v1.2.2 // indirect
|
|
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
|
|
github.com/go-zeromq/goczmq/v4 v4.2.2 // indirect
|
|
github.com/google/uuid v1.6.0 // indirect
|
|
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
|
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
|
github.com/jackc/pgpassfile v1.0.0 // indirect
|
|
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
|
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
|
github.com/knadh/koanf/maps v0.1.2 // indirect
|
|
github.com/kylelemons/godebug v1.1.0 // indirect
|
|
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
|
|
github.com/mattn/go-localereader v0.0.1 // indirect
|
|
github.com/mattn/go-runewidth v0.0.19 // indirect
|
|
github.com/mitchellh/copystructure v1.2.0 // indirect
|
|
github.com/mitchellh/reflectwalk v1.0.2 // indirect
|
|
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
|
|
github.com/muesli/cancelreader v0.2.2 // indirect
|
|
github.com/muesli/termenv v0.16.0 // indirect
|
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
|
github.com/ncruces/go-strftime v1.0.0 // indirect
|
|
github.com/prometheus/procfs v0.21.1 // indirect
|
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
|
github.com/rivo/uniseg v0.4.7 // indirect
|
|
github.com/spf13/pflag v1.0.9 // indirect
|
|
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
|
|
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
|
|
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
|
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.46.0 // indirect
|
|
go.opentelemetry.io/otel/metric v1.46.0 // indirect
|
|
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
|
golang.org/x/sys v0.48.0 // indirect
|
|
google.golang.org/genproto/googleapis/api v0.0.0-20260819154853-08b0e4226688 // indirect
|
|
google.golang.org/genproto/googleapis/rpc v0.0.0-20260819154853-08b0e4226688 // indirect
|
|
google.golang.org/grpc v1.83.1 // indirect
|
|
modernc.org/libc v1.75.7 // indirect
|
|
modernc.org/mathutil v1.7.1 // indirect
|
|
modernc.org/memory v1.12.1 // indirect
|
|
)
|