Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc38dd6096 | ||
|
|
a3f71aa5ee | ||
|
|
f0c1ca87f1 |
No files matched your search
+111
-3
@@ -6,9 +6,9 @@ is only released once its **exit criterion** is met, not when its code lands.
|
|||||||
| Version | Phase | Scope | State |
|
| Version | Phase | Scope | State |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||||||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | in progress |
|
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||||||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | planned |
|
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | in progress |
|
||||||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
||||||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
||||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||||
@@ -16,6 +16,114 @@ is only released once its **exit criterion** is met, not when its code lands.
|
|||||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||||
APIs change between minors without a deprecation period.
|
APIs change between minors without a deprecation period.
|
||||||
|
|
||||||
|
## v0.3.0 — 2026-09-16
|
||||||
|
|
||||||
|
Flow control. A batch workload no longer starves interactive traffic.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Admission control** (`internal/router/admission`), inserted between quota
|
||||||
|
checks and scheduling. It answers *when* a request may proceed; the scheduler
|
||||||
|
still answers *where* it goes. Keeping them apart is the point: an engine
|
||||||
|
queue is FIFO and knows nothing about tenants, so once a batch job is in it,
|
||||||
|
no amount of endpoint selection rescues the request behind it.
|
||||||
|
- **Saturation detector.** One number per pool, the worse of queue depth and KV
|
||||||
|
occupancy. Either signal alone misleads — a pool can have an empty queue and a
|
||||||
|
nearly full cache, or the reverse.
|
||||||
|
- **Priority bands.** Each band has a saturation ceiling; above it, that band
|
||||||
|
waits. The cluster then degrades from the bottom up. A ceiling of 1.0 means
|
||||||
|
"flow up to the healthy limit"; above 1 means never wait.
|
||||||
|
- **Tenant fairness.** Within a band, dispatch turns rotate between tenants, so
|
||||||
|
one caller sending a thousand requests occupies one slot in the rotation
|
||||||
|
rather than a thousand.
|
||||||
|
- **Bounded queues.** Capacity per band (immediate 503 when full) and a TTL
|
||||||
|
(504), because sending the GPU work the client has already abandoned helps
|
||||||
|
nobody. Cancelled requests leave the queue instead of holding capacity.
|
||||||
|
- **Metrics:** `llmgw_admission_wait_seconds`, `llmgw_admission_rejected_total`,
|
||||||
|
`llmgw_admission_queue_depth`, `llmgw_pool_saturation`. Responses carry
|
||||||
|
`x-llmgw-queue-ms` when a request waited.
|
||||||
|
|
||||||
|
### Phase 1.5 exit criterion, measured in `test/e2e`
|
||||||
|
|
||||||
|
While 500 concurrent bronze requests run continuously, interactive p99 in the
|
||||||
|
platinum band stays at **1.04–1.24x** its idle value, against a 1.5x budget.
|
||||||
|
With admission switched off the same load pushes it to **55–60x**. Both
|
||||||
|
directions are asserted, so the test fails if admission regresses *and* if the
|
||||||
|
load generator ever stops being a real load.
|
||||||
|
|
||||||
|
### Two lags, two different fixes
|
||||||
|
|
||||||
|
Getting this right took longer than the queueing itself:
|
||||||
|
|
||||||
|
- Engine metrics are scraped, so they lag by an interval. The router's inflight
|
||||||
|
count does not — but it includes *running* requests, and treating those as
|
||||||
|
queue depth makes a healthy engine look overloaded every interval. What is
|
||||||
|
genuinely queued is `inflight − batch capacity`, and batch capacity is not in
|
||||||
|
the metric contract, so it is learned as the high-water mark of observed
|
||||||
|
`running`.
|
||||||
|
- Between admission granting a request and the router dispatching it, the
|
||||||
|
request is invisible to every signal. A simultaneous burst therefore passed
|
||||||
|
the check en masse — exactly the batch arrival pattern this layer exists to
|
||||||
|
survive. The controller now counts its own outstanding leases against
|
||||||
|
dispatched inflight.
|
||||||
|
|
||||||
|
### Known limitations
|
||||||
|
|
||||||
|
- The band comes from a trusted header until authentication arrives in v1.0.0.
|
||||||
|
Anyone who can reach the port can promote themselves to platinum.
|
||||||
|
- The tenant key is the session id, for the same reason.
|
||||||
|
- Displacement (dropping queued low-band work to make room) is not implemented;
|
||||||
|
the plan defers it, and bounded queues plus TTL cover most of the need.
|
||||||
|
|
||||||
|
## v0.2.0 — 2026-09-16
|
||||||
|
|
||||||
|
The router serves traffic. `llmgw server` is real: it fronts a static set of
|
||||||
|
engine replicas with an OpenAI-compatible API.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Streaming proxy.** Relays SSE event by event with a flush after each, and
|
||||||
|
propagates cancellation: when a client disconnects, the upstream request is
|
||||||
|
aborted so the GPU stops working. Separate deadlines for dial, response
|
||||||
|
headers, first token, inter-token idle and total duration — a wedged engine is
|
||||||
|
indistinguishable from a slow one without the idle watchdog.
|
||||||
|
- **Retry and fallback.** Another endpoint is tried only while nothing has
|
||||||
|
reached the client; once bytes are out, splicing two responses together would
|
||||||
|
be worse than the short stream. Models can name fallback models, and the
|
||||||
|
response says which one served it via `x-llmgw-served-model`.
|
||||||
|
- **OpenAI surface.** `/v1/chat/completions`, `/v1/completions`,
|
||||||
|
`/v1/embeddings` and `/v1/models`, with aliases resolved from the catalog.
|
||||||
|
Only routing-relevant fields are parsed; the rest of the body passes through
|
||||||
|
untouched.
|
||||||
|
- **Usage accounting that does not depend on the client.** The router can ask
|
||||||
|
engines for usage the client never requested, then drop that chunk on the way
|
||||||
|
back, so token counts are recorded either way.
|
||||||
|
- **Configuration** (`internal/config`): YAML through koanf, overlaid on
|
||||||
|
defaults, with `LLMGW_`-prefixed environment overrides. Validation rejects
|
||||||
|
duplicate endpoint ids and aliases that would shadow another model — both
|
||||||
|
would otherwise misroute silently at request time.
|
||||||
|
- **Metrics** (plan §8): request counts, duration, TTFT, tokens, plus per-scorer
|
||||||
|
values and per-filter drop counts. Routing profile weights cannot be tuned
|
||||||
|
without seeing what the scorers actually produced. Served on a separate admin
|
||||||
|
listener alongside health and pprof.
|
||||||
|
|
||||||
|
### Phase 1 exit criterion, measured in `test/e2e`
|
||||||
|
|
||||||
|
- 1000 concurrent streams behind 3 sim-engines: all completed, ~6800 req/s.
|
||||||
|
- Every client disconnecting frees the KV blocks on every engine.
|
||||||
|
- Router overhead p99 ≈ 0.9 ms against the plan's 2 ms budget. CI asserts a
|
||||||
|
looser bound by default since a shared machine cannot hold 2 ms reliably; set
|
||||||
|
`LLMGW_E2E_STRICT=1` on quiet hardware to enforce the real target.
|
||||||
|
|
||||||
|
### Known limitations
|
||||||
|
|
||||||
|
- No admission control, so a batch workload can still starve interactive
|
||||||
|
traffic. That is v0.3.0, and it is the most valuable thing still missing.
|
||||||
|
- The endpoint list is static; the control plane arrives in v0.5.0.
|
||||||
|
- No authentication, rate limiting or budgets yet (v1.0.0).
|
||||||
|
- Prompt length is estimated as bytes/4. It becomes exact only with a real
|
||||||
|
tokenizer, which the plan defers to v2 along with tier 2 prefix routing.
|
||||||
|
|
||||||
## v0.1.0 — 2026-09-16
|
## v0.1.0 — 2026-09-16
|
||||||
|
|
||||||
First tagged version. It is useful for exactly one thing: developing and testing
|
First tagged version. It is useful for exactly one thing: developing and testing
|
||||||
|
|||||||
@@ -13,15 +13,17 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
|||||||
| Version | Phase | Scope | State |
|
| Version | Phase | Scope | State |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
|
||||||
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | in progress |
|
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
|
||||||
| v0.3.0 | 1.5 | Admission control (priority bands) | planned |
|
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
||||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | planned |
|
||||||
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
|
| v0.5.0 | 3 | Control plane: catalog, scheduler, reconciler | planned |
|
||||||
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
||||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||||
|
|
||||||
**v0.1.0 does not serve traffic**: `llmgw server` is still a stub. What works
|
As of v0.3.0 the router serves traffic and protects it: a batch workload no
|
||||||
today is `sim-engine` plus the routing libraries it exists to test.
|
longer starves interactive requests. Still missing before v1.0.0: authentication,
|
||||||
|
rate limits and budgets, and the control plane that replaces the static endpoint
|
||||||
|
list.
|
||||||
|
|
||||||
## Building
|
## Building
|
||||||
|
|
||||||
@@ -31,6 +33,37 @@ make test # unit tests
|
|||||||
make race # with the race detector
|
make race # with the race detector
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Running the router
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# Two fake engines and a router in front of them.
|
||||||
|
./bin/llmgw sim-engine --listen :8001 --model llama &
|
||||||
|
./bin/llmgw sim-engine --listen :8002 --model llama &
|
||||||
|
./bin/llmgw server \
|
||||||
|
--endpoint e1=http://127.0.0.1:8001=llama \
|
||||||
|
--endpoint e2=http://127.0.0.1:8002=llama
|
||||||
|
|
||||||
|
curl -N localhost:8080/v1/chat/completions \
|
||||||
|
-H 'content-type: application/json' \
|
||||||
|
-H 'x-session-id: demo' \
|
||||||
|
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
|
||||||
|
```
|
||||||
|
|
||||||
|
For anything real use a config file; [deploy/examples/server.yaml](deploy/examples/server.yaml)
|
||||||
|
is commented throughout. Metrics, health and pprof are on the admin port (`:9090`
|
||||||
|
by default), deliberately separate from the client-facing one.
|
||||||
|
|
||||||
|
Requests carry a priority band (`x-llmgw-band`). Under load, low bands wait at
|
||||||
|
the gateway while high bands keep flowing, so the cluster degrades from the
|
||||||
|
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
|
||||||
|
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
|
||||||
|
admission off.
|
||||||
|
|
||||||
|
Routing profiles select how endpoints are scored: `chat` favours session
|
||||||
|
affinity, `batch` favours the shortest queue, `long-context` favours free KV
|
||||||
|
cache. Pick one per model in the config, or per request with the
|
||||||
|
`x-llmgw-profile` header.
|
||||||
|
|
||||||
## sim-engine
|
## sim-engine
|
||||||
|
|
||||||
Simulates the inference-engine behaviours **the router cares about**, so the
|
Simulates the inference-engine behaviours **the router cares about**, so the
|
||||||
|
|||||||
@@ -0,0 +1,105 @@
|
|||||||
|
# llmgw router configuration (phase 1).
|
||||||
|
#
|
||||||
|
# The endpoint list is static here. From phase 3 it comes from the control
|
||||||
|
# plane instead and this section disappears; everything else stays.
|
||||||
|
#
|
||||||
|
# Any field can be overridden from the environment with an LLMGW_ prefix and
|
||||||
|
# __ for nesting: LLMGW_ROUTER__RETRIES=3
|
||||||
|
|
||||||
|
listen: ":8080" # client-facing OpenAI API
|
||||||
|
admin_listen: ":9090" # metrics, health, pprof — do not expose this publicly
|
||||||
|
|
||||||
|
router:
|
||||||
|
scrape_interval: 2s
|
||||||
|
scrape_timeout: 1s
|
||||||
|
# How long an endpoint may go unscraped before it is scored down. It is never
|
||||||
|
# dropped for staleness alone: if the whole pool went stale there would be
|
||||||
|
# nowhere left to send anything.
|
||||||
|
stale_after: 10s
|
||||||
|
|
||||||
|
# Additional endpoints to try after a failure, and only while nothing has
|
||||||
|
# reached the client yet.
|
||||||
|
retries: 2
|
||||||
|
# How long a failing endpoint leaves the candidate pool. A long fallback list
|
||||||
|
# with no cooldown turns one endpoint's rate limit into a cluster-wide retry
|
||||||
|
# storm.
|
||||||
|
cooldown: 15s
|
||||||
|
# Consecutive failures that open the breaker. Set to 1 for pure cooldown
|
||||||
|
# behaviour.
|
||||||
|
failure_threshold: 3
|
||||||
|
|
||||||
|
default_profile: chat # chat | batch | long-context | debug
|
||||||
|
session_header: x-session-id # affinity key; falls back to the `user` field
|
||||||
|
# Ask engines for usage even when the client did not, so token accounting
|
||||||
|
# works regardless of client behaviour. The extra chunk is dropped on the way
|
||||||
|
# back to a client that did not ask for it.
|
||||||
|
inject_usage: true
|
||||||
|
max_body_bytes: 33554432
|
||||||
|
|
||||||
|
timeouts:
|
||||||
|
dial: 5s
|
||||||
|
response_header: 60s
|
||||||
|
first_token: 120s
|
||||||
|
idle: 60s # between streamed tokens; catches a wedged engine
|
||||||
|
total: 30m
|
||||||
|
|
||||||
|
# Flow control: decides *when* a request may proceed, before scheduling
|
||||||
|
# decides where it goes. Without it a batch workload starves interactive
|
||||||
|
# traffic, because an engine queue is FIFO and knows nothing about tenants.
|
||||||
|
admission:
|
||||||
|
enabled: true
|
||||||
|
# Until authentication lands, the band comes from a header and is trusted.
|
||||||
|
# Do not expose this port to callers who should not be able to promote
|
||||||
|
# themselves.
|
||||||
|
band_header: x-llmgw-band
|
||||||
|
default_band: gold
|
||||||
|
# A band waits once pool saturation passes its ceiling, so the cluster
|
||||||
|
# degrades from the bottom up. A ceiling of 1.0 means "flow up to the
|
||||||
|
# healthy limit"; use a value above 1 for a band that must never wait.
|
||||||
|
bands:
|
||||||
|
- { name: platinum, priority: 400, saturation_ceiling: 1.00 }
|
||||||
|
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||||||
|
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||||||
|
- { name: bronze, priority: 100, saturation_ceiling: 0.30 }
|
||||||
|
# Drop a request that has waited this long: the client has usually given up,
|
||||||
|
# and sending the GPU work nobody is waiting for helps no one.
|
||||||
|
queue_ttl: 30s
|
||||||
|
# Reject immediately once a band's queue is this deep, rather than letting
|
||||||
|
# it accumulate silently.
|
||||||
|
queue_capacity_per_band: 2000
|
||||||
|
interval: 10ms
|
||||||
|
|
||||||
|
# What the scorers normalize against. Measure these with `ctl bench` rather
|
||||||
|
# than guessing: target_waiting in particular governs how the whole system
|
||||||
|
# behaves under load. Too low starves the GPU, too high brings back queueing
|
||||||
|
# delay.
|
||||||
|
scoring:
|
||||||
|
target_waiting: 4
|
||||||
|
target_inflight: 8
|
||||||
|
kv_ceiling: 0.9
|
||||||
|
session_load_factor: 1.25
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: llama-3.3-70b
|
||||||
|
aliases: [llama]
|
||||||
|
fallbacks: [qwen2.5-32b]
|
||||||
|
routing_profile: chat
|
||||||
|
|
||||||
|
- name: qwen2.5-32b
|
||||||
|
aliases: [qwen]
|
||||||
|
routing_profile: batch
|
||||||
|
|
||||||
|
endpoints:
|
||||||
|
- id: gpu-a-01
|
||||||
|
url: http://gpu-a-01:8000
|
||||||
|
models: [llama-3.3-70b]
|
||||||
|
pool: ib-a/h100-80g
|
||||||
|
weight: 100 # from ctl bench; do not estimate it by hand
|
||||||
|
priority: 100 # higher pools are tried first
|
||||||
|
|
||||||
|
- id: gpu-b-01
|
||||||
|
url: http://gpu-b-01:8000
|
||||||
|
models: [llama-3.3-70b, qwen2.5-32b]
|
||||||
|
pool: ib-b/a100-80g
|
||||||
|
weight: 45
|
||||||
|
priority: 50
|
||||||
@@ -3,15 +3,30 @@ module gitea.mynodes.xyz/ilker/llmgw
|
|||||||
go 1.26.3
|
go 1.26.3
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
github.com/cespare/xxhash/v2 v2.3.0
|
||||||
|
github.com/knadh/koanf/parsers/yaml v1.1.1
|
||||||
|
github.com/knadh/koanf/providers/env/v2 v2.0.1
|
||||||
|
github.com/knadh/koanf/providers/file v1.2.1
|
||||||
|
github.com/knadh/koanf/v2 v2.3.6
|
||||||
|
github.com/prometheus/client_golang v1.24.1
|
||||||
|
github.com/prometheus/client_model v0.6.3
|
||||||
github.com/prometheus/common v0.71.0
|
github.com/prometheus/common v0.71.0
|
||||||
github.com/spf13/cobra v1.10.2
|
github.com/spf13/cobra v1.10.2
|
||||||
)
|
)
|
||||||
|
|
||||||
require (
|
require (
|
||||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
github.com/beorn7/perks v1.0.1 // indirect
|
||||||
|
github.com/fsnotify/fsnotify v1.9.0 // indirect
|
||||||
|
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
|
||||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||||
|
github.com/knadh/koanf/maps v0.1.2 // indirect
|
||||||
|
github.com/mitchellh/copystructure v1.2.0 // indirect
|
||||||
|
github.com/mitchellh/reflectwalk v1.0.2 // indirect
|
||||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
|
||||||
github.com/prometheus/client_model v0.6.3 // indirect
|
github.com/prometheus/procfs v0.21.1 // indirect
|
||||||
github.com/spf13/pflag v1.0.9 // indirect
|
github.com/spf13/pflag v1.0.9 // indirect
|
||||||
|
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||||
|
golang.org/x/sync v0.23.0 // indirect
|
||||||
|
golang.org/x/sys v0.47.0 // indirect
|
||||||
google.golang.org/protobuf v1.36.12 // indirect
|
google.golang.org/protobuf v1.36.12 // indirect
|
||||||
)
|
)
|
||||||
@@ -1,18 +1,44 @@
|
|||||||
|
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
|
||||||
|
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
|
||||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||||
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||||
|
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
|
||||||
|
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
|
||||||
|
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
|
||||||
|
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
|
||||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||||
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||||
|
github.com/klauspost/compress v1.19.1 h1:VsB4HPswih7mmZ8WleSFQ75c/Ui1M4trX5oAsJnhSlk=
|
||||||
|
github.com/klauspost/compress v1.19.1/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||||
|
github.com/knadh/koanf/maps v0.1.2 h1:RBfmAW5CnZT+PJ1CVc1QSJKf4Xu9kxfQgYVQSu8hpbo=
|
||||||
|
github.com/knadh/koanf/maps v0.1.2/go.mod h1:npD/QZY3V6ghQDdcQzl1W4ICNVTkohC8E73eI2xW4yI=
|
||||||
|
github.com/knadh/koanf/parsers/yaml v1.1.1 h1:u70vV5IyaM0HvONh8HoqBC97oTgO33KcpZbTLiKVinU=
|
||||||
|
github.com/knadh/koanf/parsers/yaml v1.1.1/go.mod h1:HHmcHXUrp9cOPcuC+2wrr44GTUB0EC+PyfN3HZD9tFg=
|
||||||
|
github.com/knadh/koanf/providers/env/v2 v2.0.1 h1:a3KagndPqhcWHQv6Pz4OZmwkI/yMeTjkiZye6ZCkyW0=
|
||||||
|
github.com/knadh/koanf/providers/env/v2 v2.0.1/go.mod h1:1g01PE+Ve1gBfWNNw2wmULRP0tc8RJrjn5p2N/jNCIc=
|
||||||
|
github.com/knadh/koanf/providers/file v1.2.1 h1:bEWbtQwYrA+W2DtdBrQWyXqJaJSG3KrP3AESOJYp9wM=
|
||||||
|
github.com/knadh/koanf/providers/file v1.2.1/go.mod h1:bp1PM5f83Q+TOUu10J/0ApLBd9uIzg+n9UgthfY+nRA=
|
||||||
|
github.com/knadh/koanf/v2 v2.3.6 h1:JoQPSJmvS4aP0xNc8xMDr5tcrkSEInL23/Il7pITAKo=
|
||||||
|
github.com/knadh/koanf/v2 v2.3.6/go.mod h1:gRb40VRAbd4iJMYYD5IxZ6hfuopFcXBpc9bbQpZwo28=
|
||||||
|
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
|
||||||
|
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
|
||||||
|
github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
|
||||||
|
github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
|
||||||
|
github.com/mitchellh/reflectwalk v1.0.2 h1:G2LzWKi524PWgd3mLHV8Y5k7s6XUvT0Gef6zxSIeXaQ=
|
||||||
|
github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
|
||||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
|
||||||
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
|
||||||
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
|
github.com/prometheus/client_golang v1.24.1 h1:JnJkREXzWxUdCuPFpIWZiPispT9xVV59uiuyR2bPlnU=
|
||||||
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
|
github.com/prometheus/client_golang v1.24.1/go.mod h1:F+oSRECHg4sse5ucfYpYDeIv/hu68Zo0uoHKetWnzcE=
|
||||||
github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo=
|
github.com/prometheus/client_model v0.6.3 h1:O0jaTVAYNxTHYInEPFJt5I3+sN8zqBtVMPTB1qyxiEo=
|
||||||
github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM=
|
github.com/prometheus/client_model v0.6.3/go.mod h1:gpN5P9S7Rr6Yr92PiQ+Ixvhf6JZEkF1dnxsYL2aPBEM=
|
||||||
github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8=
|
github.com/prometheus/common v0.71.0 h1:9KDAKb7Mj3HEVKyFCK6Dc/HIwlBzZIN2l7/lrHl3KK8=
|
||||||
github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk=
|
github.com/prometheus/common v0.71.0/go.mod h1:CLJ5H8TEsGX8bl31BdMkfhIZ+QmZ9tBPPotUxUbfcmk=
|
||||||
|
github.com/prometheus/procfs v0.21.1 h1:GljZCt+zSTS+NZq88cyQ1LjZ+RCHp3uVuabBWA5+OJI=
|
||||||
|
github.com/prometheus/procfs v0.21.1/go.mod h1:aB55Cww9pdSJVHk0hUf0inxWyyjPogFIjmHKYgMKmtY=
|
||||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||||
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
|
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
|
||||||
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
|
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
|
||||||
@@ -20,11 +46,17 @@ github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
|
|||||||
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||||
|
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||||
|
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||||
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
go.yaml.in/yaml/v2 v2.4.4 h1:tuyd0P+2Ont/d6e2rl3be67goVK4R6deVxCUX5vyPaQ=
|
||||||
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
|
||||||
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
|
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
|
||||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||||
|
golang.org/x/sync v0.23.0 h1:KameEIfc1IkluZyXWLn39Wd4tURc6GbCiISGiZm2bQk=
|
||||||
|
golang.org/x/sync v0.23.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||||
|
golang.org/x/sys v0.47.0 h1:o7XGOvZQCADBQQ4Y7VNq2dRWQR7JmOUW8Kxx4ZsNgWs=
|
||||||
|
golang.org/x/sys v0.47.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
|
||||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"os/signal"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/spf13/cobra"
|
||||||
|
"golang.org/x/sync/errgroup"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newServerCmd() *cobra.Command {
|
||||||
|
var (
|
||||||
|
configPath string
|
||||||
|
listen string
|
||||||
|
adminListen string
|
||||||
|
endpoints []string
|
||||||
|
)
|
||||||
|
|
||||||
|
cmd := &cobra.Command{
|
||||||
|
Use: "server",
|
||||||
|
Short: "Run the router (phase 1; the control plane arrives in phase 3)",
|
||||||
|
Long: "Serves the OpenAI API in front of a set of engine replicas.\n\n" +
|
||||||
|
"In this phase the replica list is static: give it in the config file, or\n" +
|
||||||
|
"pass --endpoint id=URL=model[,model] once per replica for a quick test.",
|
||||||
|
RunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
|
log := logger()
|
||||||
|
|
||||||
|
cfg, err := config.Load(configPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if listen != "" {
|
||||||
|
cfg.Listen = listen
|
||||||
|
}
|
||||||
|
if adminListen != "" {
|
||||||
|
cfg.AdminListen = adminListen
|
||||||
|
}
|
||||||
|
for _, spec := range endpoints {
|
||||||
|
ep, err := parseEndpointFlag(spec)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
cfg.Endpoints = append(cfg.Endpoints, ep)
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(cfg.Endpoints) == 0 {
|
||||||
|
return errors.New("no endpoints configured; use --endpoint or a config file")
|
||||||
|
}
|
||||||
|
|
||||||
|
rt, err := router.New(cfg, log)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx, stop := signal.NotifyContext(cmd.Context(), syscall.SIGINT, syscall.SIGTERM)
|
||||||
|
defer stop()
|
||||||
|
|
||||||
|
rt.Start(ctx)
|
||||||
|
defer rt.Stop()
|
||||||
|
|
||||||
|
adminHandler, err := rt.AdminHandler()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
apiSrv := &http.Server{
|
||||||
|
Handler: rt.Handler(),
|
||||||
|
ReadHeaderTimeout: 10 * time.Second,
|
||||||
|
// No write timeout: responses stream for minutes and the proxy
|
||||||
|
// enforces its own per-phase deadlines.
|
||||||
|
}
|
||||||
|
adminSrv := &http.Server{
|
||||||
|
Handler: adminHandler,
|
||||||
|
ReadHeaderTimeout: 10 * time.Second,
|
||||||
|
}
|
||||||
|
|
||||||
|
apiLn, err := net.Listen("tcp", cfg.Listen)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("listening on %s: %w", cfg.Listen, err)
|
||||||
|
}
|
||||||
|
adminLn, err := net.Listen("tcp", cfg.AdminListen)
|
||||||
|
if err != nil {
|
||||||
|
_ = apiLn.Close()
|
||||||
|
return fmt.Errorf("listening on %s: %w", cfg.AdminListen, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Info("router listening",
|
||||||
|
"api", apiLn.Addr().String(),
|
||||||
|
"admin", adminLn.Addr().String(),
|
||||||
|
"endpoints", len(cfg.Endpoints),
|
||||||
|
"default_profile", cfg.Router.DefaultProfile,
|
||||||
|
)
|
||||||
|
|
||||||
|
g, gctx := errgroup.WithContext(ctx)
|
||||||
|
g.Go(func() error { return serve(apiSrv, apiLn) })
|
||||||
|
g.Go(func() error { return serve(adminSrv, adminLn) })
|
||||||
|
g.Go(func() error {
|
||||||
|
<-gctx.Done()
|
||||||
|
log.Info("shutting down")
|
||||||
|
sctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
_ = adminSrv.Shutdown(sctx)
|
||||||
|
return apiSrv.Shutdown(sctx)
|
||||||
|
})
|
||||||
|
return g.Wait()
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
f := cmd.Flags()
|
||||||
|
f.StringVar(&configPath, "config", "", "configuration file (optional)")
|
||||||
|
f.StringVar(&listen, "listen", "", "override the client-facing address")
|
||||||
|
f.StringVar(&adminListen, "admin-listen", "", "override the metrics/pprof address")
|
||||||
|
f.StringArrayVar(&endpoints, "endpoint", nil, "static endpoint: id=URL=model[,model] (repeatable)")
|
||||||
|
|
||||||
|
return cmd
|
||||||
|
}
|
||||||
|
|
||||||
|
func serve(s *http.Server, ln net.Listener) error {
|
||||||
|
if err := s.Serve(ln); err != nil && !errors.Is(err, http.ErrServerClosed) {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseEndpointFlag decodes id=URL=model[,model].
|
||||||
|
//
|
||||||
|
// The URL contains '=' only in pathological cases, so splitting into exactly
|
||||||
|
// three parts keeps the flag readable while staying unambiguous.
|
||||||
|
func parseEndpointFlag(spec string) (config.Endpoint, error) {
|
||||||
|
parts := strings.SplitN(spec, "=", 3)
|
||||||
|
if len(parts) != 3 {
|
||||||
|
return config.Endpoint{}, fmt.Errorf("--endpoint %q: expected id=URL=model[,model]", spec)
|
||||||
|
}
|
||||||
|
models := strings.Split(parts[2], ",")
|
||||||
|
for i := range models {
|
||||||
|
models[i] = strings.TrimSpace(models[i])
|
||||||
|
}
|
||||||
|
return config.Endpoint{
|
||||||
|
ID: strings.TrimSpace(parts[0]),
|
||||||
|
URL: strings.TrimRight(strings.TrimSpace(parts[1]), "/"),
|
||||||
|
Models: models,
|
||||||
|
Pool: "static",
|
||||||
|
Weight: 1,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
@@ -9,19 +9,6 @@ import (
|
|||||||
// errNotImplemented marks commands belonging to phases not yet built.
|
// errNotImplemented marks commands belonging to phases not yet built.
|
||||||
var errNotImplemented = errors.New("not implemented yet (see llmgw-plan.md §11)")
|
var errNotImplemented = errors.New("not implemented yet (see llmgw-plan.md §11)")
|
||||||
|
|
||||||
func newServerCmd() *cobra.Command {
|
|
||||||
var configPath string
|
|
||||||
cmd := &cobra.Command{
|
|
||||||
Use: "server",
|
|
||||||
Short: "Router and control plane (phases 1-3)",
|
|
||||||
RunE: func(_ *cobra.Command, _ []string) error {
|
|
||||||
return errNotImplemented
|
|
||||||
},
|
|
||||||
}
|
|
||||||
cmd.Flags().StringVar(&configPath, "config", "server.yaml", "configuration file")
|
|
||||||
return cmd
|
|
||||||
}
|
|
||||||
|
|
||||||
func newAgentCmd() *cobra.Command {
|
func newAgentCmd() *cobra.Command {
|
||||||
var configPath string
|
var configPath string
|
||||||
cmd := &cobra.Command{
|
cmd := &cobra.Command{
|
||||||
|
|||||||
@@ -0,0 +1,258 @@
|
|||||||
|
// Package config loads llmgw's YAML configuration.
|
||||||
|
//
|
||||||
|
// Phase 1 only needs the router's half of it: the endpoint list is static here
|
||||||
|
// and will come from the control plane in phase 3 (plan §6.4). The shape is
|
||||||
|
// already the one the plan describes, so that later change stays additive.
|
||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/knadh/koanf/parsers/yaml"
|
||||||
|
env "github.com/knadh/koanf/providers/env/v2"
|
||||||
|
"github.com/knadh/koanf/providers/file"
|
||||||
|
"github.com/knadh/koanf/v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Server is the whole server configuration.
|
||||||
|
type Server struct {
|
||||||
|
// Listen is the client-facing OpenAI API address.
|
||||||
|
Listen string `koanf:"listen"`
|
||||||
|
// AdminListen serves metrics, health and pprof. Kept separate so the
|
||||||
|
// client-facing port can be exposed without exposing diagnostics.
|
||||||
|
AdminListen string `koanf:"admin_listen"`
|
||||||
|
|
||||||
|
Router Router `koanf:"router"`
|
||||||
|
Models []Model `koanf:"models"`
|
||||||
|
Endpoints []Endpoint `koanf:"endpoints"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Router holds routing behaviour.
|
||||||
|
type Router struct {
|
||||||
|
ScrapeInterval time.Duration `koanf:"scrape_interval"`
|
||||||
|
ScrapeTimeout time.Duration `koanf:"scrape_timeout"`
|
||||||
|
// StaleAfter is how long an endpoint may go unscraped before being scored
|
||||||
|
// down (plan §7.6).
|
||||||
|
StaleAfter time.Duration `koanf:"stale_after"`
|
||||||
|
|
||||||
|
// Retries is how many *additional* endpoints may be tried after a failure,
|
||||||
|
// and only while nothing has been written to the client (plan §7.7).
|
||||||
|
Retries int `koanf:"retries"`
|
||||||
|
// Cooldown is how long a failing endpoint leaves the candidate pool.
|
||||||
|
Cooldown time.Duration `koanf:"cooldown"`
|
||||||
|
// FailureThreshold is the consecutive failure count that opens a breaker.
|
||||||
|
// 1 turns the breaker into a pure cooldown.
|
||||||
|
FailureThreshold int `koanf:"failure_threshold"`
|
||||||
|
|
||||||
|
// DefaultProfile names the routing profile used when a model picks none.
|
||||||
|
DefaultProfile string `koanf:"default_profile"`
|
||||||
|
// SessionHeader carries the session id for affinity (plan §7.5 tier 0).
|
||||||
|
SessionHeader string `koanf:"session_header"`
|
||||||
|
// InjectUsage asks engines for usage even when the client did not, so token
|
||||||
|
// accounting works regardless of client behaviour.
|
||||||
|
InjectUsage bool `koanf:"inject_usage"`
|
||||||
|
|
||||||
|
// MaxBodyBytes bounds the request body the router will read.
|
||||||
|
MaxBodyBytes int64 `koanf:"max_body_bytes"`
|
||||||
|
|
||||||
|
Timeouts Timeouts `koanf:"timeouts"`
|
||||||
|
Scoring Scoring `koanf:"scoring"`
|
||||||
|
Admission Admission `koanf:"admission"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Admission configures the flow control layer (plan §7.2).
|
||||||
|
type Admission struct {
|
||||||
|
// Enabled turns the layer on. Off, every request goes straight through,
|
||||||
|
// which is the v0.2.0 behaviour.
|
||||||
|
Enabled bool `koanf:"enabled"`
|
||||||
|
|
||||||
|
Bands []Band `koanf:"bands"`
|
||||||
|
DefaultBand string `koanf:"default_band"`
|
||||||
|
// BandHeader lets a client declare its band. Until authentication lands in
|
||||||
|
// v1.0.0 this is trusted, so the port must not be exposed to callers who
|
||||||
|
// should not be able to promote themselves.
|
||||||
|
BandHeader string `koanf:"band_header"`
|
||||||
|
|
||||||
|
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||||
|
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
|
||||||
|
Interval time.Duration `koanf:"interval"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Band is one priority class.
|
||||||
|
type Band struct {
|
||||||
|
Name string `koanf:"name"`
|
||||||
|
Priority int `koanf:"priority"`
|
||||||
|
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Timeouts bounds each phase of an upstream request.
|
||||||
|
type Timeouts struct {
|
||||||
|
Dial time.Duration `koanf:"dial"`
|
||||||
|
ResponseHeader time.Duration `koanf:"response_header"`
|
||||||
|
FirstToken time.Duration `koanf:"first_token"`
|
||||||
|
Idle time.Duration `koanf:"idle"`
|
||||||
|
Total time.Duration `koanf:"total"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scoring holds the thresholds the scorers normalize against. These want to be
|
||||||
|
// measured with ctl bench (plan §10), not guessed.
|
||||||
|
type Scoring struct {
|
||||||
|
TargetWaiting float64 `koanf:"target_waiting"`
|
||||||
|
TargetInflight float64 `koanf:"target_inflight"`
|
||||||
|
KVCeiling float64 `koanf:"kv_ceiling"`
|
||||||
|
SessionLoadFactor float64 `koanf:"session_load_factor"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Model is a catalog entry as clients see it.
|
||||||
|
type Model struct {
|
||||||
|
Name string `koanf:"name"`
|
||||||
|
Aliases []string `koanf:"aliases"`
|
||||||
|
// Fallbacks are tried, in order, when this model has no usable endpoint.
|
||||||
|
Fallbacks []string `koanf:"fallbacks"`
|
||||||
|
// RoutingProfile overrides the router's default profile for this model.
|
||||||
|
RoutingProfile string `koanf:"routing_profile"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Endpoint is one engine replica. Static in phase 1.
|
||||||
|
type Endpoint struct {
|
||||||
|
ID string `koanf:"id"`
|
||||||
|
URL string `koanf:"url"`
|
||||||
|
Models []string `koanf:"models"`
|
||||||
|
Pool string `koanf:"pool"`
|
||||||
|
Weight float64 `koanf:"weight"`
|
||||||
|
Priority int `koanf:"priority"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Default returns the configuration used when nothing is specified.
|
||||||
|
func Default() Server {
|
||||||
|
return Server{
|
||||||
|
Listen: ":8080",
|
||||||
|
AdminListen: ":9090",
|
||||||
|
Router: Router{
|
||||||
|
ScrapeInterval: 2 * time.Second,
|
||||||
|
ScrapeTimeout: time.Second,
|
||||||
|
StaleAfter: 10 * time.Second,
|
||||||
|
Retries: 2,
|
||||||
|
Cooldown: 15 * time.Second,
|
||||||
|
FailureThreshold: 3,
|
||||||
|
DefaultProfile: "chat",
|
||||||
|
SessionHeader: "x-session-id",
|
||||||
|
InjectUsage: true,
|
||||||
|
MaxBodyBytes: 32 << 20,
|
||||||
|
Timeouts: Timeouts{
|
||||||
|
Dial: 5 * time.Second,
|
||||||
|
ResponseHeader: 60 * time.Second,
|
||||||
|
FirstToken: 120 * time.Second,
|
||||||
|
Idle: 60 * time.Second,
|
||||||
|
Total: 30 * time.Minute,
|
||||||
|
},
|
||||||
|
Scoring: Scoring{
|
||||||
|
TargetWaiting: 4,
|
||||||
|
TargetInflight: 8,
|
||||||
|
KVCeiling: 0.9,
|
||||||
|
SessionLoadFactor: 1.25,
|
||||||
|
},
|
||||||
|
Admission: Admission{
|
||||||
|
Enabled: true,
|
||||||
|
BandHeader: "x-llmgw-band",
|
||||||
|
DefaultBand: "gold",
|
||||||
|
Bands: []Band{
|
||||||
|
{Name: "platinum", Priority: 400, SaturationCeiling: 1.00},
|
||||||
|
{Name: "gold", Priority: 300, SaturationCeiling: 0.85},
|
||||||
|
{Name: "silver", Priority: 200, SaturationCeiling: 0.60},
|
||||||
|
{Name: "bronze", Priority: 100, SaturationCeiling: 0.30},
|
||||||
|
},
|
||||||
|
QueueTTL: 30 * time.Second,
|
||||||
|
QueueCapacityPerBand: 2000,
|
||||||
|
Interval: 10 * time.Millisecond,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load reads configuration from path, overlaying it on the defaults, then
|
||||||
|
// applies LLMGW_-prefixed environment overrides.
|
||||||
|
//
|
||||||
|
// An empty path loads the defaults, which is what makes `llmgw server` with a
|
||||||
|
// couple of --endpoint flags usable for a smoke test.
|
||||||
|
func Load(path string) (Server, error) {
|
||||||
|
cfg := Default()
|
||||||
|
k := koanf.New(".")
|
||||||
|
|
||||||
|
if path != "" {
|
||||||
|
if _, err := os.Stat(path); err != nil {
|
||||||
|
return cfg, fmt.Errorf("config file: %w", err)
|
||||||
|
}
|
||||||
|
if err := k.Load(file.Provider(path), yaml.Parser()); err != nil {
|
||||||
|
return cfg, fmt.Errorf("parsing %s: %w", path, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// LLMGW_ROUTER__RETRIES=3 maps onto router.retries.
|
||||||
|
err := k.Load(env.Provider(".", env.Opt{
|
||||||
|
Prefix: "LLMGW_",
|
||||||
|
TransformFunc: func(key, value string) (string, any) {
|
||||||
|
key = strings.ToLower(strings.TrimPrefix(key, "LLMGW_"))
|
||||||
|
return strings.ReplaceAll(key, "__", "."), value
|
||||||
|
},
|
||||||
|
}), nil)
|
||||||
|
if err != nil {
|
||||||
|
return cfg, fmt.Errorf("environment overrides: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := k.UnmarshalWithConf("", &cfg, koanf.UnmarshalConf{Tag: "koanf"}); err != nil {
|
||||||
|
return cfg, fmt.Errorf("decoding configuration: %w", err)
|
||||||
|
}
|
||||||
|
return cfg, cfg.Validate()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate rejects configurations that would otherwise fail confusingly at
|
||||||
|
// request time.
|
||||||
|
func (s Server) Validate() error {
|
||||||
|
if s.Listen == "" {
|
||||||
|
return fmt.Errorf("listen is empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
seen := make(map[string]struct{}, len(s.Endpoints))
|
||||||
|
for i, e := range s.Endpoints {
|
||||||
|
switch {
|
||||||
|
case e.ID == "":
|
||||||
|
return fmt.Errorf("endpoints[%d]: id is required", i)
|
||||||
|
case e.URL == "":
|
||||||
|
return fmt.Errorf("endpoint %q: url is required", e.ID)
|
||||||
|
case !strings.HasPrefix(e.URL, "http://") && !strings.HasPrefix(e.URL, "https://"):
|
||||||
|
return fmt.Errorf("endpoint %q: url must start with http:// or https://", e.ID)
|
||||||
|
case len(e.Models) == 0:
|
||||||
|
return fmt.Errorf("endpoint %q: at least one model is required", e.ID)
|
||||||
|
}
|
||||||
|
if _, dup := seen[e.ID]; dup {
|
||||||
|
return fmt.Errorf("endpoint id %q is used twice", e.ID)
|
||||||
|
}
|
||||||
|
seen[e.ID] = struct{}{}
|
||||||
|
}
|
||||||
|
|
||||||
|
names := make(map[string]struct{}, len(s.Models))
|
||||||
|
for i, m := range s.Models {
|
||||||
|
if m.Name == "" {
|
||||||
|
return fmt.Errorf("models[%d]: name is required", i)
|
||||||
|
}
|
||||||
|
if _, dup := names[m.Name]; dup {
|
||||||
|
return fmt.Errorf("model %q is declared twice", m.Name)
|
||||||
|
}
|
||||||
|
names[m.Name] = struct{}{}
|
||||||
|
}
|
||||||
|
// Aliases share a namespace with names, so a collision must be caught here
|
||||||
|
// rather than silently shadowing a model at request time.
|
||||||
|
for _, m := range s.Models {
|
||||||
|
for _, a := range m.Aliases {
|
||||||
|
if _, clash := names[a]; clash {
|
||||||
|
return fmt.Errorf("model %q: alias %q collides with another model name", m.Name, a)
|
||||||
|
}
|
||||||
|
names[a] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,150 @@
|
|||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func write(t *testing.T, content string) string {
|
||||||
|
t.Helper()
|
||||||
|
p := filepath.Join(t.TempDir(), "server.yaml")
|
||||||
|
if err := os.WriteFile(p, []byte(content), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadDefaultsWithoutFile(t *testing.T) {
|
||||||
|
cfg, err := Load("")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cfg.Listen != ":8080" || cfg.Router.DefaultProfile != "chat" {
|
||||||
|
t.Errorf("defaults not applied: %+v", cfg)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadOverlaysOnDefaults(t *testing.T) {
|
||||||
|
// Specifying one field must not blank out its neighbours.
|
||||||
|
p := write(t, `
|
||||||
|
listen: ":9999"
|
||||||
|
endpoints:
|
||||||
|
- id: a
|
||||||
|
url: http://127.0.0.1:8000
|
||||||
|
models: [m]
|
||||||
|
`)
|
||||||
|
cfg, err := Load(p)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cfg.Listen != ":9999" {
|
||||||
|
t.Errorf("listen=%q", cfg.Listen)
|
||||||
|
}
|
||||||
|
if cfg.Router.ScrapeInterval != 2*time.Second {
|
||||||
|
t.Errorf("an unspecified default was lost: %v", cfg.Router.ScrapeInterval)
|
||||||
|
}
|
||||||
|
if len(cfg.Endpoints) != 1 || cfg.Endpoints[0].ID != "a" {
|
||||||
|
t.Errorf("endpoints=%+v", cfg.Endpoints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDurationsParse(t *testing.T) {
|
||||||
|
p := write(t, `
|
||||||
|
router:
|
||||||
|
scrape_interval: 500ms
|
||||||
|
timeouts:
|
||||||
|
idle: 45s
|
||||||
|
`)
|
||||||
|
cfg, err := Load(p)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cfg.Router.ScrapeInterval != 500*time.Millisecond {
|
||||||
|
t.Errorf("scrape_interval=%v", cfg.Router.ScrapeInterval)
|
||||||
|
}
|
||||||
|
if cfg.Router.Timeouts.Idle != 45*time.Second {
|
||||||
|
t.Errorf("timeouts.idle=%v", cfg.Router.Timeouts.Idle)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnvOverride(t *testing.T) {
|
||||||
|
t.Setenv("LLMGW_ROUTER__RETRIES", "7")
|
||||||
|
cfg, err := Load("")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if cfg.Router.Retries != 7 {
|
||||||
|
t.Errorf("retries=%d, expected the environment to win", cfg.Router.Retries)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMissingFileIsAnError(t *testing.T) {
|
||||||
|
if _, err := Load("/nonexistent/server.yaml"); err == nil {
|
||||||
|
t.Error("expected an error for a missing config file")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRejectsBadEndpoints(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
name string
|
||||||
|
ep Endpoint
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"no id", Endpoint{URL: "http://x", Models: []string{"m"}}, "id is required"},
|
||||||
|
{"no url", Endpoint{ID: "a", Models: []string{"m"}}, "url is required"},
|
||||||
|
{"bad scheme", Endpoint{ID: "a", URL: "127.0.0.1:8000", Models: []string{"m"}}, "http://"},
|
||||||
|
{"no models", Endpoint{ID: "a", URL: "http://x"}, "at least one model"},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
t.Run(c.name, func(t *testing.T) {
|
||||||
|
s := Default()
|
||||||
|
s.Endpoints = []Endpoint{c.ep}
|
||||||
|
err := s.Validate()
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), c.want) {
|
||||||
|
t.Errorf("error %q does not mention %q", err, c.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRejectsDuplicateEndpointID(t *testing.T) {
|
||||||
|
s := Default()
|
||||||
|
s.Endpoints = []Endpoint{
|
||||||
|
{ID: "a", URL: "http://x", Models: []string{"m"}},
|
||||||
|
{ID: "a", URL: "http://y", Models: []string{"m"}},
|
||||||
|
}
|
||||||
|
if err := s.Validate(); err == nil || !strings.Contains(err.Error(), "twice") {
|
||||||
|
t.Errorf("duplicate id not rejected: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRejectsAliasCollision(t *testing.T) {
|
||||||
|
// An alias shadowing another model would silently misroute every request
|
||||||
|
// for that model, so it has to fail at load time.
|
||||||
|
s := Default()
|
||||||
|
s.Models = []Model{
|
||||||
|
{Name: "llama-3.3-70b", Aliases: []string{"qwen"}},
|
||||||
|
{Name: "qwen"},
|
||||||
|
}
|
||||||
|
err := s.Validate()
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "collides") {
|
||||||
|
t.Errorf("alias collision not rejected: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateAcceptsDistinctAliases(t *testing.T) {
|
||||||
|
s := Default()
|
||||||
|
s.Models = []Model{
|
||||||
|
{Name: "llama-3.3-70b", Aliases: []string{"llama", "llama3"}},
|
||||||
|
{Name: "qwen2.5-32b", Aliases: []string{"qwen"}},
|
||||||
|
}
|
||||||
|
if err := s.Validate(); err != nil {
|
||||||
|
t.Errorf("valid aliases rejected: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
package router
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"net/http/pprof"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus"
|
||||||
|
"github.com/prometheus/client_golang/prometheus/collectors"
|
||||||
|
"github.com/prometheus/client_golang/prometheus/promhttp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// AdminHandler returns the diagnostics surface: metrics, health and pprof.
|
||||||
|
//
|
||||||
|
// It is served on a separate listener so the client-facing port can be exposed
|
||||||
|
// without also exposing pprof (plan §9.1).
|
||||||
|
func (r *Router) AdminHandler() (http.Handler, error) {
|
||||||
|
reg := prometheus.NewRegistry()
|
||||||
|
reg.MustRegister(
|
||||||
|
collectors.NewGoCollector(),
|
||||||
|
collectors.NewProcessCollector(collectors.ProcessCollectorOpts{}),
|
||||||
|
)
|
||||||
|
for _, c := range r.m.collectors() {
|
||||||
|
if err := reg.Register(c); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := reg.Register(newEndpointCollector(r)); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.Handle("GET /metrics", promhttp.HandlerFor(reg, promhttp.HandlerOpts{}))
|
||||||
|
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
})
|
||||||
|
mux.HandleFunc("GET /debug/pprof/", pprof.Index)
|
||||||
|
mux.HandleFunc("GET /debug/pprof/cmdline", pprof.Cmdline)
|
||||||
|
mux.HandleFunc("GET /debug/pprof/profile", pprof.Profile)
|
||||||
|
mux.HandleFunc("GET /debug/pprof/symbol", pprof.Symbol)
|
||||||
|
mux.HandleFunc("GET /debug/pprof/trace", pprof.Trace)
|
||||||
|
return mux, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// endpointCollector reports live endpoint state at scrape time rather than
|
||||||
|
// mirroring it into gauges on every request; the registry already holds it.
|
||||||
|
type endpointCollector struct {
|
||||||
|
r *Router
|
||||||
|
|
||||||
|
inflight *prometheus.Desc
|
||||||
|
waiting *prometheus.Desc
|
||||||
|
running *prometheus.Desc
|
||||||
|
kv *prometheus.Desc
|
||||||
|
stale *prometheus.Desc
|
||||||
|
drain *prometheus.Desc
|
||||||
|
breaker *prometheus.Desc
|
||||||
|
|
||||||
|
queueDepth *prometheus.Desc
|
||||||
|
saturation *prometheus.Desc
|
||||||
|
}
|
||||||
|
|
||||||
|
func newEndpointCollector(r *Router) *endpointCollector {
|
||||||
|
lbl := []string{"replica", "pool"}
|
||||||
|
return &endpointCollector{
|
||||||
|
r: r,
|
||||||
|
inflight: prometheus.NewDesc("llmgw_endpoint_inflight", "Requests the router has dispatched and not yet finished.", lbl, nil),
|
||||||
|
waiting: prometheus.NewDesc("llmgw_endpoint_waiting", "Engine queue depth, as last scraped.", lbl, nil),
|
||||||
|
running: prometheus.NewDesc("llmgw_endpoint_running", "Requests running on the engine, as last scraped.", lbl, nil),
|
||||||
|
kv: prometheus.NewDesc("llmgw_endpoint_kv_cache_usage", "Engine KV cache occupancy, 0..1.", lbl, nil),
|
||||||
|
drain: prometheus.NewDesc("llmgw_endpoint_drain_rate", "Observed completions per second.", lbl, nil),
|
||||||
|
stale: prometheus.NewDesc("llmgw_endpoint_metrics_stale", "1 when the endpoint's metrics are stale.", lbl, nil),
|
||||||
|
breaker: prometheus.NewDesc("llmgw_endpoint_breaker_open", "1 when the endpoint's circuit breaker is open.", lbl, nil),
|
||||||
|
|
||||||
|
queueDepth: prometheus.NewDesc("llmgw_admission_queue_depth", "Requests waiting in each admission band.", []string{"band"}, nil),
|
||||||
|
saturation: prometheus.NewDesc("llmgw_pool_saturation", "Pool saturation per model; 1 is the healthy limit.", []string{"model"}, nil),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *endpointCollector) Describe(ch chan<- *prometheus.Desc) {
|
||||||
|
ch <- c.inflight
|
||||||
|
ch <- c.waiting
|
||||||
|
ch <- c.running
|
||||||
|
ch <- c.kv
|
||||||
|
ch <- c.drain
|
||||||
|
ch <- c.stale
|
||||||
|
ch <- c.breaker
|
||||||
|
ch <- c.queueDepth
|
||||||
|
ch <- c.saturation
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *endpointCollector) Collect(ch chan<- prometheus.Metric) {
|
||||||
|
for _, s := range c.r.reg.Snapshots("") {
|
||||||
|
pool := s.Pool
|
||||||
|
if pool == "" {
|
||||||
|
pool = "none"
|
||||||
|
}
|
||||||
|
g := func(d *prometheus.Desc, v float64) {
|
||||||
|
ch <- prometheus.MustNewConstMetric(d, prometheus.GaugeValue, v, s.ID, pool)
|
||||||
|
}
|
||||||
|
g(c.inflight, float64(s.Inflight))
|
||||||
|
g(c.waiting, float64(s.Stats.Waiting))
|
||||||
|
g(c.running, float64(s.Stats.Running))
|
||||||
|
g(c.kv, s.Stats.KVCacheUsage)
|
||||||
|
g(c.drain, s.DrainRate)
|
||||||
|
g(c.stale, boolToFloat(s.Stale))
|
||||||
|
g(c.breaker, boolToFloat(s.Breaker.String() == "open"))
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.r.adm != nil {
|
||||||
|
for _, d := range c.r.adm.Depths() {
|
||||||
|
ch <- prometheus.MustNewConstMetric(c.queueDepth, prometheus.GaugeValue, float64(d.Depth), d.Band)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Saturation is reported per model, which is the granularity admission
|
||||||
|
// decisions are made at.
|
||||||
|
for _, model := range c.r.servedModels() {
|
||||||
|
if sat, ok := c.r.saturation(model, 0); ok {
|
||||||
|
ch <- prometheus.MustNewConstMetric(c.saturation, prometheus.GaugeValue, sat, model)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func boolToFloat(b bool) float64 {
|
||||||
|
if b {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
@@ -0,0 +1,436 @@
|
|||||||
|
package admission
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||||
|
)
|
||||||
|
|
||||||
|
func testConfig() Config {
|
||||||
|
c := DefaultConfig()
|
||||||
|
c.Interval = time.Millisecond
|
||||||
|
c.QueueTTL = 2 * time.Second
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
// fixedSat lets a test drive saturation directly.
|
||||||
|
type fixedSat struct{ v atomic.Uint64 }
|
||||||
|
|
||||||
|
func (f *fixedSat) set(v float64) { f.v.Store(uint64(v * 1000)) }
|
||||||
|
func (f *fixedSat) fn(string, int) (float64, bool) {
|
||||||
|
return float64(f.v.Load()) / 1000, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func newController(t *testing.T, cfg Config, sat SaturationFunc) *Controller {
|
||||||
|
t.Helper()
|
||||||
|
c, err := New(cfg, sat)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
c.Start()
|
||||||
|
t.Cleanup(c.Close)
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAdmitsImmediatelyWhenIdle(t *testing.T) {
|
||||||
|
s := &fixedSat{}
|
||||||
|
c := newController(t, testConfig(), s.fn)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if d := time.Since(start); d > 50*time.Millisecond {
|
||||||
|
t.Errorf("idle admission took %v; the fast path is not being used", d)
|
||||||
|
}
|
||||||
|
lease.Release()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLowBandHeldWhileHighBandFlows(t *testing.T) {
|
||||||
|
// The core of plan §7.2.2: the cluster degrades from the bottom up.
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(0.5) // above bronze (0.30), below gold (0.85)
|
||||||
|
c := newController(t, testConfig(), s.fn)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(t.Context(), 300*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
if _, err := c.Acquire(ctx, Request{Model: "m", Band: "bronze"}); err == nil {
|
||||||
|
t.Error("bronze was admitted above its ceiling")
|
||||||
|
}
|
||||||
|
|
||||||
|
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "gold"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("gold was held below its ceiling: %v", err)
|
||||||
|
}
|
||||||
|
lease.Release()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTopBandFlowsUpToTheHealthyLimit(t *testing.T) {
|
||||||
|
// A ceiling of 1.0 means "hold only once the pool is past its healthy
|
||||||
|
// limit", not "never hold". At exactly the limit the top band still flows.
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(1.0)
|
||||||
|
c := newController(t, testConfig(), s.fn)
|
||||||
|
|
||||||
|
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "platinum"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("top band held at exactly its ceiling: %v", err)
|
||||||
|
}
|
||||||
|
lease.Release()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTopBandIsHeldWhenOverloaded(t *testing.T) {
|
||||||
|
// Holding even the top band is deliberate: with nobody above it to protect,
|
||||||
|
// the benefit is a bounded engine queue. Dumping a flood straight into a
|
||||||
|
// FIFO engine queue makes every request in that band slow, whereas queueing
|
||||||
|
// at the gateway keeps ordering, TTL and tenant fairness.
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(5)
|
||||||
|
cfg := testConfig()
|
||||||
|
cfg.QueueTTL = 200 * time.Millisecond
|
||||||
|
c := newController(t, cfg, s.fn)
|
||||||
|
|
||||||
|
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "platinum"}); !errors.Is(err, ErrQueueTimeout) {
|
||||||
|
t.Fatalf("expected the top band to queue while overloaded, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCeilingAboveOneNeverHolds(t *testing.T) {
|
||||||
|
// The escape hatch for a band that must never wait, whatever the load.
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(50)
|
||||||
|
cfg := testConfig()
|
||||||
|
cfg.Bands = []Band{{Name: "critical", Priority: 900, SaturationCeiling: 1000}}
|
||||||
|
cfg.DefaultBand = "critical"
|
||||||
|
c := newController(t, cfg, s.fn)
|
||||||
|
|
||||||
|
lease, err := c.Acquire(t.Context(), Request{Model: "m", Band: "critical"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("a band with an effectively unbounded ceiling was held: %v", err)
|
||||||
|
}
|
||||||
|
lease.Release()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestQueuedRequestAdmittedWhenSaturationDrops(t *testing.T) {
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(0.9)
|
||||||
|
c := newController(t, testConfig(), s.fn)
|
||||||
|
|
||||||
|
done := make(chan error, 1)
|
||||||
|
go func() {
|
||||||
|
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||||
|
done <- err
|
||||||
|
}()
|
||||||
|
|
||||||
|
// Give it time to actually queue, then open the gate.
|
||||||
|
time.Sleep(50 * time.Millisecond)
|
||||||
|
if d := c.Depths(); d[3].Depth != 1 {
|
||||||
|
t.Fatalf("expected 1 queued bronze request, got %+v", d)
|
||||||
|
}
|
||||||
|
s.set(0.1)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case err := <-done:
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("queued request failed: %v", err)
|
||||||
|
}
|
||||||
|
case <-time.After(2 * time.Second):
|
||||||
|
t.Fatal("queued request was not admitted after saturation dropped")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestQueueTTL(t *testing.T) {
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(1.0)
|
||||||
|
cfg := testConfig()
|
||||||
|
cfg.QueueTTL = 150 * time.Millisecond
|
||||||
|
c := newController(t, cfg, s.fn)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||||
|
if !errors.Is(err, ErrQueueTimeout) {
|
||||||
|
t.Fatalf("expected ErrQueueTimeout, got %v", err)
|
||||||
|
}
|
||||||
|
if d := time.Since(start); d > time.Second {
|
||||||
|
t.Errorf("TTL fired after %v", d)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestQueueCapacity(t *testing.T) {
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(1.0)
|
||||||
|
cfg := testConfig()
|
||||||
|
cfg.QueueCapacityPerBand = 3
|
||||||
|
cfg.QueueTTL = 5 * time.Second
|
||||||
|
c := newController(t, cfg, s.fn)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
defer cancel()
|
||||||
|
for range 3 {
|
||||||
|
go c.Acquire(ctx, Request{Model: "m", Band: "bronze"}) //nolint:errcheck // queued on purpose
|
||||||
|
}
|
||||||
|
// Wait for the queue to fill.
|
||||||
|
deadline := time.Now().Add(time.Second)
|
||||||
|
for time.Now().Before(deadline) && c.Depths()[3].Depth < 3 {
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze"})
|
||||||
|
if !errors.Is(err, ErrQueueFull) {
|
||||||
|
t.Errorf("expected ErrQueueFull, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientCancellationLeavesQueue(t *testing.T) {
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(1.0)
|
||||||
|
c := newController(t, testConfig(), s.fn)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
go c.Acquire(ctx, Request{Model: "m", Band: "bronze"}) //nolint:errcheck // cancelled on purpose
|
||||||
|
|
||||||
|
deadline := time.Now().Add(time.Second)
|
||||||
|
for time.Now().Before(deadline) && c.Depths()[3].Depth == 0 {
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
deadline = time.Now().Add(time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if c.Depths()[3].Depth == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatal("a cancelled request stayed in the queue, occupying capacity")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTenantFairnessWithinBand(t *testing.T) {
|
||||||
|
// One tenant flooding the band must not take every dispatch turn.
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(1.0) // hold everything initially
|
||||||
|
cfg := testConfig()
|
||||||
|
cfg.QueueTTL = 5 * time.Second
|
||||||
|
c := newController(t, cfg, s.fn)
|
||||||
|
|
||||||
|
const flood = 40
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
order := make(chan string, flood+4)
|
||||||
|
|
||||||
|
for i := range flood {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze", Tenant: "noisy"}); err == nil {
|
||||||
|
order <- "noisy"
|
||||||
|
}
|
||||||
|
_ = i
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
// Let the flood queue up first, so the quiet tenant is strictly behind it.
|
||||||
|
deadline := time.Now().Add(2 * time.Second)
|
||||||
|
for time.Now().Before(deadline) && c.Depths()[3].Depth < flood {
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
for range 4 {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if _, err := c.Acquire(t.Context(), Request{Model: "m", Band: "bronze", Tenant: "quiet"}); err == nil {
|
||||||
|
order <- "quiet"
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
time.Sleep(30 * time.Millisecond)
|
||||||
|
|
||||||
|
s.set(0.0) // open the gate
|
||||||
|
wg.Wait()
|
||||||
|
close(order)
|
||||||
|
|
||||||
|
var seen []string
|
||||||
|
for v := range order {
|
||||||
|
seen = append(seen, v)
|
||||||
|
}
|
||||||
|
if len(seen) != flood+4 {
|
||||||
|
t.Fatalf("expected %d admissions, got %d", flood+4, len(seen))
|
||||||
|
}
|
||||||
|
|
||||||
|
// With FIFO the quiet tenant would be admitted 41st at best. Round-robin
|
||||||
|
// should interleave it into the first handful.
|
||||||
|
firstQuiet := -1
|
||||||
|
for i, v := range seen {
|
||||||
|
if v == "quiet" {
|
||||||
|
firstQuiet = i
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if firstQuiet < 0 {
|
||||||
|
t.Fatal("the quiet tenant was never admitted")
|
||||||
|
}
|
||||||
|
if firstQuiet > 8 {
|
||||||
|
t.Errorf("quiet tenant admitted at position %d; a flood is monopolising the band", firstQuiet)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnknownBandUsesDefault(t *testing.T) {
|
||||||
|
s := &fixedSat{}
|
||||||
|
s.set(0.5)
|
||||||
|
cfg := testConfig()
|
||||||
|
cfg.DefaultBand = "bronze"
|
||||||
|
c := newController(t, cfg, s.fn)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(t.Context(), 200*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
// bronze's ceiling is 0.30, so an unknown band must be held here.
|
||||||
|
if _, err := c.Acquire(ctx, Request{Model: "m", Band: "no-such-band"}); err == nil {
|
||||||
|
t.Error("unknown band did not fall back to the default")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnknownModelIsLetThrough(t *testing.T) {
|
||||||
|
// Better a precise 503 from dispatch than a 30-second queue timeout.
|
||||||
|
c := newController(t, testConfig(), func(string, int) (float64, bool) { return 0, false })
|
||||||
|
lease, err := c.Acquire(t.Context(), Request{Model: "nope", Band: "bronze"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("unknown model was queued instead of passed through: %v", err)
|
||||||
|
}
|
||||||
|
lease.Release()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfigValidation(t *testing.T) {
|
||||||
|
if _, err := New(Config{Bands: []Band{{Name: "a"}, {Name: "a"}}}, nil); err == nil {
|
||||||
|
t.Error("duplicate band names accepted")
|
||||||
|
}
|
||||||
|
if _, err := New(Config{Bands: []Band{{Name: "a"}}, DefaultBand: "b"}, nil); err == nil {
|
||||||
|
t.Error("default_band outside the band list accepted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBandsSortedByPriority(t *testing.T) {
|
||||||
|
c, err := New(Config{Bands: []Band{
|
||||||
|
{Name: "low", Priority: 1, SaturationCeiling: 0.2},
|
||||||
|
{Name: "high", Priority: 9, SaturationCeiling: 1},
|
||||||
|
}, DefaultBand: "low"}, func(string, int) (float64, bool) { return 0, true })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := c.Bands()[0].Name; got != "high" {
|
||||||
|
t.Errorf("bands are not ordered by priority: first is %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- saturation ---
|
||||||
|
|
||||||
|
func snap(id string, waiting, running, inflight int, kv float64, open bool) endpoint.Snapshot {
|
||||||
|
s := endpoint.Snapshot{
|
||||||
|
Info: endpoint.Info{ID: id},
|
||||||
|
Inflight: inflight,
|
||||||
|
Breaker: breaker.Closed,
|
||||||
|
}
|
||||||
|
s.Stats.Waiting = waiting
|
||||||
|
s.Stats.Running = running
|
||||||
|
s.Stats.KVCacheUsage = kv
|
||||||
|
if open {
|
||||||
|
s.Breaker = breaker.Open
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSaturationCombinesQueueAndKV(t *testing.T) {
|
||||||
|
th := Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9}
|
||||||
|
|
||||||
|
// Empty queue but a nearly full cache still counts as saturated.
|
||||||
|
if got := Saturation([]endpoint.Snapshot{snap("a", 0, 1, 1, 0.89, false)}, 0, th); got < 0.9 {
|
||||||
|
t.Errorf("KV pressure ignored: %v", got)
|
||||||
|
}
|
||||||
|
// Full queue with an empty cache likewise.
|
||||||
|
if got := Saturation([]endpoint.Snapshot{snap("a", 4, 1, 5, 0, false)}, 0, th); got < 1 {
|
||||||
|
t.Errorf("queue depth ignored: %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSaturationCountsUnreportedInflight(t *testing.T) {
|
||||||
|
// A burst the engine has not reported yet is real load; missing it would
|
||||||
|
// leave admission blind exactly when it needs to bite.
|
||||||
|
th := Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9}
|
||||||
|
cold := Saturation([]endpoint.Snapshot{snap("a", 0, 0, 0, 0, false)}, 0, th)
|
||||||
|
burst := Saturation([]endpoint.Snapshot{snap("a", 0, 0, 8, 0, false)}, 0, th)
|
||||||
|
if !(burst > cold) {
|
||||||
|
t.Errorf("unreported inflight not counted: cold=%v burst=%v", cold, burst)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSaturationSkipsOpenBreakers(t *testing.T) {
|
||||||
|
th := Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9}
|
||||||
|
eps := []endpoint.Snapshot{
|
||||||
|
snap("busy", 8, 4, 12, 0.5, false),
|
||||||
|
snap("dead", 0, 0, 0, 0, true), // would halve the mean if counted
|
||||||
|
}
|
||||||
|
withDead := Saturation(eps, 0, th)
|
||||||
|
alone := Saturation(eps[:1], 0, th)
|
||||||
|
if fmt.Sprintf("%.4f", withDead) != fmt.Sprintf("%.4f", alone) {
|
||||||
|
t.Errorf("an endpoint with an open breaker was counted as capacity: %v vs %v", withDead, alone)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSaturationCountsUnreportedAdmissions(t *testing.T) {
|
||||||
|
// The thundering herd guard: a simultaneous burst must register as load
|
||||||
|
// before any of it reaches an engine, or the whole burst walks through the
|
||||||
|
// fast path while every signal still reads zero.
|
||||||
|
th := Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9}
|
||||||
|
idle := []endpoint.Snapshot{snap("a", 0, 0, 0, 0, false), snap("b", 0, 0, 0, 0, false)}
|
||||||
|
|
||||||
|
if got := Saturation(idle, 0, th); got != 0 {
|
||||||
|
t.Fatalf("idle pool reported %v", got)
|
||||||
|
}
|
||||||
|
if got := Saturation(idle, 16, th); got <= 1 {
|
||||||
|
t.Errorf("16 unreported admissions across 2 replicas should saturate, got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBurstDoesNotAllPassTheFastPath(t *testing.T) {
|
||||||
|
// End to end for the same guard: with saturation derived from outstanding
|
||||||
|
// leases, a burst of 200 must not be admitted wholesale.
|
||||||
|
var admitted atomic.Int64
|
||||||
|
c := newController(t, testConfig(), func(_ string, outstanding int) (float64, bool) {
|
||||||
|
// Two replicas, healthy buffer of 4 each: 8 in flight is the limit.
|
||||||
|
return float64(outstanding) / 8, true
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for range 200 {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
if _, err := c.Acquire(ctx, Request{Model: "m", Band: "bronze"}); err == nil {
|
||||||
|
admitted.Add(1)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
// bronze's ceiling is 0.30, so roughly 8*0.3 = 2-3 should get in. Allow
|
||||||
|
// generous slack for scheduling, but nothing near 200.
|
||||||
|
if n := admitted.Load(); n > 20 {
|
||||||
|
t.Errorf("%d of 200 requests were admitted at once; the burst guard is not working", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSaturationWithNoUsableEndpoints(t *testing.T) {
|
||||||
|
th := Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9}
|
||||||
|
if got := Saturation(nil, 0, th); got != 1 {
|
||||||
|
t.Errorf("an empty pool should report full saturation, got %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,412 @@
|
|||||||
|
package admission
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"slices"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Band is a priority class. Requests in a band are held once pool saturation
|
||||||
|
// exceeds the band's ceiling, so the cluster degrades from the bottom up:
|
||||||
|
// batch traffic slows first and interactive traffic keeps its latency.
|
||||||
|
type Band struct {
|
||||||
|
Name string `koanf:"name"`
|
||||||
|
// Priority orders bands; higher is served first.
|
||||||
|
Priority int `koanf:"priority"`
|
||||||
|
// SaturationCeiling is the pool saturation above which this band waits.
|
||||||
|
// 1.0 means "never hold this band back".
|
||||||
|
SaturationCeiling float64 `koanf:"saturation_ceiling"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Config configures the controller.
|
||||||
|
type Config struct {
|
||||||
|
Bands []Band `koanf:"bands"`
|
||||||
|
DefaultBand string `koanf:"default_band"`
|
||||||
|
|
||||||
|
Thresholds Thresholds `koanf:"-"`
|
||||||
|
|
||||||
|
// QueueTTL drops a request that has waited this long. Sending the GPU work
|
||||||
|
// the client has already given up on helps nobody.
|
||||||
|
QueueTTL time.Duration `koanf:"queue_ttl"`
|
||||||
|
// QueueCapacityPerBand bounds each band's queue. Rejecting immediately
|
||||||
|
// beats accumulating silently.
|
||||||
|
QueueCapacityPerBand int `koanf:"queue_capacity_per_band"`
|
||||||
|
// Interval is how often the dispatcher re-checks saturation. Releases also
|
||||||
|
// wake it, so this is only the backstop for saturation falling on its own.
|
||||||
|
Interval time.Duration `koanf:"interval"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultBands returns the four-band scheme from plan §7.2.2.
|
||||||
|
func DefaultBands() []Band {
|
||||||
|
return []Band{
|
||||||
|
{Name: "platinum", Priority: 400, SaturationCeiling: 1.00},
|
||||||
|
{Name: "gold", Priority: 300, SaturationCeiling: 0.85},
|
||||||
|
{Name: "silver", Priority: 200, SaturationCeiling: 0.60},
|
||||||
|
{Name: "bronze", Priority: 100, SaturationCeiling: 0.30},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultConfig returns a usable configuration.
|
||||||
|
func DefaultConfig() Config {
|
||||||
|
return Config{
|
||||||
|
Bands: DefaultBands(),
|
||||||
|
DefaultBand: "gold",
|
||||||
|
Thresholds: Thresholds{TargetWaitingPerReplica: 4, KVCeiling: 0.9},
|
||||||
|
QueueTTL: 30 * time.Second,
|
||||||
|
QueueCapacityPerBand: 2000,
|
||||||
|
Interval: 10 * time.Millisecond,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) normalize() error {
|
||||||
|
if len(c.Bands) == 0 {
|
||||||
|
c.Bands = DefaultBands()
|
||||||
|
}
|
||||||
|
slices.SortFunc(c.Bands, func(a, b Band) int { return b.Priority - a.Priority })
|
||||||
|
|
||||||
|
seen := map[string]struct{}{}
|
||||||
|
for i := range c.Bands {
|
||||||
|
b := &c.Bands[i]
|
||||||
|
if b.Name == "" {
|
||||||
|
return fmt.Errorf("bands[%d]: name is required", i)
|
||||||
|
}
|
||||||
|
if _, dup := seen[b.Name]; dup {
|
||||||
|
return fmt.Errorf("band %q is declared twice", b.Name)
|
||||||
|
}
|
||||||
|
seen[b.Name] = struct{}{}
|
||||||
|
if b.SaturationCeiling <= 0 {
|
||||||
|
b.SaturationCeiling = 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if c.DefaultBand == "" {
|
||||||
|
c.DefaultBand = c.Bands[len(c.Bands)-1].Name
|
||||||
|
}
|
||||||
|
if _, ok := seen[c.DefaultBand]; !ok {
|
||||||
|
return fmt.Errorf("default_band %q is not one of the configured bands", c.DefaultBand)
|
||||||
|
}
|
||||||
|
c.Thresholds.normalize()
|
||||||
|
if c.QueueTTL <= 0 {
|
||||||
|
c.QueueTTL = 30 * time.Second
|
||||||
|
}
|
||||||
|
if c.QueueCapacityPerBand <= 0 {
|
||||||
|
c.QueueCapacityPerBand = 2000
|
||||||
|
}
|
||||||
|
if c.Interval <= 0 {
|
||||||
|
c.Interval = 10 * time.Millisecond
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// SaturationFunc reports a model's pool saturation and whether it is known.
|
||||||
|
//
|
||||||
|
// outstanding is how many requests this controller has admitted and not yet
|
||||||
|
// seen released, so the caller can fold in load the engines have not reported.
|
||||||
|
type SaturationFunc func(model string, outstanding int) (float64, bool)
|
||||||
|
|
||||||
|
// Request identifies who is asking for what.
|
||||||
|
type Request struct {
|
||||||
|
Model string
|
||||||
|
// Band selects the priority class; unknown names fall back to the default.
|
||||||
|
Band string
|
||||||
|
// Tenant is the fairness key within a band. Two requests sharing a tenant
|
||||||
|
// compete with each other rather than with everyone else.
|
||||||
|
Tenant string
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
// ErrQueueFull means the band's queue is at capacity.
|
||||||
|
ErrQueueFull = errors.New("admission queue is full")
|
||||||
|
// ErrQueueTimeout means the request waited longer than the TTL.
|
||||||
|
ErrQueueTimeout = errors.New("admission queue timeout")
|
||||||
|
// ErrClosed means the controller is shutting down.
|
||||||
|
ErrClosed = errors.New("admission controller closed")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Stats is a snapshot of one band's queue, for metrics.
|
||||||
|
type Stats struct {
|
||||||
|
Band string
|
||||||
|
Depth int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Controller admits requests according to band priority and tenant fairness.
|
||||||
|
type Controller struct {
|
||||||
|
cfg Config
|
||||||
|
satFn SaturationFunc
|
||||||
|
bandOf map[string]int
|
||||||
|
|
||||||
|
mu sync.Mutex
|
||||||
|
queues []*bandQueue
|
||||||
|
closed bool
|
||||||
|
|
||||||
|
wake chan struct{}
|
||||||
|
done chan struct{}
|
||||||
|
stop chan struct{}
|
||||||
|
once sync.Once
|
||||||
|
|
||||||
|
// outstanding counts admitted-but-unreleased requests per model. It is the
|
||||||
|
// only load signal available with zero lag, which is what stops a
|
||||||
|
// simultaneous burst from passing the check en masse.
|
||||||
|
outstanding map[string]int
|
||||||
|
|
||||||
|
// Observed counters, read through Metrics.
|
||||||
|
admitted map[string]int64
|
||||||
|
rejected map[string]int64
|
||||||
|
waitTotal map[string]time.Duration
|
||||||
|
}
|
||||||
|
|
||||||
|
// New builds a controller. Start must be called before it will admit anything
|
||||||
|
// that has to wait.
|
||||||
|
func New(cfg Config, satFn SaturationFunc) (*Controller, error) {
|
||||||
|
if err := cfg.normalize(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
c := &Controller{
|
||||||
|
cfg: cfg,
|
||||||
|
satFn: satFn,
|
||||||
|
bandOf: make(map[string]int, len(cfg.Bands)),
|
||||||
|
wake: make(chan struct{}, 1),
|
||||||
|
done: make(chan struct{}),
|
||||||
|
stop: make(chan struct{}),
|
||||||
|
outstanding: map[string]int{},
|
||||||
|
admitted: map[string]int64{},
|
||||||
|
rejected: map[string]int64{},
|
||||||
|
waitTotal: map[string]time.Duration{},
|
||||||
|
}
|
||||||
|
for i, b := range cfg.Bands {
|
||||||
|
c.bandOf[b.Name] = i
|
||||||
|
c.queues = append(c.queues, newBandQueue(b))
|
||||||
|
}
|
||||||
|
return c, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bands returns the configured bands, highest priority first.
|
||||||
|
func (c *Controller) Bands() []Band { return slices.Clone(c.cfg.Bands) }
|
||||||
|
|
||||||
|
// Start launches the dispatcher.
|
||||||
|
func (c *Controller) Start() {
|
||||||
|
go c.dispatchLoop()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Close stops the dispatcher and fails everything still queued.
|
||||||
|
func (c *Controller) Close() {
|
||||||
|
c.once.Do(func() {
|
||||||
|
close(c.stop)
|
||||||
|
<-c.done
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Lease is held for the lifetime of an admitted request.
|
||||||
|
type Lease struct {
|
||||||
|
c *Controller
|
||||||
|
band string
|
||||||
|
model string
|
||||||
|
once sync.Once
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release returns the slot and wakes the dispatcher, since finishing a request
|
||||||
|
// is the most likely reason saturation just dropped.
|
||||||
|
func (l *Lease) Release() {
|
||||||
|
if l == nil || l.c == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
l.once.Do(func() {
|
||||||
|
l.c.mu.Lock()
|
||||||
|
if n := l.c.outstanding[l.model]; n > 1 {
|
||||||
|
l.c.outstanding[l.model] = n - 1
|
||||||
|
} else {
|
||||||
|
delete(l.c.outstanding, l.model)
|
||||||
|
}
|
||||||
|
l.c.mu.Unlock()
|
||||||
|
l.c.signal()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// Band reports which band admitted the request.
|
||||||
|
func (l *Lease) Band() string {
|
||||||
|
if l == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return l.band
|
||||||
|
}
|
||||||
|
|
||||||
|
// Acquire blocks until the request may proceed, the context ends, or the TTL
|
||||||
|
// expires.
|
||||||
|
func (c *Controller) Acquire(ctx context.Context, r Request) (*Lease, error) {
|
||||||
|
idx, ok := c.bandOf[r.Band]
|
||||||
|
if !ok {
|
||||||
|
idx = c.bandOf[c.cfg.DefaultBand]
|
||||||
|
}
|
||||||
|
band := c.cfg.Bands[idx]
|
||||||
|
|
||||||
|
// Fast path: under the ceiling with nobody already waiting in this band.
|
||||||
|
// Skipping the queue here is what keeps the common case free of latency.
|
||||||
|
c.mu.Lock()
|
||||||
|
if c.closed {
|
||||||
|
c.mu.Unlock()
|
||||||
|
return nil, ErrClosed
|
||||||
|
}
|
||||||
|
if c.queues[idx].len() == 0 && c.admits(band, r.Model) {
|
||||||
|
c.grant(band.Name, r.Model)
|
||||||
|
c.mu.Unlock()
|
||||||
|
return &Lease{c: c, band: band.Name, model: r.Model}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
if c.queues[idx].len() >= c.cfg.QueueCapacityPerBand {
|
||||||
|
c.rejected[band.Name]++
|
||||||
|
c.mu.Unlock()
|
||||||
|
return nil, fmt.Errorf("%w: band %q", ErrQueueFull, band.Name)
|
||||||
|
}
|
||||||
|
|
||||||
|
w := &waiter{
|
||||||
|
tenant: tenantKey(r),
|
||||||
|
model: r.Model,
|
||||||
|
ready: make(chan struct{}),
|
||||||
|
enqueued: time.Now(),
|
||||||
|
deadline: time.Now().Add(c.cfg.QueueTTL),
|
||||||
|
}
|
||||||
|
c.queues[idx].push(w)
|
||||||
|
c.mu.Unlock()
|
||||||
|
|
||||||
|
c.signal()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-w.ready:
|
||||||
|
c.mu.Lock()
|
||||||
|
if c.closed {
|
||||||
|
c.mu.Unlock()
|
||||||
|
return nil, ErrClosed
|
||||||
|
}
|
||||||
|
c.waitTotal[band.Name] += time.Since(w.enqueued)
|
||||||
|
c.mu.Unlock()
|
||||||
|
return &Lease{c: c, band: band.Name, model: r.Model}, nil
|
||||||
|
|
||||||
|
case <-ctx.Done():
|
||||||
|
c.remove(idx, w, "client_gone")
|
||||||
|
return nil, ctx.Err()
|
||||||
|
|
||||||
|
case <-time.After(time.Until(w.deadline)):
|
||||||
|
c.remove(idx, w, "ttl")
|
||||||
|
return nil, fmt.Errorf("%w after %v", ErrQueueTimeout, c.cfg.QueueTTL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// remove takes a waiter out of its queue unless it was admitted first.
|
||||||
|
func (c *Controller) remove(idx int, w *waiter, reason string) {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
if w.admitted {
|
||||||
|
// It won the race; the caller will see ready closed next time round.
|
||||||
|
return
|
||||||
|
}
|
||||||
|
c.queues[idx].remove(w)
|
||||||
|
c.rejected[c.cfg.Bands[idx].Name+"/"+reason]++
|
||||||
|
}
|
||||||
|
|
||||||
|
// grant records an admission. The caller holds the lock.
|
||||||
|
func (c *Controller) grant(band, model string) {
|
||||||
|
c.admitted[band]++
|
||||||
|
c.outstanding[model]++
|
||||||
|
}
|
||||||
|
|
||||||
|
// admits reports whether a band may proceed right now. The caller holds the lock.
|
||||||
|
func (c *Controller) admits(b Band, model string) bool {
|
||||||
|
sat, known := c.satFn(model, c.outstanding[model])
|
||||||
|
if !known {
|
||||||
|
// Nothing is known about this model's pool; let it through so dispatch
|
||||||
|
// can return a precise error instead of a queue timeout.
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return sat <= b.SaturationCeiling
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Controller) signal() {
|
||||||
|
select {
|
||||||
|
case c.wake <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Controller) dispatchLoop() {
|
||||||
|
defer close(c.done)
|
||||||
|
t := time.NewTicker(c.cfg.Interval)
|
||||||
|
defer t.Stop()
|
||||||
|
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-c.stop:
|
||||||
|
c.drain()
|
||||||
|
return
|
||||||
|
case <-c.wake:
|
||||||
|
c.dispatch()
|
||||||
|
case <-t.C:
|
||||||
|
c.dispatch()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// dispatch walks the bands from highest priority down, admitting whatever the
|
||||||
|
// current saturation allows.
|
||||||
|
func (c *Controller) dispatch() {
|
||||||
|
now := time.Now()
|
||||||
|
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
|
||||||
|
for i, q := range c.queues {
|
||||||
|
q.dropExpired(now, func() { c.rejected[c.cfg.Bands[i].Name+"/ttl"]++ })
|
||||||
|
|
||||||
|
band := c.cfg.Bands[i]
|
||||||
|
// Each pass admits at most the queue length, so a band cannot spin
|
||||||
|
// forever while saturation is recomputed from stale metrics.
|
||||||
|
for range q.len() {
|
||||||
|
w := q.peek()
|
||||||
|
if w == nil || !c.admits(band, w.model) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
q.pop()
|
||||||
|
w.admitted = true
|
||||||
|
c.grant(band.Name, w.model)
|
||||||
|
close(w.ready)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Controller) drain() {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
c.closed = true
|
||||||
|
for _, q := range c.queues {
|
||||||
|
for {
|
||||||
|
w := q.pop()
|
||||||
|
if w == nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
// Closing without marking admitted lets Acquire report ErrClosed.
|
||||||
|
close(w.ready)
|
||||||
|
w.admitted = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Depths returns each band's current queue depth.
|
||||||
|
func (c *Controller) Depths() []Stats {
|
||||||
|
c.mu.Lock()
|
||||||
|
defer c.mu.Unlock()
|
||||||
|
out := make([]Stats, 0, len(c.queues))
|
||||||
|
for i, q := range c.queues {
|
||||||
|
out = append(out, Stats{Band: c.cfg.Bands[i].Name, Depth: q.len()})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// tenantKey picks the fairness key, falling back to the band so that unknown
|
||||||
|
// tenants at least compete as one group rather than each looking unique.
|
||||||
|
func tenantKey(r Request) string {
|
||||||
|
if r.Tenant != "" {
|
||||||
|
return r.Tenant
|
||||||
|
}
|
||||||
|
return "anonymous"
|
||||||
|
}
|
||||||
@@ -0,0 +1,168 @@
|
|||||||
|
package admission
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
// waiter is one queued request.
|
||||||
|
type waiter struct {
|
||||||
|
tenant string
|
||||||
|
model string
|
||||||
|
ready chan struct{}
|
||||||
|
enqueued time.Time
|
||||||
|
deadline time.Time
|
||||||
|
// admitted guards the race between the dispatcher admitting a waiter and
|
||||||
|
// the waiter giving up on its own.
|
||||||
|
admitted bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// bandQueue holds one band's waiters, grouped by tenant.
|
||||||
|
//
|
||||||
|
// Grouping is what makes fairness possible: a single tenant sending a thousand
|
||||||
|
// requests occupies one slot in the rotation, not a thousand, so a traffic
|
||||||
|
// spike from one caller cannot consume the band.
|
||||||
|
type bandQueue struct {
|
||||||
|
band Band
|
||||||
|
|
||||||
|
byTenant map[string][]*waiter
|
||||||
|
// rotation is the round-robin order of tenants with pending work.
|
||||||
|
rotation []string
|
||||||
|
next int
|
||||||
|
count int
|
||||||
|
}
|
||||||
|
|
||||||
|
func newBandQueue(b Band) *bandQueue {
|
||||||
|
return &bandQueue{band: b, byTenant: map[string][]*waiter{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (q *bandQueue) len() int { return q.count }
|
||||||
|
|
||||||
|
func (q *bandQueue) push(w *waiter) {
|
||||||
|
if _, ok := q.byTenant[w.tenant]; !ok {
|
||||||
|
q.byTenant[w.tenant] = nil
|
||||||
|
q.rotation = append(q.rotation, w.tenant)
|
||||||
|
}
|
||||||
|
q.byTenant[w.tenant] = append(q.byTenant[w.tenant], w)
|
||||||
|
q.count++
|
||||||
|
}
|
||||||
|
|
||||||
|
// peek returns the waiter the next pop would return, without removing it.
|
||||||
|
func (q *bandQueue) peek() *waiter {
|
||||||
|
if q.count == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
idx, tenant := q.nextTenant()
|
||||||
|
if idx < 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
_ = tenant
|
||||||
|
return q.byTenant[q.rotation[idx]][0]
|
||||||
|
}
|
||||||
|
|
||||||
|
// pop removes and returns the next waiter in tenant rotation.
|
||||||
|
func (q *bandQueue) pop() *waiter {
|
||||||
|
if q.count == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
idx, _ := q.nextTenant()
|
||||||
|
if idx < 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
tenant := q.rotation[idx]
|
||||||
|
ws := q.byTenant[tenant]
|
||||||
|
w := ws[0]
|
||||||
|
q.byTenant[tenant] = ws[1:]
|
||||||
|
q.count--
|
||||||
|
|
||||||
|
if len(q.byTenant[tenant]) == 0 {
|
||||||
|
delete(q.byTenant, tenant)
|
||||||
|
q.rotation = append(q.rotation[:idx], q.rotation[idx+1:]...)
|
||||||
|
if idx < q.next {
|
||||||
|
q.next--
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
q.next = idx + 1
|
||||||
|
}
|
||||||
|
if len(q.rotation) > 0 {
|
||||||
|
q.next %= len(q.rotation)
|
||||||
|
} else {
|
||||||
|
q.next = 0
|
||||||
|
}
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
// nextTenant finds the next rotation entry that still has waiters.
|
||||||
|
func (q *bandQueue) nextTenant() (int, string) {
|
||||||
|
n := len(q.rotation)
|
||||||
|
for i := range n {
|
||||||
|
idx := (q.next + i) % n
|
||||||
|
if len(q.byTenant[q.rotation[idx]]) > 0 {
|
||||||
|
return idx, q.rotation[idx]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return -1, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// remove takes a specific waiter out, for cancellation and TTL expiry.
|
||||||
|
func (q *bandQueue) remove(target *waiter) bool {
|
||||||
|
ws, ok := q.byTenant[target.tenant]
|
||||||
|
if !ok {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for i, w := range ws {
|
||||||
|
if w != target {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
q.byTenant[target.tenant] = append(ws[:i:i], ws[i+1:]...)
|
||||||
|
q.count--
|
||||||
|
if len(q.byTenant[target.tenant]) == 0 {
|
||||||
|
delete(q.byTenant, target.tenant)
|
||||||
|
q.dropTenant(target.tenant)
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func (q *bandQueue) dropTenant(tenant string) {
|
||||||
|
for i, t := range q.rotation {
|
||||||
|
if t != tenant {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
q.rotation = append(q.rotation[:i], q.rotation[i+1:]...)
|
||||||
|
if i < q.next {
|
||||||
|
q.next--
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if len(q.rotation) > 0 {
|
||||||
|
q.next %= len(q.rotation)
|
||||||
|
} else {
|
||||||
|
q.next = 0
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// dropExpired removes waiters past their deadline, calling onDrop for each.
|
||||||
|
// Their Acquire call is already unblocking on its own timer; this keeps the
|
||||||
|
// queue from carrying corpses that would occupy capacity.
|
||||||
|
func (q *bandQueue) dropExpired(now time.Time, onDrop func()) {
|
||||||
|
if q.count == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for tenant, ws := range q.byTenant {
|
||||||
|
kept := ws[:0]
|
||||||
|
for _, w := range ws {
|
||||||
|
if now.After(w.deadline) && !w.admitted {
|
||||||
|
q.count--
|
||||||
|
onDrop()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
kept = append(kept, w)
|
||||||
|
}
|
||||||
|
if len(kept) == 0 {
|
||||||
|
delete(q.byTenant, tenant)
|
||||||
|
q.dropTenant(tenant)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
q.byTenant[tenant] = kept
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
// Package admission answers "may this request proceed now?" (plan §7.2).
|
||||||
|
//
|
||||||
|
// It is deliberately separate from scheduling, which answers "where should it
|
||||||
|
// go?". Kept apart, the router can hold requests in its own queue and apply
|
||||||
|
// priority and tenant fairness *before* they enter an engine queue. An engine
|
||||||
|
// queue is FIFO and knows nothing about tenants: once a batch job has put 500
|
||||||
|
// requests into it, no amount of clever endpoint selection will rescue the
|
||||||
|
// interactive request that arrives next.
|
||||||
|
package admission
|
||||||
|
|
||||||
|
import (
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Thresholds define what "saturated" means for a pool.
|
||||||
|
type Thresholds struct {
|
||||||
|
// TargetWaitingPerReplica is the healthy queue depth: enough for continuous
|
||||||
|
// batching to stay efficient, not enough for queueing delay to show up.
|
||||||
|
// Measure it with ctl bench (plan §10); this single number governs how the
|
||||||
|
// whole system behaves under load.
|
||||||
|
TargetWaitingPerReplica float64
|
||||||
|
// KVCeiling is the KV occupancy treated as full.
|
||||||
|
KVCeiling float64
|
||||||
|
}
|
||||||
|
|
||||||
|
func (t *Thresholds) normalize() {
|
||||||
|
if t.TargetWaitingPerReplica <= 0 {
|
||||||
|
t.TargetWaitingPerReplica = 4
|
||||||
|
}
|
||||||
|
if t.KVCeiling <= 0 || t.KVCeiling > 1 {
|
||||||
|
t.KVCeiling = 0.9
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Saturation reduces a pool's state to a single number, where 1 means "at the
|
||||||
|
// healthy limit" and above 1 means overloaded.
|
||||||
|
//
|
||||||
|
// Two signals are combined because either alone misleads: a pool can have an
|
||||||
|
// empty queue while its KV cache is nearly full (long contexts), or plenty of
|
||||||
|
// KV headroom while requests pile up (slow decode). The worse of the two wins.
|
||||||
|
//
|
||||||
|
// unreported is load that has been admitted but that no endpoint has accounted
|
||||||
|
// for yet. Without it a simultaneous burst all passes the check before any of
|
||||||
|
// it becomes visible — which is precisely when admission needs to bite.
|
||||||
|
func Saturation(eps []endpoint.Snapshot, unreported int, t Thresholds) float64 {
|
||||||
|
t.normalize()
|
||||||
|
|
||||||
|
var sum float64
|
||||||
|
var n int
|
||||||
|
for _, e := range eps {
|
||||||
|
// A replica the breaker has taken out is not available capacity, so
|
||||||
|
// counting it would make the pool look healthier than it is.
|
||||||
|
if e.Breaker.String() == "open" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// The engine's own queue is the truth, but it is one scrape interval
|
||||||
|
// old. Fill that gap with what the router knows without any lag: how
|
||||||
|
// many requests it has dispatched here, minus what this engine can run
|
||||||
|
// at once. Anything beyond that capacity is queued by definition.
|
||||||
|
//
|
||||||
|
// Subtracting the capacity is the whole point. Counting raw inflight
|
||||||
|
// would make a fully busy but perfectly healthy engine look like it had
|
||||||
|
// a queue as long as its batch, every single scrape interval.
|
||||||
|
waiting := float64(e.Stats.Waiting)
|
||||||
|
if e.RunningCapacity > 0 {
|
||||||
|
if queued := e.Inflight - e.RunningCapacity; float64(queued) > waiting {
|
||||||
|
waiting = float64(queued)
|
||||||
|
}
|
||||||
|
} else if e.Inflight > 0 {
|
||||||
|
// Nothing observed yet, so capacity is unknown. Treat dispatched
|
||||||
|
// work as queued: conservative for the first scrape, and it relaxes
|
||||||
|
// as soon as the engine reports anything.
|
||||||
|
waiting = max(waiting, float64(e.Inflight))
|
||||||
|
}
|
||||||
|
|
||||||
|
q := waiting / t.TargetWaitingPerReplica
|
||||||
|
kv := e.Stats.KVCacheUsage / t.KVCeiling
|
||||||
|
sum += max(q, kv)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
_ = n
|
||||||
|
|
||||||
|
if n == 0 {
|
||||||
|
// Every replica is out. Report full saturation so low bands stop
|
||||||
|
// queueing work that has nowhere to go; the dispatch layer will return
|
||||||
|
// a clearer error than a queue timeout.
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
// Spread unaccounted admissions evenly; they will land somewhere.
|
||||||
|
if unreported > 0 {
|
||||||
|
sum += float64(unreported) / t.TargetWaitingPerReplica
|
||||||
|
}
|
||||||
|
|
||||||
|
// The mean, not the minimum: one idle replica does not make a busy pool
|
||||||
|
// idle, and admission is about the pool's capacity as a whole.
|
||||||
|
return sum / float64(n)
|
||||||
|
}
|
||||||
@@ -71,6 +71,14 @@ type Snapshot struct {
|
|||||||
// unavoidable within a 2s scrape interval.
|
// unavoidable within a 2s scrape interval.
|
||||||
Inflight int
|
Inflight int
|
||||||
|
|
||||||
|
// RunningCapacity is the highest concurrent request count ever observed on
|
||||||
|
// this endpoint — in practice the engine's batch capacity, learned rather
|
||||||
|
// than configured. Without it, requests the router has dispatched but the
|
||||||
|
// engine has not yet reported are indistinguishable from queue depth, and
|
||||||
|
// during a scrape interval that makes a perfectly healthy engine look
|
||||||
|
// overloaded.
|
||||||
|
RunningCapacity int
|
||||||
|
|
||||||
// Stale means no successful scrape arrived within StaleAfter. Stale
|
// Stale means no successful scrape arrived within StaleAfter. Stale
|
||||||
// endpoints are not dropped, only scored down.
|
// endpoints are not dropped, only scored down.
|
||||||
Stale bool
|
Stale bool
|
||||||
@@ -101,10 +109,11 @@ func DefaultConfig() Config {
|
|||||||
type live struct {
|
type live struct {
|
||||||
info Info
|
info Info
|
||||||
|
|
||||||
stats Stats
|
stats Stats
|
||||||
scrapedAt time.Time
|
scrapedAt time.Time
|
||||||
lastSucc float64
|
lastSucc float64
|
||||||
drainRate float64
|
drainRate float64
|
||||||
|
maxRunning int
|
||||||
|
|
||||||
inflight int
|
inflight int
|
||||||
brk *breaker.Breaker
|
brk *breaker.Breaker
|
||||||
@@ -199,6 +208,7 @@ func (r *Registry) Observe(id string, s Stats) {
|
|||||||
l.stats = s
|
l.stats = s
|
||||||
l.lastSucc = s.SuccessTotal
|
l.lastSucc = s.SuccessTotal
|
||||||
l.scrapedAt = now
|
l.scrapedAt = now
|
||||||
|
l.maxRunning = max(l.maxRunning, s.Running)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Snapshots returns a view of every endpoint serving the given model, or of all
|
// Snapshots returns a view of every endpoint serving the given model, or of all
|
||||||
@@ -220,13 +230,14 @@ func (r *Registry) Snapshots(model string) []Snapshot {
|
|||||||
stale = age > r.cfg.StaleAfter
|
stale = age > r.cfg.StaleAfter
|
||||||
}
|
}
|
||||||
out = append(out, Snapshot{
|
out = append(out, Snapshot{
|
||||||
Info: l.info,
|
Info: l.info,
|
||||||
Stats: l.stats,
|
Stats: l.stats,
|
||||||
DrainRate: l.drainRate,
|
DrainRate: l.drainRate,
|
||||||
Inflight: l.inflight,
|
Inflight: l.inflight,
|
||||||
Stale: stale,
|
RunningCapacity: l.maxRunning,
|
||||||
Age: age,
|
Stale: stale,
|
||||||
Breaker: l.brk.State(),
|
Age: age,
|
||||||
|
Breaker: l.brk.State(),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
slices.SortFunc(out, func(a, b Snapshot) int { return strcmp(a.ID, b.ID) })
|
slices.SortFunc(out, func(a, b Snapshot) int { return strcmp(a.ID, b.ID) })
|
||||||
|
|||||||
@@ -0,0 +1,361 @@
|
|||||||
|
package router
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"slices"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||||
|
)
|
||||||
|
|
||||||
|
// admit waits for the flow control layer to let the request through. With
|
||||||
|
// admission disabled it returns immediately, which is the v0.2.0 behaviour.
|
||||||
|
func (r *Router) admit(ctx context.Context, req *parsedRequest) (*admission.Lease, error) {
|
||||||
|
if r.adm == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return r.adm.Acquire(ctx, admission.Request{
|
||||||
|
Model: req.Model,
|
||||||
|
Band: req.Band,
|
||||||
|
// Until authentication arrives the session is the best tenant key we
|
||||||
|
// have: it groups one caller's traffic without pretending to identify
|
||||||
|
// them (plan §7.9).
|
||||||
|
Tenant: req.SessionID,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// rejectAdmission turns a flow control refusal into a status the client can act
|
||||||
|
// on: 503 when the queue is full (try later), 504 when it timed out waiting.
|
||||||
|
func (r *Router) rejectAdmission(w http.ResponseWriter, req *parsedRequest, err error, start time.Time) {
|
||||||
|
status := http.StatusServiceUnavailable
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
|
||||||
|
// The client gave up while queued; nobody is listening.
|
||||||
|
r.m.observeAdmissionRejected(req.Band, "client_gone")
|
||||||
|
r.m.observeRequest(req.Model, "", 499, time.Since(start), nil, 0)
|
||||||
|
return
|
||||||
|
case errors.Is(err, admission.ErrQueueTimeout):
|
||||||
|
status = http.StatusGatewayTimeout
|
||||||
|
}
|
||||||
|
reason := "unavailable"
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, admission.ErrQueueFull):
|
||||||
|
reason = "queue_full"
|
||||||
|
case errors.Is(err, admission.ErrQueueTimeout):
|
||||||
|
reason = "ttl"
|
||||||
|
case errors.Is(err, admission.ErrClosed):
|
||||||
|
reason = "shutdown"
|
||||||
|
}
|
||||||
|
r.m.observeAdmissionRejected(req.Band, reason)
|
||||||
|
writeError(w, status, "server_error", err.Error())
|
||||||
|
r.m.observeRequest(req.Model, "", status, time.Since(start), nil, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
// prefixBytes is how much of the prompt is kept for prefix-aware routing.
|
||||||
|
// Large enough to cover a system prompt plus the first message, small enough
|
||||||
|
// that hashing it stays cheap.
|
||||||
|
const prefixBytes = 2048
|
||||||
|
|
||||||
|
// Handler returns the client-facing OpenAI API.
|
||||||
|
func (r *Router) Handler() http.Handler {
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("POST /v1/chat/completions", r.handleInference("/v1/chat/completions"))
|
||||||
|
mux.HandleFunc("POST /v1/completions", r.handleInference("/v1/completions"))
|
||||||
|
mux.HandleFunc("POST /v1/embeddings", r.handleInference("/v1/embeddings"))
|
||||||
|
mux.HandleFunc("GET /v1/models", r.handleModels)
|
||||||
|
mux.HandleFunc("GET /health", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
})
|
||||||
|
return mux
|
||||||
|
}
|
||||||
|
|
||||||
|
// parsedRequest is what the router needs from a request. Everything else in the
|
||||||
|
// body is forwarded untouched — clients send engine-specific fields we have no
|
||||||
|
// business interpreting.
|
||||||
|
type parsedRequest struct {
|
||||||
|
Path string
|
||||||
|
Model string
|
||||||
|
|
||||||
|
Stream bool
|
||||||
|
SuppressUsageChunk bool
|
||||||
|
UpstreamBody []byte
|
||||||
|
Header http.Header
|
||||||
|
|
||||||
|
SessionID string
|
||||||
|
Band string
|
||||||
|
ProfileOverride string
|
||||||
|
PromptPrefix string
|
||||||
|
PromptTokens int
|
||||||
|
MaxTokens int
|
||||||
|
}
|
||||||
|
|
||||||
|
// inferenceBody covers the fields the router reads across all inference
|
||||||
|
// endpoints. Unknown fields are ignored here and preserved in UpstreamBody.
|
||||||
|
type inferenceBody struct {
|
||||||
|
Model string `json:"model"`
|
||||||
|
Stream bool `json:"stream"`
|
||||||
|
User string `json:"user"`
|
||||||
|
Messages []struct {
|
||||||
|
Role string `json:"role"`
|
||||||
|
Content json.RawMessage `json:"content"`
|
||||||
|
} `json:"messages"`
|
||||||
|
Prompt json.RawMessage `json:"prompt"`
|
||||||
|
Input json.RawMessage `json:"input"`
|
||||||
|
MaxTokens int `json:"max_tokens"`
|
||||||
|
MaxCompletionTokens int `json:"max_completion_tokens"`
|
||||||
|
StreamOptions *struct {
|
||||||
|
IncludeUsage bool `json:"include_usage"`
|
||||||
|
} `json:"stream_options"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Router) handleInference(path string) http.HandlerFunc {
|
||||||
|
return func(w http.ResponseWriter, req *http.Request) {
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
parsed, err := r.parse(req, path)
|
||||||
|
if err != nil {
|
||||||
|
writeError(w, http.StatusBadRequest, "invalid_request_error", err.Error())
|
||||||
|
r.m.observeRequest("", "", http.StatusBadRequest, time.Since(start), nil, 0)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 5 of the lifecycle: "now?" before "where?" (plan §7.1).
|
||||||
|
lease, err := r.admit(req.Context(), parsed)
|
||||||
|
if err != nil {
|
||||||
|
r.rejectAdmission(w, parsed, err, start)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer lease.Release()
|
||||||
|
queued := time.Since(start)
|
||||||
|
r.m.observeAdmission(lease.Band(), parsed.Model, queued)
|
||||||
|
if queued > time.Millisecond {
|
||||||
|
w.Header().Set("x-llmgw-queue-ms", strconv.FormatInt(queued.Milliseconds(), 10))
|
||||||
|
}
|
||||||
|
|
||||||
|
res := r.dispatch(req.Context(), w, parsed)
|
||||||
|
r.finish(w, parsed, res, start)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// finish records metrics and, when nothing was written, produces the error body.
|
||||||
|
func (r *Router) finish(w http.ResponseWriter, req *parsedRequest, res dispatchResult, start time.Time) {
|
||||||
|
status := res.outcome.Status
|
||||||
|
switch {
|
||||||
|
case res.err == nil:
|
||||||
|
if res.fallback {
|
||||||
|
// The client asked for one model and got another; say so rather
|
||||||
|
// than letting them discover it from the response contents.
|
||||||
|
w.Header().Set("x-llmgw-served-model", res.model)
|
||||||
|
}
|
||||||
|
case errors.Is(res.err, proxy.ErrClientGone):
|
||||||
|
// Nothing to report to a client that is gone; the upstream was cancelled.
|
||||||
|
status = 499
|
||||||
|
r.log.Debug("client disconnected", "model", req.Model, "endpoint", res.endpointID)
|
||||||
|
case res.outcome.Wrote:
|
||||||
|
// The response is already partly out; the stream just ends short.
|
||||||
|
r.log.Warn("stream ended early", "model", req.Model, "endpoint", res.endpointID, "err", res.err)
|
||||||
|
if status == 0 {
|
||||||
|
status = http.StatusOK
|
||||||
|
}
|
||||||
|
case errors.Is(res.err, errNoCapacity):
|
||||||
|
status = http.StatusServiceUnavailable
|
||||||
|
writeError(w, status, "server_error", res.err.Error())
|
||||||
|
default:
|
||||||
|
status = http.StatusBadGateway
|
||||||
|
writeError(w, status, "server_error", res.err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
r.m.observeRequest(req.Model, res.pool, status, time.Since(start), res.outcome.Usage, res.outcome.TTFT)
|
||||||
|
}
|
||||||
|
|
||||||
|
// parse reads the body and extracts the routing-relevant fields.
|
||||||
|
func (r *Router) parse(req *http.Request, path string) (*parsedRequest, error) {
|
||||||
|
body, err := io.ReadAll(io.LimitReader(req.Body, r.cfg.MaxBodyBytes))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var b inferenceBody
|
||||||
|
if err := json.Unmarshal(body, &b); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if b.Model == "" {
|
||||||
|
return nil, errors.New("model is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
p := &parsedRequest{
|
||||||
|
Path: path,
|
||||||
|
Model: b.Model,
|
||||||
|
Stream: b.Stream,
|
||||||
|
UpstreamBody: body,
|
||||||
|
Header: req.Header,
|
||||||
|
SessionID: sessionID(req, r.cfg.SessionHeader, b.User),
|
||||||
|
Band: req.Header.Get(r.cfg.Admission.BandHeader),
|
||||||
|
ProfileOverride: req.Header.Get("x-llmgw-profile"),
|
||||||
|
MaxTokens: max(b.MaxTokens, b.MaxCompletionTokens),
|
||||||
|
PromptPrefix: promptPrefix(&b),
|
||||||
|
}
|
||||||
|
// Bytes over four: a crude stand-in for a tokenizer, good enough for
|
||||||
|
// scoring. It becomes exact only with plan §7.5 tier 2 (see plan §14).
|
||||||
|
p.PromptTokens = len(p.PromptPrefix) / 4
|
||||||
|
|
||||||
|
// Ask the engine for usage the client did not request, so accounting works
|
||||||
|
// regardless of client behaviour; the extra chunk is dropped on the way back.
|
||||||
|
if r.cfg.InjectUsage && b.Stream && (b.StreamOptions == nil || !b.StreamOptions.IncludeUsage) {
|
||||||
|
if injected, ok := injectIncludeUsage(body); ok {
|
||||||
|
p.UpstreamBody = injected
|
||||||
|
p.SuppressUsageChunk = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return p, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// sessionID picks the affinity key: an explicit header first, then OpenAI's
|
||||||
|
// user field, which most clients already set per end user.
|
||||||
|
func sessionID(req *http.Request, header, user string) string {
|
||||||
|
if header != "" {
|
||||||
|
if v := req.Header.Get(header); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return user
|
||||||
|
}
|
||||||
|
|
||||||
|
// promptPrefix reconstructs the leading portion of the prompt for prefix-aware
|
||||||
|
// routing. Only the first prefixBytes matter, so long conversations stop early.
|
||||||
|
func promptPrefix(b *inferenceBody) string {
|
||||||
|
var sb strings.Builder
|
||||||
|
appendRaw := func(raw json.RawMessage) {
|
||||||
|
if len(raw) == 0 || sb.Len() >= prefixBytes {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var s string
|
||||||
|
if err := json.Unmarshal(raw, &s); err != nil {
|
||||||
|
s = string(raw)
|
||||||
|
}
|
||||||
|
sb.WriteString(s)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, m := range b.Messages {
|
||||||
|
if sb.Len() >= prefixBytes {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
sb.WriteString(m.Role)
|
||||||
|
sb.WriteString(":")
|
||||||
|
appendRaw(m.Content)
|
||||||
|
sb.WriteString("\n")
|
||||||
|
}
|
||||||
|
appendRaw(b.Prompt)
|
||||||
|
appendRaw(b.Input)
|
||||||
|
|
||||||
|
s := sb.String()
|
||||||
|
if len(s) > prefixBytes {
|
||||||
|
s = s[:prefixBytes]
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
// injectIncludeUsage sets stream_options.include_usage on the outgoing body.
|
||||||
|
//
|
||||||
|
// This is the one place the router rewrites a request, so it is done by
|
||||||
|
// decoding to a generic map and re-encoding: field order changes, but no field
|
||||||
|
// is lost. If anything looks unexpected it declines rather than guessing.
|
||||||
|
func injectIncludeUsage(body []byte) ([]byte, bool) {
|
||||||
|
var m map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(body, &m); err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
|
||||||
|
opts := map[string]any{}
|
||||||
|
if raw, ok := m["stream_options"]; ok {
|
||||||
|
if err := json.Unmarshal(raw, &opts); err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
opts["include_usage"] = true
|
||||||
|
|
||||||
|
encoded, err := json.Marshal(opts)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
m["stream_options"] = encoded
|
||||||
|
|
||||||
|
out, err := json.Marshal(m)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
return out, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// servedModels lists the canonical models at least one endpoint serves.
|
||||||
|
func (r *Router) servedModels() []string {
|
||||||
|
seen := map[string]struct{}{}
|
||||||
|
for _, s := range r.reg.Snapshots("") {
|
||||||
|
for _, m := range s.Models {
|
||||||
|
seen[m] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out := make([]string, 0, len(seen))
|
||||||
|
for m := range seen {
|
||||||
|
out = append(out, m)
|
||||||
|
}
|
||||||
|
slices.Sort(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// handleModels lists the models that at least one endpoint currently serves.
|
||||||
|
func (r *Router) handleModels(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
seen := map[string]struct{}{}
|
||||||
|
for _, s := range r.reg.Snapshots("") {
|
||||||
|
for _, m := range s.Models {
|
||||||
|
seen[m] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Catalog aliases are usable names too, so clients should see them.
|
||||||
|
for name := range r.models {
|
||||||
|
if _, ok := seen[r.models[name].Name]; ok {
|
||||||
|
seen[name] = struct{}{}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
names := make([]string, 0, len(seen))
|
||||||
|
for n := range seen {
|
||||||
|
names = append(names, n)
|
||||||
|
}
|
||||||
|
slices.Sort(names)
|
||||||
|
|
||||||
|
data := make([]map[string]any, 0, len(names))
|
||||||
|
for _, n := range names {
|
||||||
|
data = append(data, map[string]any{
|
||||||
|
"id": n, "object": "model", "created": time.Now().Unix(), "owned_by": "llmgw",
|
||||||
|
})
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{"object": "list", "data": data})
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeJSON(w http.ResponseWriter, code int, v any) {
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := json.NewEncoder(&buf).Encode(v); err != nil {
|
||||||
|
http.Error(w, `{"error":{"message":"encoding failed"}}`, http.StatusInternalServerError)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(code)
|
||||||
|
_, _ = w.Write(buf.Bytes())
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeError emits an OpenAI-shaped error so clients need no special handling
|
||||||
|
// for gateway failures.
|
||||||
|
func writeError(w http.ResponseWriter, code int, kind, msg string) {
|
||||||
|
writeJSON(w, code, map[string]any{"error": map[string]any{
|
||||||
|
"message": msg, "type": kind, "code": nil,
|
||||||
|
}})
|
||||||
|
}
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
package router
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/prometheus/client_golang/prometheus"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/scheduler"
|
||||||
|
)
|
||||||
|
|
||||||
|
// metrics holds the router's own Prometheus series (plan §8).
|
||||||
|
//
|
||||||
|
// The scorer series matter more than they look: routing profile weights cannot
|
||||||
|
// be tuned without seeing what each scorer actually produced in production.
|
||||||
|
type metrics struct {
|
||||||
|
requests *prometheus.CounterVec
|
||||||
|
duration *prometheus.HistogramVec
|
||||||
|
ttft *prometheus.HistogramVec
|
||||||
|
tokens *prometheus.CounterVec
|
||||||
|
|
||||||
|
scorerValue *prometheus.GaugeVec
|
||||||
|
filtered *prometheus.CounterVec
|
||||||
|
considered prometheus.Histogram
|
||||||
|
|
||||||
|
admissionWait *prometheus.HistogramVec
|
||||||
|
admissionRejected *prometheus.CounterVec
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMetrics() *metrics {
|
||||||
|
return &metrics{
|
||||||
|
requests: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||||
|
Name: "llmgw_requests_total",
|
||||||
|
Help: "Requests handled, by model, pool and response status.",
|
||||||
|
}, []string{"model", "pool", "status"}),
|
||||||
|
|
||||||
|
duration: prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||||
|
Name: "llmgw_request_duration_seconds",
|
||||||
|
Help: "End-to-end request duration.",
|
||||||
|
Buckets: []float64{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300},
|
||||||
|
}, []string{"model", "pool"}),
|
||||||
|
|
||||||
|
ttft: prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||||
|
Name: "llmgw_ttft_seconds",
|
||||||
|
Help: "Time to first token, as observed by the router.",
|
||||||
|
Buckets: []float64{0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60},
|
||||||
|
}, []string{"model", "pool"}),
|
||||||
|
|
||||||
|
tokens: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||||
|
Name: "llmgw_tokens_total",
|
||||||
|
Help: "Tokens accounted for, by direction.",
|
||||||
|
}, []string{"model", "direction"}),
|
||||||
|
|
||||||
|
scorerValue: prometheus.NewGaugeVec(prometheus.GaugeOpts{
|
||||||
|
Name: "llmgw_scorer_value",
|
||||||
|
Help: "Last score each scorer produced for the endpoint that was chosen.",
|
||||||
|
}, []string{"scorer", "replica"}),
|
||||||
|
|
||||||
|
filtered: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||||
|
Name: "llmgw_candidates_filtered_total",
|
||||||
|
Help: "Candidate endpoints removed, by filter.",
|
||||||
|
}, []string{"filter"}),
|
||||||
|
|
||||||
|
considered: prometheus.NewHistogram(prometheus.HistogramOpts{
|
||||||
|
Name: "llmgw_candidates_considered",
|
||||||
|
Help: "Candidates remaining after filtering, per scheduling decision.",
|
||||||
|
Buckets: []float64{1, 2, 3, 5, 8, 16, 32, 64},
|
||||||
|
}),
|
||||||
|
|
||||||
|
admissionWait: prometheus.NewHistogramVec(prometheus.HistogramOpts{
|
||||||
|
Name: "llmgw_admission_wait_seconds",
|
||||||
|
Help: "Time spent waiting in the admission queue.",
|
||||||
|
Buckets: []float64{0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2.5, 5, 10, 30},
|
||||||
|
}, []string{"band", "model"}),
|
||||||
|
|
||||||
|
admissionRejected: prometheus.NewCounterVec(prometheus.CounterOpts{
|
||||||
|
Name: "llmgw_admission_rejected_total",
|
||||||
|
Help: "Requests refused by flow control, by reason.",
|
||||||
|
}, []string{"band", "reason"}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// collectors returns everything to register, so registration lives in one place.
|
||||||
|
func (m *metrics) collectors() []prometheus.Collector {
|
||||||
|
return []prometheus.Collector{
|
||||||
|
m.requests, m.duration, m.ttft, m.tokens,
|
||||||
|
m.scorerValue, m.filtered, m.considered,
|
||||||
|
m.admissionWait, m.admissionRejected,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metrics) observeAdmission(band, model string, wait time.Duration) {
|
||||||
|
if band == "" {
|
||||||
|
band = "none"
|
||||||
|
}
|
||||||
|
m.admissionWait.WithLabelValues(band, model).Observe(wait.Seconds())
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metrics) observeAdmissionRejected(band, reason string) {
|
||||||
|
if band == "" {
|
||||||
|
band = "unknown"
|
||||||
|
}
|
||||||
|
m.admissionRejected.WithLabelValues(band, reason).Inc()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metrics) observeRequest(model, pool string, status int, d time.Duration, u *proxy.Usage, ttft time.Duration) {
|
||||||
|
if model == "" {
|
||||||
|
model = "unknown"
|
||||||
|
}
|
||||||
|
if pool == "" {
|
||||||
|
pool = "none"
|
||||||
|
}
|
||||||
|
m.requests.WithLabelValues(model, pool, strconv.Itoa(status)).Inc()
|
||||||
|
m.duration.WithLabelValues(model, pool).Observe(d.Seconds())
|
||||||
|
|
||||||
|
if ttft > 0 {
|
||||||
|
m.ttft.WithLabelValues(model, pool).Observe(ttft.Seconds())
|
||||||
|
}
|
||||||
|
if u != nil {
|
||||||
|
m.tokens.WithLabelValues(model, "prompt").Add(float64(u.PromptTokens))
|
||||||
|
m.tokens.WithLabelValues(model, "completion").Add(float64(u.CompletionTokens))
|
||||||
|
if u.CachedTokens > 0 {
|
||||||
|
m.tokens.WithLabelValues(model, "cached").Add(float64(u.CachedTokens))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *metrics) observeScores(d scheduler.Decision) {
|
||||||
|
for name, v := range d.Scores {
|
||||||
|
m.scorerValue.WithLabelValues(name, d.Endpoint.ID).Set(v)
|
||||||
|
}
|
||||||
|
for filter, n := range d.Dropped {
|
||||||
|
m.filtered.WithLabelValues(filter).Add(float64(n))
|
||||||
|
}
|
||||||
|
m.considered.Observe(float64(d.Considered))
|
||||||
|
}
|
||||||
@@ -0,0 +1,468 @@
|
|||||||
|
// Package proxy streams a request to one upstream engine and relays the
|
||||||
|
// response to the client (plan §7.8).
|
||||||
|
//
|
||||||
|
// It handles exactly one attempt. Choosing the endpoint, retrying and tripping
|
||||||
|
// breakers belong to the caller, because those decisions need the scheduler and
|
||||||
|
// the endpoint registry. What the proxy owes the caller is an honest answer to
|
||||||
|
// "did anything reach the client yet?" — once the first byte is out, retrying
|
||||||
|
// elsewhere would corrupt the response.
|
||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net"
|
||||||
|
"net/http"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Config holds proxy settings.
|
||||||
|
type Config struct {
|
||||||
|
// DialTimeout bounds establishing the TCP connection.
|
||||||
|
DialTimeout time.Duration
|
||||||
|
// ResponseHeaderTimeout bounds the wait for upstream response headers.
|
||||||
|
ResponseHeaderTimeout time.Duration
|
||||||
|
// FirstTokenTimeout bounds the wait for the first streamed event.
|
||||||
|
FirstTokenTimeout time.Duration
|
||||||
|
// IdleTimeout bounds the gap between streamed events. A stalled engine
|
||||||
|
// looks identical to a slow one over a single connection, so without this
|
||||||
|
// a wedged request holds a client forever.
|
||||||
|
IdleTimeout time.Duration
|
||||||
|
// TotalTimeout bounds the whole request. Generous by default: a long
|
||||||
|
// generation is legitimately slow.
|
||||||
|
TotalTimeout time.Duration
|
||||||
|
// MaxIdleConnsPerHost keeps connections warm; engines are few and busy.
|
||||||
|
MaxIdleConnsPerHost int
|
||||||
|
// MaxResponseBytes bounds a buffered (non-streaming) response.
|
||||||
|
MaxResponseBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
// DefaultConfig returns sensible defaults.
|
||||||
|
func DefaultConfig() Config {
|
||||||
|
return Config{
|
||||||
|
DialTimeout: 5 * time.Second,
|
||||||
|
ResponseHeaderTimeout: 60 * time.Second,
|
||||||
|
FirstTokenTimeout: 120 * time.Second,
|
||||||
|
IdleTimeout: 60 * time.Second,
|
||||||
|
TotalTimeout: 30 * time.Minute,
|
||||||
|
MaxIdleConnsPerHost: 64,
|
||||||
|
MaxResponseBytes: 64 << 20,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Config) normalize() {
|
||||||
|
d := DefaultConfig()
|
||||||
|
if c.DialTimeout <= 0 {
|
||||||
|
c.DialTimeout = d.DialTimeout
|
||||||
|
}
|
||||||
|
if c.ResponseHeaderTimeout <= 0 {
|
||||||
|
c.ResponseHeaderTimeout = d.ResponseHeaderTimeout
|
||||||
|
}
|
||||||
|
if c.FirstTokenTimeout <= 0 {
|
||||||
|
c.FirstTokenTimeout = d.FirstTokenTimeout
|
||||||
|
}
|
||||||
|
if c.IdleTimeout <= 0 {
|
||||||
|
c.IdleTimeout = d.IdleTimeout
|
||||||
|
}
|
||||||
|
if c.TotalTimeout <= 0 {
|
||||||
|
c.TotalTimeout = d.TotalTimeout
|
||||||
|
}
|
||||||
|
if c.MaxIdleConnsPerHost <= 0 {
|
||||||
|
c.MaxIdleConnsPerHost = d.MaxIdleConnsPerHost
|
||||||
|
}
|
||||||
|
if c.MaxResponseBytes <= 0 {
|
||||||
|
c.MaxResponseBytes = d.MaxResponseBytes
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Proxy forwards requests to engine endpoints.
|
||||||
|
type Proxy struct {
|
||||||
|
cfg Config
|
||||||
|
client *http.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// New builds a proxy.
|
||||||
|
func New(cfg Config) *Proxy {
|
||||||
|
cfg.normalize()
|
||||||
|
return &Proxy{
|
||||||
|
cfg: cfg,
|
||||||
|
client: &http.Client{
|
||||||
|
// No client timeout: streaming responses run for minutes and the
|
||||||
|
// per-phase watchdogs below bound them more precisely.
|
||||||
|
Transport: &http.Transport{
|
||||||
|
DialContext: (&net.Dialer{Timeout: cfg.DialTimeout}).DialContext,
|
||||||
|
ResponseHeaderTimeout: cfg.ResponseHeaderTimeout,
|
||||||
|
MaxIdleConnsPerHost: cfg.MaxIdleConnsPerHost,
|
||||||
|
MaxIdleConns: cfg.MaxIdleConnsPerHost * 8,
|
||||||
|
IdleConnTimeout: 90 * time.Second,
|
||||||
|
// Engines rarely speak h2c, and HTTP/2 over cleartext would
|
||||||
|
// need explicit setup anyway.
|
||||||
|
ForceAttemptHTTP2: false,
|
||||||
|
DisableCompression: true,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Request describes one upstream attempt.
|
||||||
|
type Request struct {
|
||||||
|
// URL is the fully qualified upstream URL.
|
||||||
|
URL string
|
||||||
|
// Body is the request body to send.
|
||||||
|
Body []byte
|
||||||
|
// Header carries headers to forward (authorization is deliberately not).
|
||||||
|
Header http.Header
|
||||||
|
// Stream says whether an SSE response is expected.
|
||||||
|
Stream bool
|
||||||
|
// SuppressUsageChunk drops the final usage-only chunk from the client while
|
||||||
|
// still recording it. Set when the router asked for usage the client did
|
||||||
|
// not request.
|
||||||
|
SuppressUsageChunk bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// Usage holds the token counts reported by the engine.
|
||||||
|
type Usage struct {
|
||||||
|
PromptTokens int
|
||||||
|
CompletionTokens int
|
||||||
|
TotalTokens int
|
||||||
|
CachedTokens int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Outcome reports what happened during one attempt.
|
||||||
|
type Outcome struct {
|
||||||
|
// Wrote is true once any byte reached the client. Retrying is then unsafe.
|
||||||
|
Wrote bool
|
||||||
|
// Status is the upstream status code, 0 if the request never completed.
|
||||||
|
Status int
|
||||||
|
// Retryable means the attempt failed in a way another endpoint might survive.
|
||||||
|
Retryable bool
|
||||||
|
// TTFT is the time until the first streamed event, zero if none arrived.
|
||||||
|
TTFT time.Duration
|
||||||
|
// Usage is the engine-reported token usage, if it was present.
|
||||||
|
Usage *Usage
|
||||||
|
// Bytes is how many bytes were relayed to the client.
|
||||||
|
Bytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
// ErrClientGone means the client disconnected; the upstream was cancelled.
|
||||||
|
ErrClientGone = errors.New("client disconnected")
|
||||||
|
// ErrUpstreamStalled means no event arrived within the idle or first-token timeout.
|
||||||
|
ErrUpstreamStalled = errors.New("upstream stalled")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Do performs one attempt against one endpoint.
|
||||||
|
//
|
||||||
|
// On a retryable failure it writes nothing to w, leaving the caller free to try
|
||||||
|
// another endpoint. Once it has written anything, Outcome.Wrote is true and the
|
||||||
|
// caller must not retry.
|
||||||
|
func (p *Proxy) Do(ctx context.Context, w http.ResponseWriter, r Request) (Outcome, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, p.cfg.TotalTimeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, r.URL, bytes.NewReader(r.Body))
|
||||||
|
if err != nil {
|
||||||
|
return Outcome{Retryable: false}, err
|
||||||
|
}
|
||||||
|
req.Header = cloneForUpstream(r.Header)
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
req.ContentLength = int64(len(r.Body))
|
||||||
|
|
||||||
|
resp, err := p.client.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
// The client going away is not the endpoint's fault; never blame it.
|
||||||
|
if ctx.Err() != nil && ctxCauseIsClient(ctx) {
|
||||||
|
return Outcome{Retryable: false}, ErrClientGone
|
||||||
|
}
|
||||||
|
return Outcome{Retryable: true}, fmt.Errorf("upstream request failed: %w", err)
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
|
||||||
|
if retryableStatus(resp.StatusCode) {
|
||||||
|
// Drain a little so the connection can be reused, then let the caller
|
||||||
|
// pick somewhere else. Nothing has been written to the client.
|
||||||
|
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4<<10))
|
||||||
|
return Outcome{Status: resp.StatusCode, Retryable: true},
|
||||||
|
fmt.Errorf("upstream returned %s", resp.Status)
|
||||||
|
}
|
||||||
|
|
||||||
|
if r.Stream && isEventStream(resp.Header) {
|
||||||
|
return p.relayStream(ctx, cancel, w, resp, r)
|
||||||
|
}
|
||||||
|
return p.relayBuffered(w, resp)
|
||||||
|
}
|
||||||
|
|
||||||
|
// relayBuffered copies a non-streaming response through, capturing usage.
|
||||||
|
func (p *Proxy) relayBuffered(w http.ResponseWriter, resp *http.Response) (Outcome, error) {
|
||||||
|
body, err := io.ReadAll(io.LimitReader(resp.Body, p.cfg.MaxResponseBytes))
|
||||||
|
if err != nil {
|
||||||
|
return Outcome{Status: resp.StatusCode, Retryable: true}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
copyHeaders(w.Header(), resp.Header)
|
||||||
|
w.WriteHeader(resp.StatusCode)
|
||||||
|
n, werr := w.Write(body)
|
||||||
|
|
||||||
|
out := Outcome{
|
||||||
|
Wrote: n > 0,
|
||||||
|
Status: resp.StatusCode,
|
||||||
|
Bytes: int64(n),
|
||||||
|
Usage: usageFromJSON(body),
|
||||||
|
}
|
||||||
|
if werr != nil {
|
||||||
|
return out, ErrClientGone
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// relayStream relays an SSE response event by event, flushing as it goes.
|
||||||
|
//
|
||||||
|
// abort must cancel the context the upstream request was built with: the idle
|
||||||
|
// watchdog can only stop a silent engine by aborting that read.
|
||||||
|
func (p *Proxy) relayStream(ctx context.Context, abort context.CancelFunc, w http.ResponseWriter, resp *http.Response, r Request) (Outcome, error) {
|
||||||
|
flusher, ok := w.(http.Flusher)
|
||||||
|
if !ok {
|
||||||
|
return Outcome{Status: resp.StatusCode, Retryable: false},
|
||||||
|
errors.New("response writer does not support streaming")
|
||||||
|
}
|
||||||
|
|
||||||
|
copyHeaders(w.Header(), resp.Header)
|
||||||
|
w.WriteHeader(resp.StatusCode)
|
||||||
|
flusher.Flush()
|
||||||
|
|
||||||
|
out := Outcome{Wrote: true, Status: resp.StatusCode}
|
||||||
|
start := time.Now()
|
||||||
|
|
||||||
|
// One watchdog covers both phases: until the first event it enforces
|
||||||
|
// FirstTokenTimeout, afterwards IdleTimeout.
|
||||||
|
wd := newWatchdog(p.cfg.FirstTokenTimeout)
|
||||||
|
defer wd.run(ctx, abort)()
|
||||||
|
|
||||||
|
br := bufio.NewReaderSize(resp.Body, 16<<10)
|
||||||
|
var ev eventBuffer
|
||||||
|
|
||||||
|
for {
|
||||||
|
line, err := br.ReadString('\n')
|
||||||
|
if len(line) > 0 {
|
||||||
|
wd.beat()
|
||||||
|
|
||||||
|
ev.add(line)
|
||||||
|
if ev.complete() {
|
||||||
|
if out.TTFT == 0 {
|
||||||
|
out.TTFT = time.Since(start)
|
||||||
|
wd.setInterval(p.cfg.IdleTimeout)
|
||||||
|
}
|
||||||
|
if u := ev.usage(); u != nil {
|
||||||
|
out.Usage = u
|
||||||
|
if r.SuppressUsageChunk {
|
||||||
|
// The client never asked for usage; record it and drop
|
||||||
|
// the chunk rather than changing what they receive.
|
||||||
|
ev.reset()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
n, werr := io.WriteString(w, ev.string())
|
||||||
|
out.Bytes += int64(n)
|
||||||
|
ev.reset()
|
||||||
|
if werr != nil {
|
||||||
|
return out, ErrClientGone
|
||||||
|
}
|
||||||
|
flusher.Flush()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, io.EOF) {
|
||||||
|
// Flush a trailing event that never got its blank line.
|
||||||
|
if s := ev.string(); s != "" {
|
||||||
|
n, _ := io.WriteString(w, s)
|
||||||
|
out.Bytes += int64(n)
|
||||||
|
flusher.Flush()
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
if wd.fired() {
|
||||||
|
return out, ErrUpstreamStalled
|
||||||
|
}
|
||||||
|
if ctx.Err() != nil {
|
||||||
|
return out, ErrClientGone
|
||||||
|
}
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- SSE event assembly ---
|
||||||
|
|
||||||
|
// eventBuffer accumulates the lines of one SSE event. Events are relayed whole
|
||||||
|
// rather than line by line so a usage-only chunk can be dropped atomically.
|
||||||
|
type eventBuffer struct {
|
||||||
|
lines []string
|
||||||
|
data strings.Builder
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *eventBuffer) add(line string) {
|
||||||
|
e.lines = append(e.lines, line)
|
||||||
|
if payload, ok := strings.CutPrefix(line, "data: "); ok {
|
||||||
|
e.data.WriteString(strings.TrimRight(payload, "\r\n"))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// complete reports whether the blank line terminating an event has arrived.
|
||||||
|
func (e *eventBuffer) complete() bool {
|
||||||
|
if len(e.lines) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
last := e.lines[len(e.lines)-1]
|
||||||
|
return last == "\n" || last == "\r\n"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *eventBuffer) string() string {
|
||||||
|
if len(e.lines) == 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return strings.Join(e.lines, "")
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *eventBuffer) reset() {
|
||||||
|
e.lines = e.lines[:0]
|
||||||
|
e.data.Reset()
|
||||||
|
}
|
||||||
|
|
||||||
|
// usage returns the token usage carried by this event, if any.
|
||||||
|
func (e *eventBuffer) usage() *Usage {
|
||||||
|
s := e.data.String()
|
||||||
|
if s == "" || s == "[DONE]" || !strings.Contains(s, "\"usage\"") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return usageFromJSON([]byte(s))
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- watchdog ---
|
||||||
|
|
||||||
|
// watchdog aborts the upstream read once an interval passes with no beat, and
|
||||||
|
// records that it did so, so the resulting read error can be attributed to the
|
||||||
|
// engine stalling rather than to the client going away.
|
||||||
|
type watchdog struct {
|
||||||
|
interval atomic.Int64
|
||||||
|
last atomic.Int64
|
||||||
|
tripped atomic.Bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func newWatchdog(interval time.Duration) *watchdog {
|
||||||
|
w := &watchdog{}
|
||||||
|
w.interval.Store(int64(interval))
|
||||||
|
w.last.Store(time.Now().UnixNano())
|
||||||
|
return w
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *watchdog) beat() { w.last.Store(time.Now().UnixNano()) }
|
||||||
|
|
||||||
|
func (w *watchdog) setInterval(d time.Duration) { w.interval.Store(int64(d)) }
|
||||||
|
|
||||||
|
func (w *watchdog) fired() bool { return w.tripped.Load() }
|
||||||
|
|
||||||
|
// run starts the watchdog and returns a stop function. abort is called if an
|
||||||
|
// interval elapses without a beat.
|
||||||
|
func (w *watchdog) run(ctx context.Context, abort context.CancelFunc) func() {
|
||||||
|
done := make(chan struct{})
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
// Tick well below the shortest interval so the overshoot stays small.
|
||||||
|
// The interval shrinks once the first token arrives, so derive the tick
|
||||||
|
// from the smaller of the two rather than from whatever is set now.
|
||||||
|
tick := max(time.Duration(w.interval.Load())/4, 10*time.Millisecond)
|
||||||
|
t := time.NewTicker(tick)
|
||||||
|
defer t.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-done:
|
||||||
|
return
|
||||||
|
case <-t.C:
|
||||||
|
interval := time.Duration(w.interval.Load())
|
||||||
|
if time.Since(time.Unix(0, w.last.Load())) > interval {
|
||||||
|
w.tripped.Store(true)
|
||||||
|
abort()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Keep the tick proportional as the interval changes.
|
||||||
|
if want := max(interval/4, 10*time.Millisecond); want != tick {
|
||||||
|
tick = want
|
||||||
|
t.Reset(tick)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
var once sync.Once
|
||||||
|
return func() { once.Do(func() { close(done) }) }
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- helpers ---
|
||||||
|
|
||||||
|
func retryableStatus(code int) bool {
|
||||||
|
switch code {
|
||||||
|
case http.StatusTooManyRequests,
|
||||||
|
http.StatusBadGateway,
|
||||||
|
http.StatusServiceUnavailable,
|
||||||
|
http.StatusGatewayTimeout:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func isEventStream(h http.Header) bool {
|
||||||
|
return strings.HasPrefix(h.Get("Content-Type"), "text/event-stream")
|
||||||
|
}
|
||||||
|
|
||||||
|
// hopByHop headers must not be forwarded (RFC 9110 §7.6.1).
|
||||||
|
var hopByHop = map[string]bool{
|
||||||
|
"Connection": true,
|
||||||
|
"Proxy-Connection": true,
|
||||||
|
"Keep-Alive": true,
|
||||||
|
"Proxy-Authenticate": true,
|
||||||
|
"Proxy-Authorization": true,
|
||||||
|
"Te": true,
|
||||||
|
"Trailer": true,
|
||||||
|
"Transfer-Encoding": true,
|
||||||
|
"Upgrade": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
func copyHeaders(dst, src http.Header) {
|
||||||
|
for k, vs := range src {
|
||||||
|
if hopByHop[http.CanonicalHeaderKey(k)] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, v := range vs {
|
||||||
|
dst.Add(k, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// cloneForUpstream forwards only what the engine needs. The client's
|
||||||
|
// Authorization header is deliberately dropped: it authenticates the caller to
|
||||||
|
// llmgw, not llmgw to the engine, and engines are reached over a trusted
|
||||||
|
// network (plan §14).
|
||||||
|
func cloneForUpstream(h http.Header) http.Header {
|
||||||
|
out := make(http.Header, 4)
|
||||||
|
for _, k := range []string{"Accept", "User-Agent", "Traceparent", "Tracestate"} {
|
||||||
|
if v := h.Get(k); v != "" {
|
||||||
|
out.Set(k, v)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func ctxCauseIsClient(ctx context.Context) bool {
|
||||||
|
return errors.Is(context.Cause(ctx), context.Canceled)
|
||||||
|
}
|
||||||
@@ -0,0 +1,302 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func testProxy(mut ...func(*Config)) *Proxy {
|
||||||
|
c := DefaultConfig()
|
||||||
|
c.FirstTokenTimeout = 2 * time.Second
|
||||||
|
c.IdleTimeout = 500 * time.Millisecond
|
||||||
|
c.TotalTimeout = 10 * time.Second
|
||||||
|
for _, f := range mut {
|
||||||
|
f(&c)
|
||||||
|
}
|
||||||
|
return New(c)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sseServer serves a scripted SSE stream.
|
||||||
|
func sseServer(t *testing.T, chunks []string, gap time.Duration) (*httptest.Server, *atomic.Bool) {
|
||||||
|
t.Helper()
|
||||||
|
var cancelled atomic.Bool
|
||||||
|
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
f := w.(http.Flusher)
|
||||||
|
f.Flush()
|
||||||
|
for _, c := range chunks {
|
||||||
|
select {
|
||||||
|
case <-r.Context().Done():
|
||||||
|
cancelled.Store(true)
|
||||||
|
return
|
||||||
|
case <-time.After(gap):
|
||||||
|
}
|
||||||
|
if _, err := io.WriteString(w, c); err != nil {
|
||||||
|
cancelled.Store(true)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
f.Flush()
|
||||||
|
}
|
||||||
|
}))
|
||||||
|
t.Cleanup(srv.Close)
|
||||||
|
return srv, &cancelled
|
||||||
|
}
|
||||||
|
|
||||||
|
func dataChunk(s string) string { return "data: " + s + "\n\n" }
|
||||||
|
|
||||||
|
func TestBufferedResponseIsRelayed(t *testing.T) {
|
||||||
|
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
io.WriteString(w, `{"id":"x","usage":{"prompt_tokens":7,"completion_tokens":3}}`)
|
||||||
|
}))
|
||||||
|
defer up.Close()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy().Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Status != 200 || !out.Wrote {
|
||||||
|
t.Errorf("outcome=%+v", out)
|
||||||
|
}
|
||||||
|
if out.Usage == nil || out.Usage.PromptTokens != 7 || out.Usage.TotalTokens != 10 {
|
||||||
|
t.Errorf("usage not captured: %+v", out.Usage)
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), `"id":"x"`) {
|
||||||
|
t.Errorf("body not relayed: %q", rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStreamIsRelayedEventByEvent(t *testing.T) {
|
||||||
|
up, _ := sseServer(t, []string{
|
||||||
|
dataChunk(`{"choices":[{"delta":{"content":"a"}}]}`),
|
||||||
|
dataChunk(`{"choices":[{"delta":{"content":"b"}}]}`),
|
||||||
|
"data: [DONE]\n\n",
|
||||||
|
}, time.Millisecond)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy().Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`), Stream: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
body := rec.Body.String()
|
||||||
|
if n := strings.Count(body, "data: "); n != 3 {
|
||||||
|
t.Errorf("expected 3 data lines, got %d: %q", n, body)
|
||||||
|
}
|
||||||
|
if !strings.HasSuffix(body, "[DONE]\n\n") {
|
||||||
|
t.Errorf("stream did not end with [DONE]: %q", body)
|
||||||
|
}
|
||||||
|
if out.TTFT == 0 {
|
||||||
|
t.Error("TTFT was not recorded")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUsageCapturedFromStream(t *testing.T) {
|
||||||
|
up, _ := sseServer(t, []string{
|
||||||
|
dataChunk(`{"choices":[{"delta":{"content":"a"}}]}`),
|
||||||
|
dataChunk(`{"choices":[],"usage":{"prompt_tokens":11,"completion_tokens":5,"total_tokens":16}}`),
|
||||||
|
"data: [DONE]\n\n",
|
||||||
|
}, time.Millisecond)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy().Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`), Stream: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Usage == nil || out.Usage.TotalTokens != 16 {
|
||||||
|
t.Fatalf("usage not captured from stream: %+v", out.Usage)
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), `"usage"`) {
|
||||||
|
t.Error("usage chunk should reach a client that asked for it")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUsageChunkSuppressed(t *testing.T) {
|
||||||
|
// The router asked the engine for usage the client never requested: record
|
||||||
|
// it, but do not change what the client receives.
|
||||||
|
up, _ := sseServer(t, []string{
|
||||||
|
dataChunk(`{"choices":[{"delta":{"content":"a"}}]}`),
|
||||||
|
dataChunk(`{"choices":[],"usage":{"prompt_tokens":11,"completion_tokens":5,"total_tokens":16}}`),
|
||||||
|
"data: [DONE]\n\n",
|
||||||
|
}, time.Millisecond)
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy().Do(t.Context(), rec, Request{
|
||||||
|
URL: up.URL, Body: []byte(`{}`), Stream: true, SuppressUsageChunk: true,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Usage == nil || out.Usage.TotalTokens != 16 {
|
||||||
|
t.Fatalf("usage not recorded: %+v", out.Usage)
|
||||||
|
}
|
||||||
|
if strings.Contains(rec.Body.String(), `"usage"`) {
|
||||||
|
t.Errorf("usage chunk leaked to the client: %q", rec.Body.String())
|
||||||
|
}
|
||||||
|
if !strings.Contains(rec.Body.String(), "[DONE]") {
|
||||||
|
t.Error("[DONE] was dropped along with the usage chunk")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRetryableStatusWritesNothing(t *testing.T) {
|
||||||
|
// The caller must be free to try another endpoint, which is only safe if
|
||||||
|
// the client has seen nothing.
|
||||||
|
for _, code := range []int{429, 502, 503, 504} {
|
||||||
|
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
http.Error(w, "busy", code)
|
||||||
|
}))
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy().Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`)})
|
||||||
|
up.Close()
|
||||||
|
|
||||||
|
if err == nil {
|
||||||
|
t.Errorf("%d: expected an error", code)
|
||||||
|
}
|
||||||
|
if !out.Retryable {
|
||||||
|
t.Errorf("%d: not marked retryable", code)
|
||||||
|
}
|
||||||
|
if out.Wrote || rec.Body.Len() > 0 {
|
||||||
|
t.Errorf("%d: wrote %q to the client", code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNonRetryableStatusIsRelayed(t *testing.T) {
|
||||||
|
// A 400 is the client's fault; retrying elsewhere would produce the same
|
||||||
|
// answer, so relay it.
|
||||||
|
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusBadRequest)
|
||||||
|
io.WriteString(w, `{"error":{"message":"bad"}}`)
|
||||||
|
}))
|
||||||
|
defer up.Close()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy().Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Retryable {
|
||||||
|
t.Error("400 must not be retryable")
|
||||||
|
}
|
||||||
|
if rec.Code != http.StatusBadRequest || !strings.Contains(rec.Body.String(), "bad") {
|
||||||
|
t.Errorf("error response not relayed: %d %q", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConnectionFailureIsRetryable(t *testing.T) {
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
out, err := testProxy(func(c *Config) { c.DialTimeout = 200 * time.Millisecond }).
|
||||||
|
Do(t.Context(), rec, Request{URL: "http://127.0.0.1:1/x", Body: []byte(`{}`)})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected an error")
|
||||||
|
}
|
||||||
|
if !out.Retryable || out.Wrote {
|
||||||
|
t.Errorf("outcome=%+v", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestClientDisconnectCancelsUpstream(t *testing.T) {
|
||||||
|
// Phase 1's exit criterion at the proxy level: the GPU must stop working
|
||||||
|
// when the client goes away.
|
||||||
|
chunks := make([]string, 200)
|
||||||
|
for i := range chunks {
|
||||||
|
chunks[i] = dataChunk(fmt.Sprintf(`{"choices":[{"delta":{"content":"%d"}}]}`, i))
|
||||||
|
}
|
||||||
|
up, cancelled := sseServer(t, chunks, 10*time.Millisecond)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
|
||||||
|
done := make(chan Outcome, 1)
|
||||||
|
go func() {
|
||||||
|
out, _ := testProxy().Do(ctx, rec, Request{URL: up.URL, Body: []byte(`{}`), Stream: true})
|
||||||
|
done <- out
|
||||||
|
}()
|
||||||
|
|
||||||
|
time.Sleep(80 * time.Millisecond) // let a few events through
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(3 * time.Second):
|
||||||
|
t.Fatal("proxy did not return after client cancellation")
|
||||||
|
}
|
||||||
|
|
||||||
|
deadline := time.Now().Add(2 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if cancelled.Load() {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatal("upstream kept generating after the client disconnected")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestStalledUpstreamHitsIdleTimeout(t *testing.T) {
|
||||||
|
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
f := w.(http.Flusher)
|
||||||
|
io.WriteString(w, dataChunk(`{"choices":[{"delta":{"content":"a"}}]}`))
|
||||||
|
f.Flush()
|
||||||
|
<-r.Context().Done() // then go silent forever
|
||||||
|
}))
|
||||||
|
defer up.Close()
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
p := testProxy(func(c *Config) { c.IdleTimeout = 200 * time.Millisecond })
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
out, err := p.Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`), Stream: true})
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
|
||||||
|
if !errors.Is(err, ErrUpstreamStalled) {
|
||||||
|
t.Fatalf("expected ErrUpstreamStalled, got %v", err)
|
||||||
|
}
|
||||||
|
if !out.Wrote {
|
||||||
|
t.Error("the events delivered before the stall should be reported as written")
|
||||||
|
}
|
||||||
|
if elapsed > 3*time.Second {
|
||||||
|
t.Errorf("idle timeout took too long: %v", elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHopByHopHeadersNotForwarded(t *testing.T) {
|
||||||
|
var got http.Header
|
||||||
|
up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||||
|
got = r.Header.Clone()
|
||||||
|
io.WriteString(w, `{}`)
|
||||||
|
}))
|
||||||
|
defer up.Close()
|
||||||
|
|
||||||
|
h := http.Header{}
|
||||||
|
h.Set("Authorization", "Bearer sk-llmgw-secret")
|
||||||
|
h.Set("Accept", "text/event-stream")
|
||||||
|
h.Set("Traceparent", "00-trace-span-01")
|
||||||
|
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
if _, err := testProxy().Do(t.Context(), rec, Request{URL: up.URL, Body: []byte(`{}`), Header: h}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.Get("Authorization") != "" {
|
||||||
|
t.Error("the client's Authorization header was forwarded upstream")
|
||||||
|
}
|
||||||
|
if got.Get("Accept") != "text/event-stream" {
|
||||||
|
t.Error("Accept was not forwarded")
|
||||||
|
}
|
||||||
|
if got.Get("Traceparent") != "00-trace-span-01" {
|
||||||
|
t.Error("Traceparent was not forwarded; tracing would break")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package proxy
|
||||||
|
|
||||||
|
import "encoding/json"
|
||||||
|
|
||||||
|
// usageEnvelope mirrors just the usage block of an OpenAI response.
|
||||||
|
// Everything else in the payload is passed through untouched, so only these
|
||||||
|
// fields are ever decoded.
|
||||||
|
type usageEnvelope struct {
|
||||||
|
Usage *struct {
|
||||||
|
PromptTokens int `json:"prompt_tokens"`
|
||||||
|
CompletionTokens int `json:"completion_tokens"`
|
||||||
|
TotalTokens int `json:"total_tokens"`
|
||||||
|
PromptTokensDetails *struct {
|
||||||
|
CachedTokens int `json:"cached_tokens"`
|
||||||
|
} `json:"prompt_tokens_details"`
|
||||||
|
} `json:"usage"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// usageFromJSON extracts token usage from a response or an SSE chunk.
|
||||||
|
// It returns nil when the payload carries no usage block, which is the normal
|
||||||
|
// case for every chunk but the last.
|
||||||
|
func usageFromJSON(b []byte) *Usage {
|
||||||
|
var env usageEnvelope
|
||||||
|
if err := json.Unmarshal(b, &env); err != nil || env.Usage == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
u := &Usage{
|
||||||
|
PromptTokens: env.Usage.PromptTokens,
|
||||||
|
CompletionTokens: env.Usage.CompletionTokens,
|
||||||
|
TotalTokens: env.Usage.TotalTokens,
|
||||||
|
}
|
||||||
|
if d := env.Usage.PromptTokensDetails; d != nil {
|
||||||
|
u.CachedTokens = d.CachedTokens
|
||||||
|
}
|
||||||
|
if u.TotalTokens == 0 {
|
||||||
|
u.TotalTokens = u.PromptTokens + u.CompletionTokens
|
||||||
|
}
|
||||||
|
return u
|
||||||
|
}
|
||||||
@@ -0,0 +1,341 @@
|
|||||||
|
// Package router wires the pieces into a working OpenAI-compatible gateway:
|
||||||
|
// resolve the model, pick an endpoint, proxy the stream, retry when it is safe.
|
||||||
|
//
|
||||||
|
// The sub-packages hold the reusable parts (scheduler, endpoint, scrape, proxy,
|
||||||
|
// breaker); this package holds the orchestration and the HTTP surface.
|
||||||
|
package router
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/admission"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/breaker"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/endpoint"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/proxy"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/scheduler"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router/scrape"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Router serves the OpenAI API in front of a set of engine replicas.
|
||||||
|
type Router struct {
|
||||||
|
cfg config.Router
|
||||||
|
log *slog.Logger
|
||||||
|
|
||||||
|
reg *endpoint.Registry
|
||||||
|
poller *scrape.Poller
|
||||||
|
px *proxy.Proxy
|
||||||
|
m *metrics
|
||||||
|
// adm is nil when admission control is disabled.
|
||||||
|
adm *admission.Controller
|
||||||
|
|
||||||
|
profiles map[string]*scheduler.Profile
|
||||||
|
// models maps every name a client may use — canonical names and aliases —
|
||||||
|
// onto its catalog entry.
|
||||||
|
models map[string]config.Model
|
||||||
|
}
|
||||||
|
|
||||||
|
// New builds a router from configuration. Start must be called to begin scraping.
|
||||||
|
func New(cfg config.Server, log *slog.Logger) (*Router, error) {
|
||||||
|
if log == nil {
|
||||||
|
log = slog.Default()
|
||||||
|
}
|
||||||
|
|
||||||
|
reg := endpoint.NewRegistry(endpoint.Config{
|
||||||
|
StaleAfter: cfg.Router.StaleAfter,
|
||||||
|
Breaker: breaker.Config{
|
||||||
|
FailureThreshold: cfg.Router.FailureThreshold,
|
||||||
|
BaseCooldown: cfg.Router.Cooldown,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
reg.Replace(toInfos(cfg.Endpoints))
|
||||||
|
|
||||||
|
sc := scheduler.ProfileConfig{
|
||||||
|
TargetWaiting: cfg.Router.Scoring.TargetWaiting,
|
||||||
|
TargetInflight: cfg.Router.Scoring.TargetInflight,
|
||||||
|
KVCeiling: cfg.Router.Scoring.KVCeiling,
|
||||||
|
SessionLoadFactor: cfg.Router.Scoring.SessionLoadFactor,
|
||||||
|
}
|
||||||
|
profiles := map[string]*scheduler.Profile{}
|
||||||
|
for _, name := range []string{"chat", "batch", "long-context", "debug"} {
|
||||||
|
profiles[name] = scheduler.Builtin(name, sc)
|
||||||
|
}
|
||||||
|
if _, ok := profiles[cfg.Router.DefaultProfile]; !ok {
|
||||||
|
return nil, fmt.Errorf("unknown default_profile %q", cfg.Router.DefaultProfile)
|
||||||
|
}
|
||||||
|
|
||||||
|
models := map[string]config.Model{}
|
||||||
|
for _, m := range cfg.Models {
|
||||||
|
models[m.Name] = m
|
||||||
|
for _, a := range m.Aliases {
|
||||||
|
models[a] = m
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
r := &Router{
|
||||||
|
cfg: cfg.Router,
|
||||||
|
log: log,
|
||||||
|
reg: reg,
|
||||||
|
profiles: profiles,
|
||||||
|
models: models,
|
||||||
|
m: newMetrics(),
|
||||||
|
px: proxy.New(proxy.Config{
|
||||||
|
DialTimeout: cfg.Router.Timeouts.Dial,
|
||||||
|
ResponseHeaderTimeout: cfg.Router.Timeouts.ResponseHeader,
|
||||||
|
FirstTokenTimeout: cfg.Router.Timeouts.FirstToken,
|
||||||
|
IdleTimeout: cfg.Router.Timeouts.Idle,
|
||||||
|
TotalTimeout: cfg.Router.Timeouts.Total,
|
||||||
|
MaxResponseBytes: cfg.Router.MaxBodyBytes,
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
r.poller = scrape.NewPoller(reg, scrape.Config{
|
||||||
|
Interval: cfg.Router.ScrapeInterval,
|
||||||
|
Timeout: cfg.Router.ScrapeTimeout,
|
||||||
|
}, log)
|
||||||
|
|
||||||
|
if cfg.Router.Admission.Enabled {
|
||||||
|
adm, err := admission.New(admissionConfig(cfg.Router), r.saturation)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("admission: %w", err)
|
||||||
|
}
|
||||||
|
r.adm = adm
|
||||||
|
}
|
||||||
|
return r, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func admissionConfig(rc config.Router) admission.Config {
|
||||||
|
bands := make([]admission.Band, 0, len(rc.Admission.Bands))
|
||||||
|
for _, b := range rc.Admission.Bands {
|
||||||
|
bands = append(bands, admission.Band{
|
||||||
|
Name: b.Name,
|
||||||
|
Priority: b.Priority,
|
||||||
|
SaturationCeiling: b.SaturationCeiling,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return admission.Config{
|
||||||
|
Bands: bands,
|
||||||
|
DefaultBand: rc.Admission.DefaultBand,
|
||||||
|
Thresholds: admission.Thresholds{
|
||||||
|
// Admission and the queue scorer must agree on what a healthy
|
||||||
|
// buffer is, or the two layers fight each other.
|
||||||
|
TargetWaitingPerReplica: rc.Scoring.TargetWaiting,
|
||||||
|
KVCeiling: rc.Scoring.KVCeiling,
|
||||||
|
},
|
||||||
|
QueueTTL: rc.Admission.QueueTTL,
|
||||||
|
QueueCapacityPerBand: rc.Admission.QueueCapacityPerBand,
|
||||||
|
Interval: rc.Admission.Interval,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// saturation reports the pool pressure for a model, for the admission layer.
|
||||||
|
//
|
||||||
|
// outstanding is what admission has let through. It is compared against the
|
||||||
|
// registry's inflight counters rather than the scraped engine stats, because
|
||||||
|
// inflight updates the instant a request is dispatched while scraped stats lag
|
||||||
|
// by a whole interval. Comparing against scraped data would make every
|
||||||
|
// dispatched-but-unscraped request look like queue depth and throttle the
|
||||||
|
// steady state down to a few requests per scrape.
|
||||||
|
//
|
||||||
|
// What is left over — admitted but not yet dispatched — is the thundering herd:
|
||||||
|
// a burst that passed the check together before any of it became visible.
|
||||||
|
func (r *Router) saturation(model string, outstanding int) (float64, bool) {
|
||||||
|
eps := r.reg.Snapshots(model)
|
||||||
|
if len(eps) == 0 {
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
dispatched := 0
|
||||||
|
for _, e := range eps {
|
||||||
|
dispatched += e.Inflight
|
||||||
|
}
|
||||||
|
return admission.Saturation(eps, max(0, outstanding-dispatched), admission.Thresholds{
|
||||||
|
TargetWaitingPerReplica: r.cfg.Scoring.TargetWaiting,
|
||||||
|
KVCeiling: r.cfg.Scoring.KVCeiling,
|
||||||
|
}), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func toInfos(eps []config.Endpoint) []endpoint.Info {
|
||||||
|
out := make([]endpoint.Info, 0, len(eps))
|
||||||
|
for _, e := range eps {
|
||||||
|
out = append(out, endpoint.Info{
|
||||||
|
ID: e.ID,
|
||||||
|
URL: strings.TrimRight(e.URL, "/"),
|
||||||
|
Models: e.Models,
|
||||||
|
Pool: e.Pool,
|
||||||
|
Weight: e.Weight,
|
||||||
|
Priority: e.Priority,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start begins scraping endpoint metrics and dispatching admissions.
|
||||||
|
// It returns immediately.
|
||||||
|
func (r *Router) Start(ctx context.Context) {
|
||||||
|
r.poller.Sync(ctx)
|
||||||
|
if r.adm != nil {
|
||||||
|
r.adm.Start()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stop halts scraping and admission.
|
||||||
|
func (r *Router) Stop() {
|
||||||
|
r.poller.Stop()
|
||||||
|
if r.adm != nil {
|
||||||
|
r.adm.Close()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Registry exposes the endpoint registry, for tests and admin endpoints.
|
||||||
|
func (r *Router) Registry() *endpoint.Registry { return r.reg }
|
||||||
|
|
||||||
|
// --- model resolution ---
|
||||||
|
|
||||||
|
// resolve maps a client-supplied model name onto the catalog. Unknown names are
|
||||||
|
// passed through unchanged: an endpoint may serve a model the catalog does not
|
||||||
|
// list, and refusing it would be unhelpful.
|
||||||
|
func (r *Router) resolve(name string) (canonical string, m config.Model) {
|
||||||
|
if e, ok := r.models[name]; ok {
|
||||||
|
return e.Name, e
|
||||||
|
}
|
||||||
|
return name, config.Model{Name: name}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Router) profileFor(m config.Model, override string) *scheduler.Profile {
|
||||||
|
for _, name := range []string{override, m.RoutingProfile, r.cfg.DefaultProfile} {
|
||||||
|
if p, ok := r.profiles[name]; ok && name != "" {
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return r.profiles["chat"]
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- dispatch ---
|
||||||
|
|
||||||
|
// attempt describes one upstream try, for logging and metrics.
|
||||||
|
type dispatchResult struct {
|
||||||
|
endpointID string
|
||||||
|
pool string
|
||||||
|
servedModel
|
||||||
|
outcome proxy.Outcome
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
type servedModel struct {
|
||||||
|
model string
|
||||||
|
// fallback is true when the served model is not the one that was requested.
|
||||||
|
fallback bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// dispatch resolves the model, then tries endpoints until one works, the client
|
||||||
|
// has been written to, or the attempt budget runs out.
|
||||||
|
//
|
||||||
|
// The retry rule (plan §7.7) is strict: retry only while nothing has reached the
|
||||||
|
// client. Once bytes are out, a retry would splice two different responses
|
||||||
|
// together, which is worse than the error the client would otherwise see.
|
||||||
|
func (r *Router) dispatch(ctx context.Context, w http.ResponseWriter, req *parsedRequest) dispatchResult {
|
||||||
|
canonical, entry := r.resolve(req.Model)
|
||||||
|
|
||||||
|
// The requested model first, then its fallbacks. A fallback is a different
|
||||||
|
// model, so the client is told about it via x-llmgw-served-model.
|
||||||
|
chain := append([]string{canonical}, entry.Fallbacks...)
|
||||||
|
|
||||||
|
var last dispatchResult
|
||||||
|
for i, model := range chain {
|
||||||
|
res := r.dispatchOne(ctx, w, req, model, entry)
|
||||||
|
res.model = model
|
||||||
|
res.fallback = i > 0
|
||||||
|
|
||||||
|
if res.err == nil || res.outcome.Wrote || errors.Is(res.err, proxy.ErrClientGone) {
|
||||||
|
return res
|
||||||
|
}
|
||||||
|
last = res
|
||||||
|
// Only a total absence of capacity justifies moving to another model;
|
||||||
|
// a bad request would fail identically there.
|
||||||
|
if !errors.Is(res.err, errNoCapacity) {
|
||||||
|
return res
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return last
|
||||||
|
}
|
||||||
|
|
||||||
|
var errNoCapacity = errors.New("no endpoint available")
|
||||||
|
|
||||||
|
func (r *Router) dispatchOne(ctx context.Context, w http.ResponseWriter, req *parsedRequest, model string, entry config.Model) dispatchResult {
|
||||||
|
candidates := r.reg.Snapshots(model)
|
||||||
|
if len(candidates) == 0 {
|
||||||
|
return dispatchResult{err: fmt.Errorf("%w: no endpoint serves %q", errNoCapacity, model)}
|
||||||
|
}
|
||||||
|
|
||||||
|
profile := r.profileFor(entry, req.ProfileOverride)
|
||||||
|
sctx := &scheduler.Ctx{
|
||||||
|
Now: time.Now(),
|
||||||
|
Req: scheduler.Request{
|
||||||
|
Model: model,
|
||||||
|
SessionID: req.SessionID,
|
||||||
|
PromptPrefix: req.PromptPrefix,
|
||||||
|
PromptTokens: req.PromptTokens,
|
||||||
|
MaxTokens: req.MaxTokens,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
attempts := max(r.cfg.Retries+1, 1)
|
||||||
|
var last dispatchResult
|
||||||
|
|
||||||
|
for range attempts {
|
||||||
|
if len(candidates) == 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
dec, err := profile.Schedule(sctx, candidates)
|
||||||
|
if err != nil {
|
||||||
|
last = dispatchResult{err: fmt.Errorf("%w: %v", errNoCapacity, err)}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
release, granted := r.reg.Acquire(dec.Endpoint.ID)
|
||||||
|
if !granted {
|
||||||
|
// The breaker opened between scoring and dispatch; drop it and
|
||||||
|
// score again rather than failing the request.
|
||||||
|
candidates = without(candidates, dec.Endpoint.ID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
out, perr := r.px.Do(ctx, w, proxy.Request{
|
||||||
|
URL: dec.Endpoint.URL + req.Path,
|
||||||
|
Body: req.UpstreamBody,
|
||||||
|
Header: req.Header,
|
||||||
|
Stream: req.Stream,
|
||||||
|
SuppressUsageChunk: req.SuppressUsageChunk,
|
||||||
|
})
|
||||||
|
// A retryable failure is the endpoint's fault; a client disconnect is not.
|
||||||
|
release(perr == nil || errors.Is(perr, proxy.ErrClientGone))
|
||||||
|
|
||||||
|
res := dispatchResult{endpointID: dec.Endpoint.ID, pool: dec.Endpoint.Pool, outcome: out, err: perr}
|
||||||
|
r.m.observeScores(dec)
|
||||||
|
|
||||||
|
if perr == nil || out.Wrote || !out.Retryable {
|
||||||
|
return res
|
||||||
|
}
|
||||||
|
last = res
|
||||||
|
r.log.Warn("upstream attempt failed, trying another endpoint",
|
||||||
|
"endpoint", dec.Endpoint.ID, "model", model, "err", perr)
|
||||||
|
candidates = without(candidates, dec.Endpoint.ID)
|
||||||
|
}
|
||||||
|
|
||||||
|
if last.err == nil {
|
||||||
|
last.err = fmt.Errorf("%w: every endpoint for %q failed", errNoCapacity, model)
|
||||||
|
} else if !errors.Is(last.err, errNoCapacity) {
|
||||||
|
last.err = fmt.Errorf("%w: %v", errNoCapacity, last.err)
|
||||||
|
}
|
||||||
|
return last
|
||||||
|
}
|
||||||
|
|
||||||
|
func without(eps []endpoint.Snapshot, id string) []endpoint.Snapshot {
|
||||||
|
return slices.DeleteFunc(slices.Clone(eps), func(e endpoint.Snapshot) bool { return e.ID == id })
|
||||||
|
}
|
||||||
@@ -0,0 +1,315 @@
|
|||||||
|
package router
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"strings"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// fakeEngine serves the minimum an endpoint needs: the metric contract plus a
|
||||||
|
// scripted chat response.
|
||||||
|
type fakeEngine struct {
|
||||||
|
srv *httptest.Server
|
||||||
|
calls atomic.Int64
|
||||||
|
// status, when non-zero, is returned instead of a completion.
|
||||||
|
status atomic.Int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func newFakeEngine(t *testing.T, model string) *fakeEngine {
|
||||||
|
t.Helper()
|
||||||
|
f := &fakeEngine{}
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
fmt.Fprintf(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n")
|
||||||
|
fmt.Fprintf(w, "# TYPE vllm:num_requests_waiting gauge\nvllm:num_requests_waiting 0\n")
|
||||||
|
fmt.Fprintf(w, "# TYPE vllm:kv_cache_usage_perc gauge\nvllm:kv_cache_usage_perc 0.1\n")
|
||||||
|
})
|
||||||
|
mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
f.calls.Add(1)
|
||||||
|
if code := int(f.status.Load()); code != 0 {
|
||||||
|
http.Error(w, "unavailable", code)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
fmt.Fprintf(w, `{"model":%q,"choices":[{"message":{"content":"hi"}}],"usage":{"prompt_tokens":4,"completion_tokens":2}}`, model)
|
||||||
|
})
|
||||||
|
f.srv = httptest.NewServer(mux)
|
||||||
|
t.Cleanup(f.srv.Close)
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTestRouter(t *testing.T, cfg config.Server) *Router {
|
||||||
|
t.Helper()
|
||||||
|
rt, err := New(cfg, slog.New(slog.DiscardHandler))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
rt.Start(t.Context())
|
||||||
|
t.Cleanup(rt.Stop)
|
||||||
|
return rt
|
||||||
|
}
|
||||||
|
|
||||||
|
func post(t *testing.T, h http.Handler, body any, hdr map[string]string) *httptest.ResponseRecorder {
|
||||||
|
t.Helper()
|
||||||
|
b, err := json.Marshal(body)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/v1/chat/completions", bytes.NewReader(b))
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
for k, v := range hdr {
|
||||||
|
req.Header.Set(k, v)
|
||||||
|
}
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
h.ServeHTTP(rec, req)
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
func baseConfig(eps ...config.Endpoint) config.Server {
|
||||||
|
c := config.Default()
|
||||||
|
c.Router.ScrapeInterval = 50 * time.Millisecond
|
||||||
|
c.Router.StaleAfter = 5 * time.Second
|
||||||
|
c.Endpoints = eps
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRoutesToHealthyEndpoint(t *testing.T) {
|
||||||
|
e := newFakeEngine(t, "m")
|
||||||
|
rt := newTestRouter(t, baseConfig(config.Endpoint{
|
||||||
|
ID: "a", URL: e.srv.URL, Models: []string{"m"},
|
||||||
|
}))
|
||||||
|
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if e.calls.Load() != 1 {
|
||||||
|
t.Errorf("engine received %d calls", e.calls.Load())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRetriesOnAnotherEndpoint(t *testing.T) {
|
||||||
|
bad := newFakeEngine(t, "m")
|
||||||
|
bad.status.Store(http.StatusServiceUnavailable)
|
||||||
|
good := newFakeEngine(t, "m")
|
||||||
|
|
||||||
|
rt := newTestRouter(t, baseConfig(
|
||||||
|
config.Endpoint{ID: "bad", URL: bad.srv.URL, Models: []string{"m"}},
|
||||||
|
config.Endpoint{ID: "good", URL: good.srv.URL, Models: []string{"m"}},
|
||||||
|
))
|
||||||
|
|
||||||
|
// With two endpoints and two retries, the bad one must not sink the request.
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}}, nil)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if good.calls.Load() == 0 {
|
||||||
|
t.Error("the healthy endpoint was never tried")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNoRetryAfterBytesReachClient(t *testing.T) {
|
||||||
|
// An endpoint that starts streaming then dies must not be retried: splicing
|
||||||
|
// two responses together is worse than a short stream (plan §7.7).
|
||||||
|
var served atomic.Int64
|
||||||
|
mux := http.NewServeMux()
|
||||||
|
mux.HandleFunc("GET /metrics", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
io.WriteString(w, "# TYPE vllm:num_requests_running gauge\nvllm:num_requests_running 0\n"+
|
||||||
|
"# TYPE vllm:num_requests_waiting gauge\nvllm:num_requests_waiting 0\n"+
|
||||||
|
"# TYPE vllm:kv_cache_usage_perc gauge\nvllm:kv_cache_usage_perc 0\n")
|
||||||
|
})
|
||||||
|
mux.HandleFunc("POST /v1/chat/completions", func(w http.ResponseWriter, _ *http.Request) {
|
||||||
|
served.Add(1)
|
||||||
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
|
w.WriteHeader(http.StatusOK)
|
||||||
|
f := w.(http.Flusher)
|
||||||
|
io.WriteString(w, "data: {\"choices\":[{\"delta\":{\"content\":\"a\"}}]}\n\n")
|
||||||
|
f.Flush()
|
||||||
|
// Then hang up mid-stream.
|
||||||
|
})
|
||||||
|
srv := httptest.NewServer(mux)
|
||||||
|
defer srv.Close()
|
||||||
|
|
||||||
|
rt := newTestRouter(t, baseConfig(
|
||||||
|
config.Endpoint{ID: "a", URL: srv.URL, Models: []string{"m"}},
|
||||||
|
config.Endpoint{ID: "b", URL: srv.URL, Models: []string{"m"}},
|
||||||
|
))
|
||||||
|
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"model": "m", "messages": []any{}, "stream": true}, nil)
|
||||||
|
if !strings.Contains(rec.Body.String(), `"content":"a"`) {
|
||||||
|
t.Fatalf("partial stream not delivered: %q", rec.Body.String())
|
||||||
|
}
|
||||||
|
if n := served.Load(); n != 1 {
|
||||||
|
t.Errorf("request was sent %d times; it must not be retried once streaming began", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFallbackModel(t *testing.T) {
|
||||||
|
primary := newFakeEngine(t, "primary")
|
||||||
|
primary.status.Store(http.StatusServiceUnavailable)
|
||||||
|
backup := newFakeEngine(t, "backup")
|
||||||
|
|
||||||
|
cfg := baseConfig(
|
||||||
|
config.Endpoint{ID: "p", URL: primary.srv.URL, Models: []string{"primary"}},
|
||||||
|
config.Endpoint{ID: "b", URL: backup.srv.URL, Models: []string{"backup"}},
|
||||||
|
)
|
||||||
|
cfg.Models = []config.Model{{Name: "primary", Fallbacks: []string{"backup"}}}
|
||||||
|
rt := newTestRouter(t, cfg)
|
||||||
|
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"model": "primary", "messages": []any{}}, nil)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("status %d: %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
if backup.calls.Load() == 0 {
|
||||||
|
t.Fatal("the fallback model was never tried")
|
||||||
|
}
|
||||||
|
// The client asked for one model and got another; they must be told.
|
||||||
|
if got := rec.Header().Get("x-llmgw-served-model"); got != "backup" {
|
||||||
|
t.Errorf("x-llmgw-served-model=%q, expected \"backup\"", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAliasResolution(t *testing.T) {
|
||||||
|
e := newFakeEngine(t, "llama-3.3-70b")
|
||||||
|
cfg := baseConfig(config.Endpoint{
|
||||||
|
ID: "a", URL: e.srv.URL, Models: []string{"llama-3.3-70b"},
|
||||||
|
})
|
||||||
|
cfg.Models = []config.Model{{Name: "llama-3.3-70b", Aliases: []string{"llama"}}}
|
||||||
|
rt := newTestRouter(t, cfg)
|
||||||
|
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"model": "llama", "messages": []any{}}, nil)
|
||||||
|
if rec.Code != http.StatusOK {
|
||||||
|
t.Fatalf("alias was not resolved: %d %s", rec.Code, rec.Body.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestUnknownModelIs503(t *testing.T) {
|
||||||
|
e := newFakeEngine(t, "m")
|
||||||
|
rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
|
||||||
|
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"model": "nope", "messages": []any{}}, nil)
|
||||||
|
if rec.Code != http.StatusServiceUnavailable {
|
||||||
|
t.Errorf("status %d, expected 503", rec.Code)
|
||||||
|
}
|
||||||
|
var body map[string]any
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
|
||||||
|
t.Fatalf("error body is not JSON: %q", rec.Body.String())
|
||||||
|
}
|
||||||
|
if _, ok := body["error"]; !ok {
|
||||||
|
t.Errorf("error body is not OpenAI-shaped: %v", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMissingModelIs400(t *testing.T) {
|
||||||
|
e := newFakeEngine(t, "m")
|
||||||
|
rt := newTestRouter(t, baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"m"}}))
|
||||||
|
|
||||||
|
rec := post(t, rt.Handler(), map[string]any{"messages": []any{}}, nil)
|
||||||
|
if rec.Code != http.StatusBadRequest {
|
||||||
|
t.Errorf("status %d, expected 400", rec.Code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestModelsEndpointListsAliases(t *testing.T) {
|
||||||
|
e := newFakeEngine(t, "llama-3.3-70b")
|
||||||
|
cfg := baseConfig(config.Endpoint{ID: "a", URL: e.srv.URL, Models: []string{"llama-3.3-70b"}})
|
||||||
|
cfg.Models = []config.Model{{Name: "llama-3.3-70b", Aliases: []string{"llama"}}}
|
||||||
|
rt := newTestRouter(t, cfg)
|
||||||
|
|
||||||
|
req := httptest.NewRequest(http.MethodGet, "/v1/models", nil)
|
||||||
|
rec := httptest.NewRecorder()
|
||||||
|
rt.Handler().ServeHTTP(rec, req)
|
||||||
|
|
||||||
|
var out struct {
|
||||||
|
Data []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
} `json:"data"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(rec.Body.Bytes(), &out); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ids := map[string]bool{}
|
||||||
|
for _, d := range out.Data {
|
||||||
|
ids[d.ID] = true
|
||||||
|
}
|
||||||
|
if !ids["llama-3.3-70b"] || !ids["llama"] {
|
||||||
|
t.Errorf("both the model and its alias should be listed: %v", ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInjectIncludeUsagePreservesFields(t *testing.T) {
|
||||||
|
// The one place the router rewrites a request body; nothing may be lost.
|
||||||
|
in := []byte(`{"model":"m","stream":true,"temperature":0.7,"tools":[{"x":1}]}`)
|
||||||
|
out, ok := injectIncludeUsage(in)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("injection declined a well-formed body")
|
||||||
|
}
|
||||||
|
var m map[string]any
|
||||||
|
if err := json.Unmarshal(out, &m); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, k := range []string{"model", "stream", "temperature", "tools"} {
|
||||||
|
if _, present := m[k]; !present {
|
||||||
|
t.Errorf("field %q was dropped", k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
opts, _ := m["stream_options"].(map[string]any)
|
||||||
|
if opts["include_usage"] != true {
|
||||||
|
t.Errorf("include_usage not set: %v", m["stream_options"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestInjectIncludeUsageKeepsExistingOptions(t *testing.T) {
|
||||||
|
in := []byte(`{"model":"m","stream":true,"stream_options":{"something":"else"}}`)
|
||||||
|
out, ok := injectIncludeUsage(in)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("injection declined")
|
||||||
|
}
|
||||||
|
var m map[string]any
|
||||||
|
if err := json.Unmarshal(out, &m); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
opts, ok := m["stream_options"].(map[string]any)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("stream_options is not an object: %v", m["stream_options"])
|
||||||
|
}
|
||||||
|
if opts["something"] != "else" {
|
||||||
|
t.Error("an existing stream_options field was dropped")
|
||||||
|
}
|
||||||
|
if opts["include_usage"] != true {
|
||||||
|
t.Error("include_usage not set")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptPrefixIsBounded(t *testing.T) {
|
||||||
|
long := strings.Repeat("x", prefixBytes*3)
|
||||||
|
b := inferenceBody{Messages: []struct {
|
||||||
|
Role string `json:"role"`
|
||||||
|
Content json.RawMessage `json:"content"`
|
||||||
|
}{{Role: "user", Content: json.RawMessage(`"` + long + `"`)}}}
|
||||||
|
|
||||||
|
if got := len(promptPrefix(&b)); got > prefixBytes {
|
||||||
|
t.Errorf("prefix is %d bytes, over the %d cap", got, prefixBytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSessionHeaderFallsBackToUser(t *testing.T) {
|
||||||
|
req := httptest.NewRequest(http.MethodPost, "/", nil)
|
||||||
|
if got := sessionID(req, "x-session-id", "user-42"); got != "user-42" {
|
||||||
|
t.Errorf("expected the user field to be used, got %q", got)
|
||||||
|
}
|
||||||
|
req.Header.Set("x-session-id", "sess-1")
|
||||||
|
if got := sessionID(req, "x-session-id", "user-42"); got != "sess-1" {
|
||||||
|
t.Errorf("the header should win, got %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
+25
-5
@@ -144,8 +144,7 @@ llmgw/
|
|||||||
│ │ ├── scheduler/ # placement algorithm
|
│ │ ├── scheduler/ # placement algorithm
|
||||||
│ │ ├── reconciler/ # desired vs actual loop
|
│ │ ├── reconciler/ # desired vs actual loop
|
||||||
│ │ └── agentsvc/ # gRPC server
|
│ │ └── agentsvc/ # gRPC server
|
||||||
│ ├── router/
|
│ ├── router/ # HTTP handlers, dispatch, metrics
|
||||||
│ │ ├── openai/ # HTTP handlers, request parsing
|
|
||||||
│ │ ├── proxy/ # streaming proxy to upstream
|
│ │ ├── proxy/ # streaming proxy to upstream
|
||||||
│ │ ├── admission/ # saturation detector, band queues (§7.2)
|
│ │ ├── admission/ # saturation detector, band queues (§7.2)
|
||||||
│ │ ├── scheduler/ # Filter/Scorer/Picker pipeline (§7.3)
|
│ │ ├── scheduler/ # Filter/Scorer/Picker pipeline (§7.3)
|
||||||
@@ -166,6 +165,10 @@ llmgw/
|
|||||||
└── docs/
|
└── docs/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> The OpenAI handlers live in `internal/router` itself rather than an `openai/`
|
||||||
|
> subpackage: they need the dispatch loop, and splitting them apart would have
|
||||||
|
> meant an interface whose only implementation is the router.
|
||||||
|
|
||||||
### 4.1 Libraries
|
### 4.1 Libraries
|
||||||
|
|
||||||
| Need | Choice |
|
| Need | Choice |
|
||||||
@@ -521,18 +524,35 @@ batching to stay efficient, but not enough for the queue to balloon. Set
|
|||||||
`TargetWaitingPerReplica` too low and the GPU starves; too high and queueing delay returns. Both
|
`TargetWaitingPerReplica` too low and the GPU starves; too high and queueing delay returns. Both
|
||||||
thresholds must be tunable per model/profile, and `ctl bench` (§10) should propose them.
|
thresholds must be tunable per model/profile, and `ctl bench` (§10) should propose them.
|
||||||
|
|
||||||
|
**Two lags have to be closed, and they need different answers.** The engine's queue is scraped,
|
||||||
|
so it is up to one interval old; the router's own inflight count is not. But inflight includes
|
||||||
|
*running* requests, and treating those as queue makes a perfectly healthy engine look overloaded
|
||||||
|
every interval. What is actually queued is `inflight − batch capacity`, and batch capacity is not
|
||||||
|
in the metric contract — so llmgw learns it as the high-water mark of observed `running`. The
|
||||||
|
second lag is shorter and subtler: between admission granting a request and the router
|
||||||
|
dispatching it, the request is invisible everywhere. A simultaneous burst can therefore pass the
|
||||||
|
check en masse, which is exactly the batch-job arrival pattern this layer exists to survive. The
|
||||||
|
controller closes it by counting its own outstanding leases against dispatched inflight.
|
||||||
|
|
||||||
#### 7.2.2 Priority bands
|
#### 7.2.2 Priority bands
|
||||||
|
|
||||||
Every API key maps to a band, and the band sets the saturation ceiling:
|
Every API key maps to a band, and the band sets the saturation ceiling:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
bands:
|
bands:
|
||||||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 } # never held back
|
- { name: platinum, priority: 400, saturation_ceiling: 1.00 } # flows to the healthy limit
|
||||||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||||||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||||||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 } # batch jobs
|
- { name: bronze, priority: 100, saturation_ceiling: 0.30 } # batch jobs
|
||||||
```
|
```
|
||||||
|
|
||||||
|
A ceiling of 1.00 means "hold only once the pool is past its healthy limit",
|
||||||
|
not "never hold". Holding even the top band is deliberate: there is nobody above
|
||||||
|
it to protect, so the benefit is a bounded engine queue. A flood dumped straight
|
||||||
|
into a FIFO engine queue makes every request in that band slow, while queueing it
|
||||||
|
at the gateway preserves ordering, TTL and tenant fairness. A band that must
|
||||||
|
never wait — a health probe, say — gets a ceiling above 1 (`1000`).
|
||||||
|
|
||||||
The dispatch loop scans bands from highest to lowest. Once pool saturation exceeds a band's
|
The dispatch loop scans bands from highest to lowest. Once pool saturation exceeds a band's
|
||||||
ceiling, that band's requests wait in the queue. The result: as the cluster fills, batch traffic
|
ceiling, that band's requests wait in the queue. The result: as the cluster fills, batch traffic
|
||||||
slows down first and interactive traffic is protected. (In llm-d's measurements platinum, gold
|
slows down first and interactive traffic is protected. (In llm-d's measurements platinum, gold
|
||||||
@@ -953,8 +973,8 @@ work_dir: /var/lib/llmgw-agent
|
|||||||
- [x] CI: `go test ./...`, race detector
|
- [x] CI: `go test ./...`, race detector
|
||||||
|
|
||||||
### Phase 1 — Static router (2 weeks)
|
### Phase 1 — Static router (2 weeks)
|
||||||
- [ ] OpenAI handlers, streaming proxy, cancellation propagation
|
- [x] OpenAI handlers, streaming proxy, cancellation propagation
|
||||||
- [ ] Static endpoint list from configuration
|
- [x] Static endpoint list from configuration
|
||||||
- [x] **The `Filter`/`Scorer`/`Picker` pipeline (§7.3)** — a plugin pipeline from the start, not one "strategy"
|
- [x] **The `Filter`/`Scorer`/`Picker` pipeline (§7.3)** — a plugin pipeline from the start, not one "strategy"
|
||||||
- [x] `queue`, `kv`, `inflight` scorers; `max-score` and `p2c` pickers
|
- [x] `queue`, `kv`, `inflight` scorers; `max-score` and `p2c` pickers
|
||||||
- [x] Session affinity (§7.5 tier 0) — cheap, an early win
|
- [x] Session affinity (§7.5 tier 0) — cheap, an early win
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ MILESTONES = [
|
|||||||
"against. Exit criterion: sim-engine models batching, KV blocks, prefix "
|
"against. Exit criterion: sim-engine models batching, KV blocks, prefix "
|
||||||
"caching and cancellation, and exposes the §7.6 metric contract."
|
"caching and cancellation, and exposes the §7.6 metric contract."
|
||||||
),
|
),
|
||||||
"due_on": "2026-09-16T23:59:59Z",
|
"due_on": "2026-09-16T12:00:00Z",
|
||||||
"closed": True,
|
"closed": True,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -46,7 +46,7 @@ MILESTONES = [
|
|||||||
"Exit criterion: 1000 concurrent streams behind 3 sim-engines; upstream "
|
"Exit criterion: 1000 concurrent streams behind 3 sim-engines; upstream "
|
||||||
"cancellation on client disconnect verified; router overhead p99 < 2 ms."
|
"cancellation on client disconnect verified; router overhead p99 < 2 ms."
|
||||||
),
|
),
|
||||||
"due_on": "2026-09-30T23:59:59Z",
|
"due_on": "2026-09-30T12:00:00Z",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "v0.3.0 — Phase 1.5: admission control",
|
"title": "v0.3.0 — Phase 1.5: admission control",
|
||||||
@@ -57,7 +57,7 @@ MILESTONES = [
|
|||||||
"bronze band, interactive TTFT p99 in the platinum band stays under 1.5x "
|
"bronze band, interactive TTFT p99 in the platinum band stays under 1.5x "
|
||||||
"its idle value. This test stays in CI permanently."
|
"its idle value. This test stays in CI permanently."
|
||||||
),
|
),
|
||||||
"due_on": "2026-10-07T23:59:59Z",
|
"due_on": "2026-10-07T12:00:00Z",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "v0.4.0 — Phase 2: agent",
|
"title": "v0.4.0 — Phase 2: agent",
|
||||||
@@ -66,7 +66,7 @@ MILESTONES = [
|
|||||||
"the control plane stream.\n\n"
|
"the control plane stream.\n\n"
|
||||||
"Exit criterion: start and stop a model on a real node via ctl."
|
"Exit criterion: start and stop a model on a real node via ctl."
|
||||||
),
|
),
|
||||||
"due_on": "2026-10-21T23:59:59Z",
|
"due_on": "2026-10-21T12:00:00Z",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "v0.5.0 — Phase 3: control plane",
|
"title": "v0.5.0 — Phase 3: control plane",
|
||||||
@@ -76,7 +76,7 @@ MILESTONES = [
|
|||||||
"Exit criterion: shutting down a node moves its replicas elsewhere while "
|
"Exit criterion: shutting down a node moves its replicas elsewhere while "
|
||||||
"the router keeps serving."
|
"the router keeps serving."
|
||||||
),
|
),
|
||||||
"due_on": "2026-11-11T23:59:59Z",
|
"due_on": "2026-11-11T12:00:00Z",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "v0.6.0 — Phase 4: multi-node and fabrics",
|
"title": "v0.6.0 — Phase 4: multi-node and fabrics",
|
||||||
@@ -86,7 +86,7 @@ MILESTONES = [
|
|||||||
"Exit criterion: a 2-node replica on each fabric; killing a node rebuilds "
|
"Exit criterion: a 2-node replica on each fabric; killing a node rebuilds "
|
||||||
"the replica on the same fabric and never spreads it across two."
|
"the replica on the same fabric and never spreads it across two."
|
||||||
),
|
),
|
||||||
"due_on": "2026-11-25T23:59:59Z",
|
"due_on": "2026-11-25T12:00:00Z",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "v1.0.0 — Phase 5: production features",
|
"title": "v1.0.0 — Phase 5: production features",
|
||||||
@@ -95,7 +95,7 @@ MILESTONES = [
|
|||||||
"prefix tree, fallback models and pool spillover, ctl bench, tracing, "
|
"prefix tree, fallback models and pool spillover, ctl bench, tracing, "
|
||||||
"systemd units and documentation."
|
"systemd units and documentation."
|
||||||
),
|
),
|
||||||
"due_on": "2026-12-16T23:59:59Z",
|
"due_on": "2026-12-16T12:00:00Z",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"title": "Phase 6 — v2 ideas",
|
"title": "Phase 6 — v2 ideas",
|
||||||
|
|||||||
@@ -0,0 +1,242 @@
|
|||||||
|
// Package e2e runs the router against real sim-engines over real HTTP.
|
||||||
|
//
|
||||||
|
// These tests exist to check the phase 1 exit criterion from the plan, so they
|
||||||
|
// deliberately use the whole stack: no fakes, no in-process shortcuts beyond
|
||||||
|
// httptest listeners.
|
||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/router"
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/sim"
|
||||||
|
)
|
||||||
|
|
||||||
|
const testModel = "test-model"
|
||||||
|
|
||||||
|
type engine struct {
|
||||||
|
eng *sim.Engine
|
||||||
|
srv *httptest.Server
|
||||||
|
}
|
||||||
|
|
||||||
|
type harness struct {
|
||||||
|
engines []engine
|
||||||
|
router *httptest.Server
|
||||||
|
rt *router.Router
|
||||||
|
}
|
||||||
|
|
||||||
|
// simConfig is fast enough for tests while still exercising queueing.
|
||||||
|
func simConfig(name string, seed int64) sim.Config {
|
||||||
|
c := sim.Default()
|
||||||
|
c.Model = testModel
|
||||||
|
c.Seed = seed
|
||||||
|
c.Tick = time.Millisecond
|
||||||
|
c.MaxNumSeqs = 512
|
||||||
|
c.NumGPUBlocks = 200000
|
||||||
|
c.PrefillTokensPerSec = 5e6
|
||||||
|
c.DecodeTokensPerSec = 2000
|
||||||
|
c.BatchContention = 0
|
||||||
|
_ = name
|
||||||
|
return c
|
||||||
|
}
|
||||||
|
|
||||||
|
// engineOpt tunes the sim-engines a harness starts.
|
||||||
|
type engineOpt func(*sim.Config)
|
||||||
|
|
||||||
|
// engineCapacity is the batch capacity used by withLoadedEngines, chosen to sit
|
||||||
|
// in the range a real vLLM deployment runs at. Making it much smaller would
|
||||||
|
// shift the bottleneck to slot starvation inside the engine, which admission
|
||||||
|
// cannot do anything about — the test would then be measuring the wrong thing.
|
||||||
|
const engineCapacity = 64
|
||||||
|
|
||||||
|
// withLoadedEngines sizes the engines so that a few hundred concurrent requests
|
||||||
|
// genuinely overflow them, which is what the priority isolation test needs.
|
||||||
|
func withLoadedEngines() func(*config.Server) {
|
||||||
|
return nil // marker; see start()
|
||||||
|
}
|
||||||
|
|
||||||
|
// start brings up n sim-engines behind a router.
|
||||||
|
//
|
||||||
|
// A nil tune function is the "small engines" marker, which keeps the call sites
|
||||||
|
// readable without a second variadic parameter.
|
||||||
|
func start(t *testing.T, n int, tune ...func(*config.Server)) *harness {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
small := false
|
||||||
|
kept := tune[:0]
|
||||||
|
for _, f := range tune {
|
||||||
|
if f == nil {
|
||||||
|
small = true
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
kept = append(kept, f)
|
||||||
|
}
|
||||||
|
tune = kept
|
||||||
|
|
||||||
|
h := &harness{}
|
||||||
|
cfg := config.Default()
|
||||||
|
cfg.Router.ScrapeInterval = 200 * time.Millisecond
|
||||||
|
cfg.Router.ScrapeTimeout = 150 * time.Millisecond
|
||||||
|
cfg.Router.StaleAfter = 2 * time.Second
|
||||||
|
|
||||||
|
for i := range n {
|
||||||
|
sc := simConfig(fmt.Sprintf("sim-%d", i), int64(i+1))
|
||||||
|
if small {
|
||||||
|
sc.MaxNumSeqs = engineCapacity
|
||||||
|
sc.DecodeTokensPerSec = 800
|
||||||
|
sc.BatchContention = 0.005
|
||||||
|
}
|
||||||
|
e := sim.New(sc)
|
||||||
|
srv := httptest.NewServer(sim.NewServer(e, slog.New(slog.DiscardHandler)))
|
||||||
|
h.engines = append(h.engines, engine{eng: e, srv: srv})
|
||||||
|
cfg.Endpoints = append(cfg.Endpoints, config.Endpoint{
|
||||||
|
ID: fmt.Sprintf("e%d", i),
|
||||||
|
URL: srv.URL,
|
||||||
|
Models: []string{testModel},
|
||||||
|
Pool: "test",
|
||||||
|
Weight: 1,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
for _, f := range tune {
|
||||||
|
f(&cfg)
|
||||||
|
}
|
||||||
|
|
||||||
|
rt, err := router.New(cfg, slog.New(slog.DiscardHandler))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
rt.Start(ctx)
|
||||||
|
|
||||||
|
h.rt = rt
|
||||||
|
h.router = httptest.NewServer(rt.Handler())
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
h.router.Close()
|
||||||
|
cancel()
|
||||||
|
rt.Stop()
|
||||||
|
for _, e := range h.engines {
|
||||||
|
e.srv.Close()
|
||||||
|
e.eng.Close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
// Let the first scrape land so scheduling starts from real data.
|
||||||
|
h.waitForFreshMetrics(t)
|
||||||
|
return h
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *harness) waitForFreshMetrics(t *testing.T) {
|
||||||
|
t.Helper()
|
||||||
|
deadline := time.Now().Add(5 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
fresh := 0
|
||||||
|
for _, s := range h.rt.Registry().Snapshots(testModel) {
|
||||||
|
if !s.Stale {
|
||||||
|
fresh++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if fresh == len(h.engines) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(20 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatal("endpoint metrics never became fresh")
|
||||||
|
}
|
||||||
|
|
||||||
|
// running counts requests actively generating across all engines.
|
||||||
|
func (h *harness) running() int {
|
||||||
|
n := 0
|
||||||
|
for _, e := range h.engines {
|
||||||
|
n += e.eng.Stats().Running
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *harness) totalRunning() int {
|
||||||
|
n := 0
|
||||||
|
for _, e := range h.engines {
|
||||||
|
s := e.eng.Stats()
|
||||||
|
n += s.Running + s.Waiting
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
// chatBody builds a streaming chat request.
|
||||||
|
func chatBody(prompt string, maxTokens int, stream bool) []byte {
|
||||||
|
b, _ := json.Marshal(map[string]any{
|
||||||
|
"model": testModel,
|
||||||
|
"messages": []map[string]string{{"role": "user", "content": prompt}},
|
||||||
|
"max_tokens": maxTokens,
|
||||||
|
"stream": stream,
|
||||||
|
})
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
// streamRequest sends a streaming request and counts the content chunks.
|
||||||
|
func streamRequest(ctx context.Context, url string, body []byte, hdr map[string]string) (chunks int, ttfb time.Duration, err error) {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url+"/v1/chat/completions", bytes.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0, err
|
||||||
|
}
|
||||||
|
req.Header.Set("Content-Type", "application/json")
|
||||||
|
for k, v := range hdr {
|
||||||
|
req.Header.Set(k, v)
|
||||||
|
}
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
resp, err := http.DefaultClient.Do(req)
|
||||||
|
if err != nil {
|
||||||
|
return 0, 0, err
|
||||||
|
}
|
||||||
|
defer resp.Body.Close()
|
||||||
|
if resp.StatusCode != http.StatusOK {
|
||||||
|
b, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<12))
|
||||||
|
return 0, 0, fmt.Errorf("status %d: %s", resp.StatusCode, b)
|
||||||
|
}
|
||||||
|
|
||||||
|
br := bufio.NewReader(resp.Body)
|
||||||
|
for {
|
||||||
|
line, rerr := br.ReadString('\n')
|
||||||
|
if strings.HasPrefix(line, "data: ") {
|
||||||
|
if chunks == 0 {
|
||||||
|
ttfb = time.Since(start)
|
||||||
|
}
|
||||||
|
if !strings.Contains(line, "[DONE]") {
|
||||||
|
chunks++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if rerr != nil {
|
||||||
|
if rerr == io.EOF {
|
||||||
|
return chunks, ttfb, nil
|
||||||
|
}
|
||||||
|
return chunks, ttfb, rerr
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// envInt reads an integer from the environment, falling back to def.
|
||||||
|
func envInt(key string, def int) int {
|
||||||
|
v := os.Getenv(key)
|
||||||
|
if v == "" {
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
var n int
|
||||||
|
if _, err := fmt.Sscanf(v, "%d", &n); err != nil || n <= 0 {
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
@@ -0,0 +1,203 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"slices"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.mynodes.xyz/ilker/llmgw/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestPriorityIsolation is the phase 1.5 exit criterion.
|
||||||
|
//
|
||||||
|
// While a sustained flood of batch requests runs in the bronze band,
|
||||||
|
// interactive requests in the platinum band must keep their latency. This is
|
||||||
|
// the whole reason admission exists: an engine queue is FIFO, so once a batch
|
||||||
|
// job is in it no amount of clever endpoint selection can protect anyone.
|
||||||
|
//
|
||||||
|
// It is meant to stay in CI permanently.
|
||||||
|
func TestPriorityIsolation(t *testing.T) {
|
||||||
|
workers := envInt("LLMGW_E2E_BATCH", 500)
|
||||||
|
probes := envInt("LLMGW_E2E_PROBES", 40)
|
||||||
|
if testing.Short() {
|
||||||
|
workers, probes = 80, 15
|
||||||
|
}
|
||||||
|
|
||||||
|
h := start(t, 3, func(c *config.Server) {
|
||||||
|
c.Router.Admission.Enabled = true
|
||||||
|
}, withLoadedEngines())
|
||||||
|
|
||||||
|
idle := measureProbes(t, h, probes, "platinum")
|
||||||
|
t.Logf("idle platinum: p50=%v p99=%v", idle.p50.Round(time.Millisecond), idle.p99.Round(time.Millisecond))
|
||||||
|
|
||||||
|
load := startBronzeLoad(t, h, workers, "bronze")
|
||||||
|
defer load.stop()
|
||||||
|
|
||||||
|
loaded := measureProbes(t, h, probes, "platinum")
|
||||||
|
t.Logf("loaded platinum: p50=%v p99=%v (%s)",
|
||||||
|
loaded.p50.Round(time.Millisecond), loaded.p99.Round(time.Millisecond), load.summary())
|
||||||
|
|
||||||
|
// The criterion: interactive p99 under load stays within 1.5x of idle.
|
||||||
|
// The floor keeps a few-millisecond baseline from making the bound absurd.
|
||||||
|
budget := max(time.Duration(float64(idle.p99)*1.5), idle.p99+50*time.Millisecond)
|
||||||
|
t.Logf("platinum p99 %v against a %v budget (%.2fx idle)",
|
||||||
|
loaded.p99.Round(time.Millisecond), budget.Round(time.Millisecond),
|
||||||
|
float64(loaded.p99)/float64(max(idle.p99, time.Microsecond)))
|
||||||
|
|
||||||
|
if loaded.p99 > budget {
|
||||||
|
t.Errorf("interactive p99 rose to %v under bronze load, over the %v budget; "+
|
||||||
|
"admission control is not protecting the high band",
|
||||||
|
loaded.p99.Round(time.Millisecond), budget.Round(time.Millisecond))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestAdmissionDisabledShowsTheProblem documents why the layer exists: the same
|
||||||
|
// load does wreck interactive latency once admission is switched off. If this
|
||||||
|
// ever stops showing degradation, the load generator has stopped being a load
|
||||||
|
// and the test above proves less than it appears to.
|
||||||
|
func TestAdmissionDisabledShowsTheProblem(t *testing.T) {
|
||||||
|
if testing.Short() {
|
||||||
|
t.Skip("load-dependent")
|
||||||
|
}
|
||||||
|
workers := envInt("LLMGW_E2E_BATCH", 500)
|
||||||
|
|
||||||
|
h := start(t, 3, func(c *config.Server) {
|
||||||
|
c.Router.Admission.Enabled = false
|
||||||
|
}, withLoadedEngines())
|
||||||
|
|
||||||
|
idle := measureProbes(t, h, 20, "")
|
||||||
|
|
||||||
|
load := startBronzeLoad(t, h, workers, "")
|
||||||
|
defer load.stop()
|
||||||
|
|
||||||
|
loaded := measureProbes(t, h, 20, "")
|
||||||
|
ratio := float64(loaded.p99) / float64(max(idle.p99, time.Microsecond))
|
||||||
|
t.Logf("without admission: idle p99=%v loaded p99=%v (%.1fx)",
|
||||||
|
idle.p99.Round(time.Millisecond), loaded.p99.Round(time.Millisecond), ratio)
|
||||||
|
|
||||||
|
if ratio < 2 {
|
||||||
|
t.Skipf("the load did not degrade latency (%.1fx); this machine is too fast "+
|
||||||
|
"for %d workers, so the isolation test proves less than it should", ratio, workers)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- load generation ---
|
||||||
|
|
||||||
|
// bronzeLoad is a sustained batch workload.
|
||||||
|
//
|
||||||
|
// Workers loop rather than firing once: a one-shot burst drains while the
|
||||||
|
// probes are still running, and the measurement then catches a recovering
|
||||||
|
// cluster instead of a loaded one.
|
||||||
|
type bronzeLoad struct {
|
||||||
|
cancel context.CancelFunc
|
||||||
|
wg sync.WaitGroup
|
||||||
|
done atomic.Int64
|
||||||
|
failed atomic.Int64
|
||||||
|
firstEr atomic.Value
|
||||||
|
}
|
||||||
|
|
||||||
|
func startBronzeLoad(t *testing.T, h *harness, workers int, band string) *bronzeLoad {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
l := &bronzeLoad{cancel: cancel}
|
||||||
|
|
||||||
|
for i := range workers {
|
||||||
|
l.wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer l.wg.Done()
|
||||||
|
hdr := map[string]string{"x-session-id": fmt.Sprintf("batch-%d", i%8)}
|
||||||
|
if band != "" {
|
||||||
|
hdr["x-llmgw-band"] = band
|
||||||
|
}
|
||||||
|
for ctx.Err() == nil {
|
||||||
|
_, _, err := streamRequest(ctx, h.router.URL,
|
||||||
|
chatBody(fmt.Sprintf("batch job %d", i), 200, true), hdr)
|
||||||
|
switch {
|
||||||
|
case err == nil:
|
||||||
|
l.done.Add(1)
|
||||||
|
case ctx.Err() != nil: // shutting down, expected
|
||||||
|
default:
|
||||||
|
l.failed.Add(1)
|
||||||
|
l.firstEr.CompareAndSwap(nil, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
l.waitUntilBusy(t, h)
|
||||||
|
return l
|
||||||
|
}
|
||||||
|
|
||||||
|
// waitUntilBusy blocks until the engines are demonstrably working, so probes
|
||||||
|
// are not taken while the load is still ramping up.
|
||||||
|
//
|
||||||
|
// It waits on engines being busy rather than full: when admission is doing its
|
||||||
|
// job the pool is deliberately held below capacity, so a fullness check would
|
||||||
|
// time out exactly when the system is behaving correctly.
|
||||||
|
func (l *bronzeLoad) waitUntilBusy(t *testing.T, h *harness) {
|
||||||
|
t.Helper()
|
||||||
|
const busyFor = 500 * time.Millisecond
|
||||||
|
|
||||||
|
deadline := time.Now().Add(20 * time.Second)
|
||||||
|
var since time.Time
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if h.running() > 0 {
|
||||||
|
if since.IsZero() {
|
||||||
|
since = time.Now()
|
||||||
|
} else if time.Since(since) >= busyFor {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
since = time.Time{}
|
||||||
|
}
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatalf("engines never became busy; %s", l.summary())
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *bronzeLoad) summary() string {
|
||||||
|
return fmt.Sprintf("bronze: completed=%d failed=%d firstErr=%v",
|
||||||
|
l.done.Load(), l.failed.Load(), l.firstEr.Load())
|
||||||
|
}
|
||||||
|
|
||||||
|
func (l *bronzeLoad) stop() {
|
||||||
|
l.cancel()
|
||||||
|
l.wg.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- probes ---
|
||||||
|
|
||||||
|
type latency struct{ p50, p99 time.Duration }
|
||||||
|
|
||||||
|
// measureProbes sends sequential short requests and reports their latency.
|
||||||
|
// Sequential on purpose: these stand in for a single interactive user, and
|
||||||
|
// concurrency would measure something else.
|
||||||
|
func measureProbes(t *testing.T, h *harness, n int, band string) latency {
|
||||||
|
t.Helper()
|
||||||
|
hdr := map[string]string{"x-session-id": "probe"}
|
||||||
|
if band != "" {
|
||||||
|
hdr["x-llmgw-band"] = band
|
||||||
|
}
|
||||||
|
|
||||||
|
samples := make([]time.Duration, 0, n)
|
||||||
|
for range n {
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||||
|
start := time.Now()
|
||||||
|
_, _, err := streamRequest(ctx, h.router.URL, chatBody("ping", 4, true), hdr)
|
||||||
|
cancel()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("probe failed: %v", err)
|
||||||
|
}
|
||||||
|
samples = append(samples, time.Since(start))
|
||||||
|
}
|
||||||
|
slices.Sort(samples)
|
||||||
|
return latency{
|
||||||
|
p50: samples[len(samples)/2],
|
||||||
|
p99: samples[min(int(float64(len(samples))*0.99), len(samples)-1)],
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package e2e
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"slices"
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func bytesReader(b []byte) *bytes.Reader { return bytes.NewReader(b) }
|
||||||
|
|
||||||
|
func drainBody(resp *http.Response) {
|
||||||
|
_, _ = io.Copy(io.Discard, resp.Body)
|
||||||
|
_ = resp.Body.Close()
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestConcurrentStreams is the first half of the phase 1 exit criterion:
|
||||||
|
// 1000 concurrent streams behind 3 sim-engines, all of them completing.
|
||||||
|
func TestConcurrentStreams(t *testing.T) {
|
||||||
|
concurrency := envInt("LLMGW_E2E_CONCURRENCY", 1000)
|
||||||
|
if testing.Short() {
|
||||||
|
concurrency = 50
|
||||||
|
}
|
||||||
|
const tokens = 16
|
||||||
|
|
||||||
|
h := start(t, 3)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
var (
|
||||||
|
ok atomic.Int64
|
||||||
|
short atomic.Int64
|
||||||
|
failed atomic.Int64
|
||||||
|
firstErr atomic.Value
|
||||||
|
wg sync.WaitGroup
|
||||||
|
)
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
for i := range concurrency {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
n, _, err := streamRequest(ctx, h.router.URL,
|
||||||
|
chatBody(fmt.Sprintf("request %d", i), tokens, true),
|
||||||
|
map[string]string{"x-session-id": fmt.Sprintf("s%d", i%97)})
|
||||||
|
switch {
|
||||||
|
case err != nil:
|
||||||
|
failed.Add(1)
|
||||||
|
firstErr.CompareAndSwap(nil, err)
|
||||||
|
case n < tokens:
|
||||||
|
short.Add(1)
|
||||||
|
default:
|
||||||
|
ok.Add(1)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
|
||||||
|
t.Logf("%d streams in %v (%.0f req/s): ok=%d short=%d failed=%d",
|
||||||
|
concurrency, elapsed.Round(time.Millisecond),
|
||||||
|
float64(concurrency)/elapsed.Seconds(), ok.Load(), short.Load(), failed.Load())
|
||||||
|
|
||||||
|
if f := failed.Load(); f > 0 {
|
||||||
|
t.Fatalf("%d of %d streams failed; first error: %v", f, concurrency, firstErr.Load())
|
||||||
|
}
|
||||||
|
if s := short.Load(); s > 0 {
|
||||||
|
t.Errorf("%d streams ended before %d tokens", s, tokens)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Load must actually have been spread; a single engine taking everything
|
||||||
|
// would pass the checks above while defeating the point of the router.
|
||||||
|
used := 0
|
||||||
|
for _, e := range h.engines {
|
||||||
|
if st := e.eng.Stats(); st.PrefixQueries > 0 || st.CachedBlocks > 0 {
|
||||||
|
used++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if used < 2 {
|
||||||
|
t.Errorf("only %d of %d engines saw traffic", used, len(h.engines))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestClientDisconnectStopsUpstream is the second half of the exit criterion:
|
||||||
|
// when a client goes away, the GPU must stop working on its request.
|
||||||
|
func TestClientDisconnectStopsUpstream(t *testing.T) {
|
||||||
|
const streams = 40
|
||||||
|
|
||||||
|
h := start(t, 3)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithCancel(context.Background())
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := range streams {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
// Long generations, so every request is still running when cancelled.
|
||||||
|
_, _, _ = streamRequest(ctx, h.router.URL,
|
||||||
|
chatBody(fmt.Sprintf("long request %d", i), 100000, true), nil)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wait until the engines are actually busy before pulling the plug.
|
||||||
|
deadline := time.Now().Add(10 * time.Second)
|
||||||
|
for h.totalRunning() < streams && time.Now().Before(deadline) {
|
||||||
|
time.Sleep(10 * time.Millisecond)
|
||||||
|
}
|
||||||
|
if busy := h.totalRunning(); busy < streams {
|
||||||
|
t.Fatalf("only %d of %d requests reached the engines", busy, streams)
|
||||||
|
}
|
||||||
|
|
||||||
|
cancel()
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
// The engines must drain: cancellation has to propagate router -> engine.
|
||||||
|
deadline = time.Now().Add(15 * time.Second)
|
||||||
|
for time.Now().Before(deadline) {
|
||||||
|
if h.totalRunning() == 0 {
|
||||||
|
// And the KV blocks must come back, not just the request slots.
|
||||||
|
for _, e := range h.engines {
|
||||||
|
if u := e.eng.Stats().KVCacheUsage; u != 0 {
|
||||||
|
t.Fatalf("KV cache still held after cancellation: %v", u)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
time.Sleep(20 * time.Millisecond)
|
||||||
|
}
|
||||||
|
t.Fatalf("engines still working %d requests after every client disconnected", h.totalRunning())
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRouterOverhead is the third part of the exit criterion: the router should
|
||||||
|
// add little latency of its own.
|
||||||
|
//
|
||||||
|
// It measures the same request against an engine directly and through the
|
||||||
|
// router, and reports the p99 difference. The assertion is deliberately loose,
|
||||||
|
// because a shared CI machine cannot hold a 2 ms bound reliably; set
|
||||||
|
// LLMGW_E2E_STRICT=1 on quiet hardware to enforce the plan's actual target.
|
||||||
|
func TestRouterOverhead(t *testing.T) {
|
||||||
|
samples := envInt("LLMGW_E2E_SAMPLES", 300)
|
||||||
|
if testing.Short() {
|
||||||
|
samples = 50
|
||||||
|
}
|
||||||
|
|
||||||
|
h := start(t, 1)
|
||||||
|
direct := h.engines[0].srv.URL
|
||||||
|
body := chatBody("overhead probe", 1, false)
|
||||||
|
|
||||||
|
measure := func(url string) []time.Duration {
|
||||||
|
out := make([]time.Duration, 0, samples)
|
||||||
|
for range samples {
|
||||||
|
start := time.Now()
|
||||||
|
resp, err := http.Post(url+"/v1/chat/completions", "application/json", bytesReader(body)) //nolint:noctx // test
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
drainBody(resp)
|
||||||
|
out = append(out, time.Since(start))
|
||||||
|
}
|
||||||
|
slices.Sort(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// Warm both paths so connection setup does not land in the samples.
|
||||||
|
measure(direct)
|
||||||
|
measure(h.router.URL)
|
||||||
|
|
||||||
|
d := measure(direct)
|
||||||
|
r := measure(h.router.URL)
|
||||||
|
|
||||||
|
p := func(s []time.Duration, q float64) time.Duration {
|
||||||
|
return s[min(int(float64(len(s))*q), len(s)-1)]
|
||||||
|
}
|
||||||
|
overheadP50 := p(r, 0.50) - p(d, 0.50)
|
||||||
|
overheadP99 := p(r, 0.99) - p(d, 0.99)
|
||||||
|
|
||||||
|
t.Logf("direct p50=%v p99=%v | routed p50=%v p99=%v | overhead p50=%v p99=%v",
|
||||||
|
p(d, 0.50).Round(time.Microsecond), p(d, 0.99).Round(time.Microsecond),
|
||||||
|
p(r, 0.50).Round(time.Microsecond), p(r, 0.99).Round(time.Microsecond),
|
||||||
|
overheadP50.Round(time.Microsecond), overheadP99.Round(time.Microsecond))
|
||||||
|
|
||||||
|
limit := 25 * time.Millisecond
|
||||||
|
if os.Getenv("LLMGW_E2E_STRICT") == "1" {
|
||||||
|
limit = 2 * time.Millisecond
|
||||||
|
}
|
||||||
|
if overheadP99 > limit {
|
||||||
|
t.Errorf("router added %v at p99, over the %v budget", overheadP99, limit)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user