ilkerandClaude Opus 5
dc38dd6096
CI / test (push) Successful in 48s
Inserted between quota checks and scheduling: admission answers when a request may proceed, the scheduler still answers where it goes. Keeping them apart is the point — an engine queue is FIFO and knows nothing about tenants, so once a batch job is in it, no endpoint selection rescues the request behind it. - saturation detector: one number per pool, the worse of queue depth and KV occupancy, since either alone misleads. - priority bands with per-band saturation ceilings, so the cluster degrades from the bottom up. - tenant round-robin within a band: one caller sending a thousand requests takes one slot in the rotation, not a thousand. - bounded queues with capacity (503) and TTL (504); cancelled requests leave the queue instead of holding capacity. - metrics for queue depth, wait time, rejections and pool saturation. Exit criterion, measured in test/e2e: with 500 concurrent bronze requests running continuously, interactive p99 in the platinum band stays at 1.04-1.24x idle against a 1.5x budget. With admission off the same load pushes it to 55-60x. Both directions are asserted, so the test fails if admission regresses and also if the load generator stops being a real load. Two lags had to be closed, and they needed different answers. Engine metrics are scraped and lag an interval; router inflight does not, but it counts running requests, and treating those as queue makes a healthy engine look overloaded every interval. What is actually queued is inflight minus batch capacity, and capacity is not in the metric contract, so it is learned as the high-water mark of observed running. Separately, between admission granting a request and the router dispatching it the request is invisible everywhere, so a simultaneous burst passed the check en masse; the controller now counts its own outstanding leases against dispatched inflight. One plan correction: a saturation ceiling of 1.0 was documented as "never held back". It is not — it means "flow up to the healthy limit". A band that must never wait needs a ceiling above 1. Plan §7.2.2 updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
106 lines
3.8 KiB
YAML
106 lines
3.8 KiB
YAML
# llmgw router configuration (phase 1).
|
|
#
|
|
# The endpoint list is static here. From phase 3 it comes from the control
|
|
# plane instead and this section disappears; everything else stays.
|
|
#
|
|
# Any field can be overridden from the environment with an LLMGW_ prefix and
|
|
# __ for nesting: LLMGW_ROUTER__RETRIES=3
|
|
|
|
listen: ":8080" # client-facing OpenAI API
|
|
admin_listen: ":9090" # metrics, health, pprof — do not expose this publicly
|
|
|
|
router:
|
|
scrape_interval: 2s
|
|
scrape_timeout: 1s
|
|
# How long an endpoint may go unscraped before it is scored down. It is never
|
|
# dropped for staleness alone: if the whole pool went stale there would be
|
|
# nowhere left to send anything.
|
|
stale_after: 10s
|
|
|
|
# Additional endpoints to try after a failure, and only while nothing has
|
|
# reached the client yet.
|
|
retries: 2
|
|
# How long a failing endpoint leaves the candidate pool. A long fallback list
|
|
# with no cooldown turns one endpoint's rate limit into a cluster-wide retry
|
|
# storm.
|
|
cooldown: 15s
|
|
# Consecutive failures that open the breaker. Set to 1 for pure cooldown
|
|
# behaviour.
|
|
failure_threshold: 3
|
|
|
|
default_profile: chat # chat | batch | long-context | debug
|
|
session_header: x-session-id # affinity key; falls back to the `user` field
|
|
# Ask engines for usage even when the client did not, so token accounting
|
|
# works regardless of client behaviour. The extra chunk is dropped on the way
|
|
# back to a client that did not ask for it.
|
|
inject_usage: true
|
|
max_body_bytes: 33554432
|
|
|
|
timeouts:
|
|
dial: 5s
|
|
response_header: 60s
|
|
first_token: 120s
|
|
idle: 60s # between streamed tokens; catches a wedged engine
|
|
total: 30m
|
|
|
|
# Flow control: decides *when* a request may proceed, before scheduling
|
|
# decides where it goes. Without it a batch workload starves interactive
|
|
# traffic, because an engine queue is FIFO and knows nothing about tenants.
|
|
admission:
|
|
enabled: true
|
|
# Until authentication lands, the band comes from a header and is trusted.
|
|
# Do not expose this port to callers who should not be able to promote
|
|
# themselves.
|
|
band_header: x-llmgw-band
|
|
default_band: gold
|
|
# A band waits once pool saturation passes its ceiling, so the cluster
|
|
# degrades from the bottom up. A ceiling of 1.0 means "flow up to the
|
|
# healthy limit"; use a value above 1 for a band that must never wait.
|
|
bands:
|
|
- { name: platinum, priority: 400, saturation_ceiling: 1.00 }
|
|
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
|
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
|
- { name: bronze, priority: 100, saturation_ceiling: 0.30 }
|
|
# Drop a request that has waited this long: the client has usually given up,
|
|
# and sending the GPU work nobody is waiting for helps no one.
|
|
queue_ttl: 30s
|
|
# Reject immediately once a band's queue is this deep, rather than letting
|
|
# it accumulate silently.
|
|
queue_capacity_per_band: 2000
|
|
interval: 10ms
|
|
|
|
# What the scorers normalize against. Measure these with `ctl bench` rather
|
|
# than guessing: target_waiting in particular governs how the whole system
|
|
# behaves under load. Too low starves the GPU, too high brings back queueing
|
|
# delay.
|
|
scoring:
|
|
target_waiting: 4
|
|
target_inflight: 8
|
|
kv_ceiling: 0.9
|
|
session_load_factor: 1.25
|
|
|
|
models:
|
|
- name: llama-3.3-70b
|
|
aliases: [llama]
|
|
fallbacks: [qwen2.5-32b]
|
|
routing_profile: chat
|
|
|
|
- name: qwen2.5-32b
|
|
aliases: [qwen]
|
|
routing_profile: batch
|
|
|
|
endpoints:
|
|
- id: gpu-a-01
|
|
url: http://gpu-a-01:8000
|
|
models: [llama-3.3-70b]
|
|
pool: ib-a/h100-80g
|
|
weight: 100 # from ctl bench; do not estimate it by hand
|
|
priority: 100 # higher pools are tried first
|
|
|
|
- id: gpu-b-01
|
|
url: http://gpu-b-01:8000
|
|
models: [llama-3.3-70b, qwen2.5-32b]
|
|
pool: ib-b/a100-80g
|
|
weight: 45
|
|
priority: 50
|