Comments, doc comments, test names, error and log messages, CLI help, README, Makefile, CI step names and the design document are now all in English. No behaviour changes; the full suite passes under the race detector. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1079 lines
53 KiB
Markdown
1079 lines
53 KiB
Markdown
# llmgw — A Single-Binary LLM Gateway for Multi-Fabric InfiniBand and Heterogeneous GPUs
|
||
|
||
> Status: Design / Plan v0.2
|
||
> What changed in v0.2: existing LLM gateway and router projects were surveyed (§2.5) and
|
||
> their critical designs folded in — above all **separating admission from scheduling** (§7.2),
|
||
> the plugin-based **filter → score → pick** pipeline (§7.3), and the **three-tier model of
|
||
> prefix/KV cache awareness** (§7.5).
|
||
> Goal: one Go binary (running under different roles) that serves the LLMs on our own GPU
|
||
> servers behind a single OpenAI-compatible endpoint. Behind it sit several independent
|
||
> InfiniBand fabrics, nodes without IB, and different GPU types.
|
||
|
||
---
|
||
|
||
## 1. Scope
|
||
|
||
### 1.1 In scope
|
||
- OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`), including SSE streaming.
|
||
- An **agent** on every GPU node: GPU and IB discovery; starting, supervising and stopping inference engine processes (vLLM / SGLang).
|
||
- **Control plane**: node inventory, model catalog, replica placement, state tracking.
|
||
- **Router**, in two layers: **admission** (priority bands, tenant fairness, queueing under
|
||
saturation) and **scheduling** (plugin-based filter/score/pick with queue, KV-cache and
|
||
prefix awareness), plus fallback, retry, cooldown and circuit breaking.
|
||
- Fabric awareness: a multi-node replica **never** spans two fabrics; a TP group **never** mixes GPU types.
|
||
- API keys, rate limits, token accounting, Prometheus metrics.
|
||
|
||
### 1.2 Out of scope (at least for v1)
|
||
- Inference itself. Go only orchestrates and proxies; vLLM/SGLang run the model.
|
||
- A Kubernetes operator (bare metal / systemd is the target; K8s can come later).
|
||
- Prefill/decode disaggregation (a v2 goal, see §12).
|
||
- External providers (OpenAI, Anthropic) — the architecture stays open to them, but not in v1.
|
||
|
||
---
|
||
|
||
## 2. Terminology
|
||
|
||
| Term | Meaning |
|
||
|---|---|
|
||
| **Node** | A physical server running the agent. |
|
||
| **Fabric / Island** | An independent IB fabric. Nodes without IB belong to the `none` island and each counts as its own single-node island. |
|
||
| **GPU Class** | A group of GPUs with the same model and VRAM (e.g. `h100-80g`, `a100-40g`, `l40s-48g`). |
|
||
| **Pool** | A (fabric, gpu class) pair. The unit of placement. |
|
||
| **Model** | A logical model in the catalog (e.g. `llama-3.3-70b`). This is the name users see. |
|
||
| **Profile** | How a model runs on a particular GPU class (TP, PP, quantization, max length, engine arguments). |
|
||
| **Replica** | One running copy of a model. May span 1..N nodes. This is the unit the router sees. |
|
||
| **Shard** | A replica's part on a single node (one engine process). |
|
||
| **Routing Profile** | The scorer set and weights the router uses for a class of request (chat / batch / long-context). Not to be confused with the model **Profile** above. |
|
||
| **Band** | A priority class (platinum/gold/silver/bronze). The unit of admission decisions. |
|
||
| **Block** | The unit of KV cache accounting; a fixed number of tokens (16 in vLLM, 64 by default in the GIE indexer). |
|
||
|
||
---
|
||
|
||
## 2.5 Reference Implementations and What We Took
|
||
|
||
Rather than design from scratch, we inherited the problems already solved by mature projects
|
||
in this space.
|
||
|
||
| Project | What it is | What llmgw takes from it |
|
||
|---|---|---|
|
||
| **Gateway API Inference Extension (EPP)** | The K8s standard; an Endpoint Picker running over Envoy `ext_proc` | Admission/scheduling separation, the plugin-based `filter → score → pick` pipeline, the **model server metric contract** (§7.6) |
|
||
| **llm-d** | "Well-lit paths" built on GIE | KV-event-based **precise** prefix index, priority bands with tenant fairness, the saturation detector |
|
||
| **AIBrix** (a vLLM project) | An Envoy ext_proc gateway plus a heterogeneous GPU optimizer | The **heterogeneous GPU SLO/cost model** (§6.2.1), queueing at the gateway (`slo-least-load-pulling`), disabling prefix affinity under load imbalance |
|
||
| **NVIDIA Dynamo** | Radix-tree KV router plus SLA planner | The shape of the cost function (§7.4) and `router_temperature` softmax sampling against herding |
|
||
| **SGLang router** | Cache-aware load balancing over an approximate radix tree | v1's prefix answer: approximate tree plus threshold logic (§7.5 tier 1) |
|
||
| **vLLM production-stack / vllm-project/router** | vLLM's own routers | Session affinity (measured 96% vs 93% cache hit), consistent hashing |
|
||
| **LiteLLM** | The most widely deployed multi-provider proxy | Virtual key → team/project → budget hierarchy, cooldowns, keeping spend writes off the request path, budget-triggered fallback |
|
||
| **Envoy AI Gateway** | An Envoy-based AI gateway (v1.0, June 2026) | Token-based (not merely request-based) rate limiting |
|
||
|
||
What we deliberately **did not** take: a Kubernetes dependency (bare metal is the target), the
|
||
Envoy `ext_proc` architecture (single binary is the goal — we write the router directly in Go),
|
||
semantic routing / LLM-based model selection (out of scope), and external provider management
|
||
(not in v1).
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
┌───────────────────────────────────────────┐
|
||
Clients ────────► │ llmgw server │
|
||
(OpenAI API) │ ┌────────────┐ ┌──────────────────────┐ │
|
||
│ │ Router │ │ Control Plane │ │
|
||
│ │ (data) │◄─┤ - Inventory │ │
|
||
│ │ │ │ - Catalog │ │
|
||
│ └─────┬──────┘ │ - Scheduler │ │
|
||
│ │ │ - Store (SQLite) │ │
|
||
│ │ └─────────▲────────────┘ │
|
||
└────────┼───────────────────┼──────────────┘
|
||
HTTP (eth) │ │ gRPC stream (mTLS, agent dials out)
|
||
┌────────────────────┼───────────────────┼─────────────────────┐
|
||
▼ ▼ │ ▼
|
||
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
|
||
│ llmgw │ IB-A │ llmgw │ │ llmgw │ IB-B │ llmgw │
|
||
│ agent │◄──────►│ agent │ │ agent │◄───────►│ agent │ ...
|
||
│ vLLM rank0│ NCCL │ vLLM rank1│ │ SGLang │ │ (no IB) │
|
||
└───────────┘ └───────────┘ └───────────┘ └───────────┘
|
||
Fabric A island Fabric B island none island
|
||
```
|
||
|
||
### 3.1 One binary, several roles
|
||
|
||
```
|
||
llmgw server --config server.yaml # router + control plane (same process in v1)
|
||
llmgw agent --config agent.yaml # on every GPU node
|
||
llmgw ctl ... # CLI: list nodes, deploy models, drain, ...
|
||
llmgw sim-engine # fake OpenAI-compatible engine for testing
|
||
```
|
||
|
||
`router` and `control` must eventually be able to run as separate processes
|
||
(`llmgw server --roles=router`), so the dependency between them goes through an interface
|
||
(see §6.4).
|
||
|
||
### 3.2 Build note
|
||
`go-nvml` requires cgo. Put the agent's NVML code behind a `//go:build nvml` tag and use a
|
||
fake / `nvidia-smi`-based discovery path in untagged builds. That keeps the server side
|
||
cgo-free and lets development machines without GPUs run the tests.
|
||
|
||
---
|
||
|
||
## 4. Project Layout
|
||
|
||
```
|
||
llmgw/
|
||
├── cmd/llmgw/main.go # cobra root; subcommands
|
||
├── api/
|
||
│ └── proto/llmgw/v1/*.proto # agent <-> control plane
|
||
├── internal/
|
||
│ ├── config/ # YAML + env via koanf
|
||
│ ├── agent/
|
||
│ │ ├── agent.go # main loop, control plane connection
|
||
│ │ ├── discovery/
|
||
│ │ │ ├── gpu_nvml.go # //go:build nvml
|
||
│ │ │ ├── gpu_fallback.go
|
||
│ │ │ ├── ib.go # reads /sys/class/infiniband
|
||
│ │ │ └── topology.go # NVLink, PCIe, NUMA, GPU<->HCA affinity
|
||
│ │ ├── supervisor/ # engine process management (exec, restart, logs)
|
||
│ │ ├── engines/
|
||
│ │ │ ├── engine.go # Engine interface
|
||
│ │ │ ├── vllm.go
|
||
│ │ │ └── sglang.go
|
||
│ │ └── health/ # polls engine /health and /metrics
|
||
│ ├── control/
|
||
│ │ ├── inventory/ # node and GPU state
|
||
│ │ ├── catalog/ # model and profile definitions
|
||
│ │ ├── scheduler/ # placement algorithm
|
||
│ │ ├── reconciler/ # desired vs actual loop
|
||
│ │ └── agentsvc/ # gRPC server
|
||
│ ├── router/
|
||
│ │ ├── openai/ # HTTP handlers, request parsing
|
||
│ │ ├── proxy/ # streaming proxy to upstream
|
||
│ │ ├── admission/ # saturation detector, band queues (§7.2)
|
||
│ │ ├── scheduler/ # Filter/Scorer/Picker pipeline (§7.3)
|
||
│ │ ├── prefix/ # tier 0 session hash, tier 1 approx tree, tier 2 kv index
|
||
│ │ ├── endpoint/ # live endpoint state, inflight, staleness
|
||
│ │ ├── breaker/ # circuit breaker + cooldown
|
||
│ │ └── scrape/ # reads engine /metrics (the §7.6 contract)
|
||
│ ├── auth/ # API keys, org/team/project, budgets
|
||
│ ├── ratelimit/ # token bucket (requests + tokens)
|
||
│ ├── usage/ # token counting and recording
|
||
│ ├── store/ # SQLite (modernc.org/sqlite), migrations
|
||
│ ├── obs/ # slog, prometheus, otel
|
||
│ └── sim/ # fake engine
|
||
├── deploy/
|
||
│ ├── systemd/
|
||
│ └── examples/
|
||
├── test/e2e/
|
||
└── docs/
|
||
```
|
||
|
||
### 4.1 Libraries
|
||
|
||
| Need | Choice |
|
||
|---|---|
|
||
| CLI | `github.com/spf13/cobra` |
|
||
| Config | `github.com/knadh/koanf/v2` |
|
||
| RPC | `connectrpc.com/connect` (gRPC compatible, HTTP/2, easy to debug) |
|
||
| GPU | `github.com/NVIDIA/go-nvml` |
|
||
| DB | `modernc.org/sqlite` (cgo-free) + `pressly/goose` migrations |
|
||
| Metrics | `github.com/prometheus/client_golang`, parsing via `github.com/prometheus/common/expfmt` |
|
||
| Tracing | `go.opentelemetry.io/otel` |
|
||
| Logging | `log/slog` (stdlib) |
|
||
| Rate limiting | `golang.org/x/time/rate` (plus our own bucket for token limits) |
|
||
| Hashing | `github.com/cespare/xxhash/v2` |
|
||
| HA (v2) | `github.com/hashicorp/raft` or etcd |
|
||
| Testing | stdlib `testing`, `github.com/stretchr/testify` (optional) |
|
||
|
||
---
|
||
|
||
## 5. Agent
|
||
|
||
### 5.1 GPU discovery
|
||
For every GPU:
|
||
- index, UUID, name, VRAM, compute capability (for FP8 and similar support), driver/CUDA version
|
||
- NUMA node, PCIe bus id
|
||
- NVLink peers (NVML `GetNvLinkRemotePciInfo`) → which GPUs pair efficiently for TP within a node
|
||
- health: XID errors, ECC, temperature, memory in use (are there non-engine processes?)
|
||
|
||
`GPUClass` is derived automatically: `normalize(name) + "-" + round(vram_gb) + "g"` → `h100-80g`.
|
||
It can be overridden in configuration.
|
||
|
||
### 5.2 IB discovery and fabric identity
|
||
From `/sys/class/infiniband/<hca>/ports/<n>/`:
|
||
- `state` (ACTIVE?), `phys_state`, `rate`, `link_layer` (InfiniBand / Ethernet → RoCE)
|
||
- `gids/0` → subnet prefix
|
||
- `sm_lid`, `lid`
|
||
- the HCA's PCIe address and NUMA node → GPU affinity
|
||
|
||
**How is fabric identity determined?** Automatic detection is not reliable, because OpenSM's
|
||
default subnet prefix (`fe80::`) is the same on every fabric. In order of preference:
|
||
|
||
1. An explicit label in the agent config: `fabrics: [{name: ib-a, hcas: [mlx5_0, mlx5_1]}]` **(recommended)**
|
||
2. If every OpenSM has a unique `subnet_prefix`, automatic mapping from the GID prefix (a `prefix → fabric` table in the server config)
|
||
3. Otherwise the node is marked `fabric=unknown` and **excluded from multi-node placement**.
|
||
|
||
Bonus verification (v1.1): for two nodes claiming the same fabric, the agents verify
|
||
reachability with a short `ibv_rc_pingpong`-style test or `ibping`. A mismatch raises an alert.
|
||
|
||
A node may have HCAs attached to more than one fabric, and the model must allow it
|
||
(`Node.Fabrics []FabricAttachment`). A replica, however, picks exactly one fabric and uses only
|
||
that fabric's HCAs.
|
||
|
||
### 5.3 Engine interface
|
||
|
||
```go
|
||
type Engine interface {
|
||
Name() string // "vllm", "sglang"
|
||
Validate(p Profile, gpus []GPU) error // compute capability, quantization fit, ...
|
||
BuildCommand(spec ShardSpec) (*exec.Cmd, error) // builds arguments and environment
|
||
HealthURL(spec ShardSpec) string
|
||
MetricsURL(spec ShardSpec) string
|
||
ParseMetrics(r io.Reader) (EngineStats, error) // engine-specific metric names
|
||
}
|
||
|
||
type ShardSpec struct {
|
||
ReplicaID string
|
||
Rank int // 0 = leader (the one serving HTTP)
|
||
NNodes int
|
||
LeaderAddr string // dist init address (ethernet IP)
|
||
LeaderPort int
|
||
GPUIndices []int
|
||
HCAs []string // HCAs of the chosen fabric
|
||
HTTPPort int
|
||
Profile Profile
|
||
ModelPath string
|
||
}
|
||
|
||
type EngineStats struct {
|
||
// The mandatory contract from §7.6 — without these the router cannot route.
|
||
Running, Waiting int
|
||
KVCacheUsage float64 // 0..1 (vllm:kv_cache_usage_perc)
|
||
// Optional.
|
||
BlockSize, NumGPUBlocks int
|
||
PrefixHitRate float64
|
||
TTFTp50, TPOTp50 time.Duration // from histograms
|
||
DrainRate float64 // observed completions/sec; for the `load` scorer
|
||
LoRA LoRAState // v2
|
||
ScrapedAt time.Time // for staleness detection — the router uses this
|
||
}
|
||
```
|
||
|
||
Supporting a new engine is defined as: being able to fill in the mandatory three from §7.6.
|
||
|
||
**Multi-node launch:**
|
||
- SGLang supports it natively: `--nnodes N --node-rank R --dist-init-addr host:port --tp T`.
|
||
- vLLM: depending on the version, either Ray (`ray start --head` on the leader, `ray start --address` on the workers) or the newer multi-node multiprocessing flags. **Check the documentation for the vLLM version you deploy**; `vllm.go` should offer both paths behind a `launchMode`.
|
||
|
||
**Environment generation (critical):**
|
||
```
|
||
CUDA_VISIBLE_DEVICES=<GPUIndices>
|
||
NCCL_IB_HCA=<HCAs, comma separated> # the chosen fabric only
|
||
NCCL_SOCKET_IFNAME=<mgmt interface> # bootstrap
|
||
NCCL_IB_DISABLE=1 # single node, or no IB
|
||
NCCL_CROSS_NIC / NCCL_NET_GDR_LEVEL # optional, from the profile
|
||
GLOO_SOCKET_IFNAME=<mgmt interface>
|
||
```
|
||
|
||
### 5.4 Supervisor
|
||
- Start processes in their own process group (`SysProcAttr.Setpgid`) and kill the whole group when stopping (SIGTERM → timeout → SIGKILL).
|
||
- stdout/stderr → line-oriented slog plus a ring buffer (the last N lines, for `llmgw ctl logs`).
|
||
- State machine: `Pending → Starting → Loading → Ready → Draining → Stopped | Failed`.
|
||
- `Ready` only once `/health` returns 200 **and** `/v1/models` lists the model.
|
||
- Model loading takes a long time; `startupTimeout` comes from the profile (e.g. 20 min).
|
||
- On a crash: report to the control plane and let **the control plane** decide about restarting. Restarting a single shard of a multi-node replica accomplishes nothing; the whole replica must be rebuilt.
|
||
- If the agent restarts, try to adopt still-running engine processes (pid file plus cmdline check) and clean up whatever cannot be adopted.
|
||
|
||
### 5.5 Model weights
|
||
- v1: a shared filesystem (NFS/Lustre/Weka) or a path pre-populated on the node. The agent only checks existence and size.
|
||
- v1.1: the agent downloads from HF/S3 into a local cache, reports progress, and enforces a disk quota.
|
||
|
||
### 5.6 Agent ↔ control plane protocol
|
||
The agent dials out (NAT- and firewall-friendly) over a bidirectional stream:
|
||
|
||
```proto
|
||
service AgentService {
|
||
rpc Connect(stream AgentMessage) returns (stream ControlMessage);
|
||
}
|
||
|
||
message AgentMessage {
|
||
oneof msg {
|
||
Register register = 1; // node info, GPUs, fabrics, versions
|
||
Heartbeat heartbeat = 2; // every 5s: GPU health, shard states
|
||
ShardStatus shard = 3; // sent on every state change
|
||
EngineStats stats = 4; // every 2s (the router may also scrape directly)
|
||
}
|
||
}
|
||
|
||
message ControlMessage {
|
||
oneof msg {
|
||
StartShard start = 1;
|
||
StopShard stop = 2;
|
||
DrainShard drain = 3;
|
||
Ack ack = 4;
|
||
}
|
||
}
|
||
```
|
||
|
||
- Commands are idempotent: a repeated `StartShard` with the same `shard_id` is a no-op.
|
||
- mTLS, with node identity taken from the certificate CN. A simple internal CA in v1; `llmgw ctl cert issue`.
|
||
- Three missed heartbeats mark the node `Unreachable`; after 60s it becomes `Lost` and its replicas are rescheduled.
|
||
|
||
---
|
||
|
||
## 6. Control Plane
|
||
|
||
### 6.1 Data model (abridged)
|
||
|
||
```go
|
||
type Node struct {
|
||
ID, Hostname, MgmtAddr string
|
||
Labels map[string]string
|
||
GPUs []GPU
|
||
Fabrics []FabricAttachment // empty means no IB
|
||
State NodeState // Ready, Cordoned, Draining, Unreachable, Lost
|
||
LastSeen time.Time
|
||
}
|
||
|
||
type FabricAttachment struct {
|
||
Fabric string // "ib-a"
|
||
HCAs []string
|
||
Rate int // Gbps
|
||
}
|
||
|
||
type Model struct {
|
||
Name string // what users see
|
||
Aliases []string
|
||
Source string // path or HF id
|
||
Profiles []Profile
|
||
Scaling Scaling // min/max replicas, target queue, ...
|
||
Fallbacks []string // other model names
|
||
}
|
||
|
||
type Profile struct {
|
||
Name string
|
||
GPUClass string // "h100-80g"
|
||
Engine string // "vllm"
|
||
TP, PP int
|
||
GPUsPerNode int // usually TP
|
||
Nodes int // PP*... ; a fabric is mandatory when > 1
|
||
Args []string // --quantization fp8 --max-model-len 32768 ...
|
||
Env map[string]string
|
||
Weight int // capacity weight for the router (from benchmarks)
|
||
Priority int // placement preference
|
||
StartupTimeout time.Duration
|
||
}
|
||
|
||
type Replica struct {
|
||
ID, Model, Profile string
|
||
Fabric string // set for multi-node replicas
|
||
Shards []Shard
|
||
State ReplicaState
|
||
Endpoint string // rank0's HTTP address
|
||
}
|
||
```
|
||
|
||
### 6.2 Scheduler (placement)
|
||
|
||
Input: the desired replica count, the profile list, and the inventory.
|
||
|
||
**Hard** constraints:
|
||
1. All shards share one `GPUClass`.
|
||
2. When `Nodes > 1`, every node must attach to the same `Fabric` and have ACTIVE HCAs on it.
|
||
3. `GPUsPerNode` free GPUs, preferably in the same NVLink domain and on the same NUMA node as the HCA.
|
||
4. The node is `Ready` and not `Cordoned`.
|
||
5. The engine's `Validate` must pass (e.g. an FP8 profile never lands on a card with compute capability < 8.9).
|
||
|
||
**Soft** preferences (scored):
|
||
- the profile's `Priority`
|
||
- bin packing: fill partially used nodes first, keeping empty nodes free for large models
|
||
- spread a model's replicas across fabrics and nodes (failure domain distribution)
|
||
- GPU-to-HCA NUMA affinity
|
||
|
||
Algorithm (v1, simple and deterministic):
|
||
|
||
```
|
||
for each profile in model.Profiles sorted by Priority desc:
|
||
pools := groupNodes(by GPUClass == profile.GPUClass, by Fabric if profile.Nodes > 1)
|
||
for each pool sorted by score:
|
||
nodes := pickNodes(pool, profile.Nodes, profile.GPUsPerNode) // greedy + NVLink group selection
|
||
if nodes != nil:
|
||
return Placement{profile, pool.Fabric, nodes, gpuAssignments}
|
||
return ErrUnschedulable (with reasons — essential for debugging)
|
||
```
|
||
|
||
`groupNodes`: nodes without IB join the pool for `Nodes == 1` profiles, and never for `Nodes > 1`.
|
||
|
||
`llmgw ctl explain-placement <model>` must show why each pool was rejected.
|
||
|
||
### 6.2.1 Heterogeneous GPUs: the SLO/capacity model
|
||
|
||
Comparing GPU classes through a single scalar "weight" is too crude; the model taken from
|
||
AIBrix's heterogeneous GPU optimizer is more accurate. For each (model, profile) pair,
|
||
`ctl bench` (§10) produces a **capacity profile**:
|
||
|
||
```go
|
||
type CapacityProfile struct {
|
||
Model, Profile string
|
||
GPUClass string
|
||
// Maximum concurrency that still meets the SLO; the real capacity measure.
|
||
MaxConcurrencyAtSLO int
|
||
ThroughputTokPerSec float64
|
||
TTFTp99, TPOTp99 time.Duration
|
||
RelativeCost float64 // relative GPU-hour cost, from configuration
|
||
MeasuredAt time.Time
|
||
EngineVersion string // the profile expires when the version changes
|
||
}
|
||
```
|
||
|
||
An example SLO: `TPOT p99 ≤ 50ms && TTFT p99 ≤ 2s`.
|
||
|
||
The profile is used in three places:
|
||
1. **Router weight:** `Profile.Weight` is no longer hand-written; it derives from `MaxConcurrencyAtSLO`.
|
||
2. **Admission thresholds:** `TargetWaitingPerReplica` (§7.2.1) is suggested from the measurement.
|
||
3. **Placement and autoscaling objective:** "minimize total cost while meeting the SLO". For a
|
||
predicted load, the replica count needed per GPU class is `ceil(load / MaxConcurrencyAtSLO)`,
|
||
and the cheapest combination wins. In v1 this only produces advice via
|
||
`ctl plan <model> --target-rps N`; automatic application arrives with the v1.1 autoscaler.
|
||
|
||
When `EngineVersion` or the profile arguments change the measurement goes stale. llmgw must
|
||
surface that (`llmgw_capacity_profile_stale{model,profile}`) rather than silently reusing an
|
||
obsolete number.
|
||
|
||
### 6.3 Reconciler
|
||
- Every 5s, or on an event: compare desired (model scaling) with actual (replica states).
|
||
- Missing replica → schedule → send `StartShard` for each shard (workers first, then rank0, or whatever order the engine requires).
|
||
- Any shard in a replica `Failed` → stop the whole replica and reschedule with backoff.
|
||
- Excess replicas → `Drain` first (the router stops sending new requests, existing ones finish), then `Stop`.
|
||
- Node `Lost` → treat its replicas as `Failed`.
|
||
|
||
Autoscaling (v1.1): a per-model `targetWaitingPerReplica`, averaged from router metrics, with
|
||
cooldown-guarded scaling. Shrinking lower-priority models when no GPUs are free (preemption) is v2.
|
||
|
||
### 6.4 Interface to the router
|
||
|
||
```go
|
||
type EndpointSource interface {
|
||
// model name -> Ready replicas (+ pool, weight, fabric, gpuclass, capacity profile)
|
||
Endpoints(model string) []Endpoint
|
||
Subscribe() <-chan struct{} // change notification
|
||
}
|
||
```
|
||
In-memory in v1; once the router and control plane split apart, the same interface is backed by
|
||
a watch stream.
|
||
|
||
The router **always caches** this list: even if the control plane goes down it keeps routing to
|
||
the replicas it knows about, which mitigates the single-point-of-failure risk in §14. When the
|
||
cache ages past `endpoint_cache_max_age` (say 5 min) it only raises a warning metric; it never
|
||
stops traffic.
|
||
|
||
### 6.5 Store
|
||
- SQLite in WAL mode. Tables: `nodes`, `models`, `profiles`, `replicas`, `shards`, `api_keys`, `usage_rollup`, `events`.
|
||
- Inventory and replica state can be rebuilt from the agents; what genuinely needs to persist is the catalog, the keys and the usage records.
|
||
- HA (v2): Raft-replicated state, or an external Postgres/etcd.
|
||
|
||
---
|
||
|
||
## 7. Router
|
||
|
||
### 7.1 Request lifecycle
|
||
|
||
```
|
||
Request
|
||
├─ 1. Auth : key → project/team, permitted models, priority band, routing profile
|
||
├─ 2. Quota : RPM/TPM and budget (in-memory counters; no DB on the request path)
|
||
├─ 3. Parse : model, stream, messages/prompt, max_tokens, user.
|
||
│ The rest of the body passes through as untouched JSON.
|
||
├─ 4. Resolve : model/alias → candidate pools (fabric × gpu_class)
|
||
├─ 5. ADMISSION : "now?" §7.2 ← waits here while the pool is saturated
|
||
├─ 6. SCHEDULING : "where?" §7.3 filter → score → pick
|
||
├─ 7. Proxy : stream to upstream, propagate cancellation §7.8
|
||
└─ 8. After the request (async): usage, spend, metrics, logs
|
||
```
|
||
|
||
**The critical separation (taken from GIE / llm-d).** *When* a request proceeds and *where* it
|
||
goes are two distinct decisions and must stay distinct. Kept apart, the router can hold requests
|
||
in its own central queue and apply priority and tenant fairness **before they enter an engine
|
||
queue**.
|
||
|
||
Why this matters: an engine's own queue is FIFO and has no notion of tenants or priority. Once a
|
||
request is in there, no prioritization is possible. When a batch job fills the vLLM queue with
|
||
500 requests, the TTFT of an interactive chat request arriving behind them collapses and there
|
||
is nothing the router can do. v0.1 had no such layer; **for a shared cluster this was the most
|
||
serious omission.**
|
||
|
||
### 7.2 Admission control (flow control)
|
||
|
||
#### 7.2.1 Saturation detector
|
||
|
||
One saturation signal is produced per pool, combining two metrics (llm-d's approach — queue
|
||
depth alone is not enough, KV pressure is a genuine exhaustion signal too):
|
||
|
||
```go
|
||
// per replica, then aggregated across the pool as p90 or mean
|
||
func saturation(e EngineStats, t Thresholds) float64 {
|
||
q := float64(e.Waiting) / t.TargetWaitingPerReplica // e.g. target 4
|
||
kv := e.KVCacheUsage / t.KVCeiling // e.g. ceiling 0.90
|
||
return math.Max(q, kv)
|
||
}
|
||
```
|
||
|
||
The goal is to keep a **small but healthy buffer** on the engine: enough requests for continuous
|
||
batching to stay efficient, but not enough for the queue to balloon. Set
|
||
`TargetWaitingPerReplica` too low and the GPU starves; too high and queueing delay returns. Both
|
||
thresholds must be tunable per model/profile, and `ctl bench` (§10) should propose them.
|
||
|
||
#### 7.2.2 Priority bands
|
||
|
||
Every API key maps to a band, and the band sets the saturation ceiling:
|
||
|
||
```yaml
|
||
bands:
|
||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 } # never held back
|
||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 } # batch jobs
|
||
```
|
||
|
||
The dispatch loop scans bands from highest to lowest. Once pool saturation exceeds a band's
|
||
ceiling, that band's requests wait in the queue. The result: as the cluster fills, batch traffic
|
||
slows down first and interactive traffic is protected. (In llm-d's measurements platinum, gold
|
||
and silver stayed under a second while bronze batch fell far behind — exactly the desired
|
||
behaviour.)
|
||
|
||
#### 7.2.3 Fairness and queue management
|
||
|
||
- **Fairness within a band:** when several tenants share a band, dispatch turns are distributed
|
||
round-robin. One tenant's traffic spike cannot consume the whole band.
|
||
- **Capacity:** each band's queue is bounded. When it is full, new requests get an immediate
|
||
`503` — far better than accumulating silently.
|
||
- **TTL:** a request that waits longer than `queue_ttl` is dropped (`504`). There is no point
|
||
sending the GPU work the client has already given up on.
|
||
- **Client disconnect:** if the connection drops while queued, the request is discarded quietly
|
||
(context cancellation).
|
||
- **Displacement (v1.1):** to make room for a higher band, drop the **newest** requests from the
|
||
lowest band — the oldest ones are already near their TTL and have invested the most waiting.
|
||
|
||
```go
|
||
type Admission interface {
|
||
// Blocks until the request may proceed; errors on cancellation or TTL.
|
||
Acquire(ctx context.Context, req *RequestInfo) (Lease, error)
|
||
}
|
||
```
|
||
|
||
`Lease.Release()` is called once the response completes; it also maintains the router's inflight
|
||
counter.
|
||
|
||
### 7.3 Scheduler: filter → score → pick
|
||
|
||
The list of "strategies" from v0.1 is replaced by a plugin pipeline (the GIE EPP model). No
|
||
strategy is a separate monolith; scorers are combined by weight.
|
||
|
||
```go
|
||
type Filter interface { // drops candidates (hard constraints)
|
||
Name() string
|
||
Filter(ctx *SchedCtx, eps []Endpoint) []Endpoint
|
||
}
|
||
|
||
type Scorer interface { // 0..1 score; 1 is best
|
||
Name() string
|
||
Score(ctx *SchedCtx, eps []Endpoint) map[EndpointID]float64
|
||
}
|
||
|
||
type Picker interface { // chooses from the scores
|
||
Pick(ctx *SchedCtx, scores map[EndpointID]float64) Endpoint
|
||
}
|
||
|
||
type RoutingProfile struct {
|
||
Name string
|
||
Filters []Filter
|
||
Scorers []WeightedScorer // {Scorer, Weight}
|
||
Picker Picker
|
||
}
|
||
```
|
||
|
||
Total score: `score(ep) = Σ wᵢ · scoreᵢ(ep)`, with each scorer normalizing to 0..1 internally.
|
||
|
||
**A two-stage decision.** Because the topology is heterogeneous, the pool is chosen first and
|
||
the replica second:
|
||
|
||
1. **Pool selection:** the pool with the highest `Priority`; if every replica there is saturated
|
||
(§7.2.1), spill over to the next pool. This prevents falling back to the A100 pool before the
|
||
H100 pool is actually full.
|
||
2. **Replica selection:** filter/score/pick within the chosen pool.
|
||
|
||
**Routing profiles**, from configuration (a per-model default, overridable per request via the
|
||
`x-llmgw-profile` header):
|
||
|
||
| Profile | Weights | Use |
|
||
|---|---|---|
|
||
| `chat` | prefix 3.0, queue 1.0, kv 1.0 | Multi-turn chat and agents — high cache reuse |
|
||
| `batch` | queue 3.0, kv 1.0, prefix 0.5 | Bulk work — throughput matters, TTFT does not |
|
||
| `long-context` | kv 3.0, prefix 2.0, queue 1.0 | Long prompts — KV pressure is the real bottleneck |
|
||
| `debug` | round-robin picker | Troubleshooting |
|
||
|
||
### 7.4 Scorers and their formulas
|
||
|
||
| Scorer | Formula | Notes |
|
||
|---|---|---|
|
||
| `queue` | `1 − min(1, waiting / target_waiting)` | Queue depth |
|
||
| `kv` | `1 − kv_cache_usage` | KV occupancy; a hard penalty above the ceiling |
|
||
| `prefix` | `matched_blocks / prompt_blocks` | §7.5 |
|
||
| `load` | `1 − norm(running / drain_rate)` | AIBrix: `drain_rate` is the observed completion rate. `running` alone cannot spot a slow replica. |
|
||
| `inflight` | `1 − norm(router_inflight)` | The router's own count; compensates for scrape lag. **At a 2s scrape interval, herding is unavoidable without it.** |
|
||
| `pool-priority` | constant, from the profile's `Priority` | Pool preference |
|
||
| `session` | session hash match | §7.5 tier 0 |
|
||
| `lora` (v2) | is the adapter loaded | `vllm:lora_requests_info` |
|
||
|
||
**An alternative cost formulation (Dynamo).** Modelling prefill and decode cost directly in
|
||
blocks may be more accurate than normalized scores:
|
||
|
||
```
|
||
cost(ep) = w_overlap · (prompt_blocks − cached_blocks(ep)) + decode_blocks(ep)
|
||
choice = argmin cost
|
||
```
|
||
|
||
A high `w_overlap` favours cache reuse (better TTFT); a low one evens out load (better ITL).
|
||
Both can be expressed as a `Scorer`, so the pipeline survives either way: weighted scores in v1,
|
||
with this cost form added as a scorer in v1.1 and compared using `ctl bench`.
|
||
|
||
**Pickers:**
|
||
- `max-score`: the highest score. Deterministic but prone to herding.
|
||
- `p2c` (power of two choices): two random candidates, the better one. The default once there are many replicas.
|
||
- `softmax(temperature)`: Dynamo's `router_temperature`. Treats scores as logits and samples. `temperature=0` degenerates to argmax. A small temperature breaks the oscillation that synchronized decisions cause among equally scored replicas.
|
||
|
||
### 7.5 Prefix / KV cache awareness — three tiers
|
||
|
||
This is the single largest win the router can deliver, but the cost gap between the tiers is
|
||
enormous. They stay separate so they can be enabled per model.
|
||
|
||
#### Tier 0 — session affinity (v1, cheap, high return)
|
||
The client-supplied `x-session-id` header (falling back to OpenAI's `user` field) maps to a
|
||
replica via bounded-load consistent hashing. If that replica is overloaded — beyond
|
||
`load_factor` times the average — the next one on the ring is used. Roughly 100 lines of code;
|
||
production-stack measured cache hits going from about 93% to 96% versus round robin. For
|
||
multi-turn chat and agent workloads this is the best cost/benefit ratio available.
|
||
|
||
#### Tier 1 — approximate radix tree (v1.1, moderate cost)
|
||
The router maintains a tree that **estimates** each worker's cache from the request prefixes it
|
||
has seen (the SGLang router's approach). It does not know the true cache state; it updates
|
||
lazily and costs little.
|
||
|
||
The thresholds — the part that matters in real implementations:
|
||
- `cache_threshold`: below this match ratio, treat it as "no cache" and let load balancing decide.
|
||
- `balance_abs_threshold` **and** `balance_rel_threshold`: if `max_load − min_load > abs`
|
||
**and** `max_load / min_load > rel`, **ignore the prefix entirely** and send to the least
|
||
loaded replica. (AIBrix adds the same guard.) Without it, prefix affinity drowns one replica:
|
||
the cache hit rate rises while p99 latency collapses.
|
||
- `eviction_interval` plus `max_tree_size`: keep the tree from growing without bound.
|
||
|
||
Which branch a decision took must be recorded as a metric: `cache_hit | cache_miss_min_load |
|
||
load_balance_override | stale | no_tree | no_healthy_worker`. Without that, tuning prefix routing
|
||
is guesswork.
|
||
|
||
#### Tier 2 — precise KV index (v2, expensive, optional)
|
||
vLLM publishes block events over ZMQ with `--kv-events-config` (`BlockStored`, `BlockRemoved`).
|
||
The router subscribes and maintains a `block_hash → {replica}` map, giving it a true global view.
|
||
|
||
**The trap — a cost the plan must acknowledge:** block hashes must be computed identically to
|
||
vLLM's **token-id-based** scheme. That forces the router to tokenize requests with **the same
|
||
tokenizer as the engine**. On the Go side that means loading a tokenizer per model (HF
|
||
`tokenizers` via cgo, or a pure-Go implementation), tracking version compatibility, and paying
|
||
the tokenization CPU cost per request. This is why AIBrix requires a "remote tokenizer".
|
||
**Tiers 0 and 1 need no tokenizer** — hence they are v1 and this is v2.
|
||
|
||
Event loss: if a gap appears in the event sequence numbers, request the missing range from the
|
||
replay endpoint; if that fails, drop the index for that replica and fall back to **tier 1** for
|
||
it. Returning to approximate information beats silently routing on wrong data. If a replica
|
||
restarts, its index is reset.
|
||
|
||
### 7.6 The engine metric contract
|
||
|
||
This is the formal definition of "adding a new engine". If an engine exposes these three metrics
|
||
in Prometheus format, llmgw can route to it (GIE's model server protocol):
|
||
|
||
| Contract field | vLLM metric | Mandatory |
|
||
|---|---|---|
|
||
| `Waiting` | `vllm:num_requests_waiting` | yes |
|
||
| `Running` | `vllm:num_requests_running` | yes |
|
||
| `KVCacheUsage` | `vllm:kv_cache_usage_perc` | yes |
|
||
| `BlockSize`, `NumGPUBlocks` | `vllm:cache_config_info` | no |
|
||
| LoRA state | `vllm:lora_requests_info` (`max_lora`, `running_lora_adapters`, `waiting_lora_adapters`) | no |
|
||
|
||
> Correction: v0.1 said `vllm:gpu_cache_usage_perc`; the correct name is
|
||
> `vllm:kv_cache_usage_perc`. Metric names drift between versions, so the `ParseMetrics` mapping
|
||
> must be overridable from configuration, and the agent must report the engine version when it
|
||
> registers.
|
||
|
||
Scrape rules:
|
||
- Every Ready endpoint every 1–2s, one goroutine per endpoint, with jitter.
|
||
- A failed scrape marks the endpoint stale; its score drops but it is not removed.
|
||
- Three consecutive failures trip the breaker (§7.7).
|
||
- **So that decisions are never made purely on stale data**, the `inflight` scorer (§7.4) stays
|
||
enabled at all times.
|
||
|
||
### 7.7 Reliability
|
||
|
||
- **Retry:** only before the first byte has been written upstream, only on connection errors,
|
||
503 or 429, always against a different endpoint, at most N times. No retries once the stream
|
||
has started.
|
||
- **Cooldown (the LiteLLM lesson):** an endpoint that errors leaves the candidate pool for
|
||
`cooldown_seconds`. A long fallback list plus retries without cooldown equals a retry storm:
|
||
one endpoint's rate limit spreads across the whole cluster.
|
||
- **Circuit breaker:** a closed/open/half-open state machine per endpoint, with exponential backoff.
|
||
- **Fallback:** when a model has no usable endpoints, fall back to a model in its `Fallbacks`
|
||
list (if the key permits it). The response carries an `x-llmgw-served-model` header.
|
||
**Budget fallback:** when a key exceeds its budget, optionally downgrade to a cheaper model
|
||
rather than rejecting the request (§7.9).
|
||
- **Timeouts:** separate limits for connection, first token (TTFT) and total, plus an
|
||
inter-token idle timeout while streaming.
|
||
- **Cancellation:** when the client disconnects, cancel the upstream request too (context
|
||
propagation) so the GPU stops working. This applies to queued requests as well.
|
||
- **Draining:** a `Draining` replica receives no new requests; active ones run to completion or
|
||
time out.
|
||
- **Control plane loss:** the router keeps serving from its last known endpoint list (§14).
|
||
|
||
### 7.8 Streaming proxy details
|
||
|
||
- `http.Transport` with a high `MaxIdleConnsPerHost` and HTTP/1.1 keep-alive (engines rarely
|
||
speak h2c).
|
||
- Read lines with `bufio.Reader`, pass `data:` lines through, and call `http.Flusher.Flush()`
|
||
after every event.
|
||
- If the client did not send `stream_options.include_usage`, the router may add it
|
||
(configurable) to capture usage from the final chunk, and then withhold that chunk from a
|
||
client that did not ask for it.
|
||
- Normalize upstream error JSON into OpenAI's format.
|
||
- Response headers: `x-llmgw-replica`, `x-llmgw-pool`, `x-llmgw-served-model`,
|
||
`x-llmgw-queue-ms` (time spent in admission) and `x-llmgw-cache-decision` (§7.5).
|
||
|
||
### 7.9 Identity, quota and usage (tenancy)
|
||
|
||
LiteLLM's production-proven model:
|
||
|
||
```
|
||
Organization ─► Team ─► Project ─► API Key
|
||
```
|
||
Each level carries a budget (in tokens or notional cost units), RPM/TPM limits and a permitted
|
||
model list. A key additionally carries its **priority band** (§7.2.2) and its **default routing
|
||
profile** (§7.3).
|
||
|
||
- API keys look like `sk-llmgw-...`; only the SHA-256 hash is stored. A leaked key is rotated
|
||
without touching upstream credentials.
|
||
- Rate limiting: RPM via `x/time/rate` plus **TPM**. The token limit is reserved at request
|
||
start as `max_tokens + estimated prompt` and reconciled against the real figure when the
|
||
response completes. (The prompt estimate is bytes/4 in v1; it becomes exact once the tier 2
|
||
tokenizer arrives.)
|
||
- **Budgets:** periodic (daily/monthly) spend. On exhaustion: `429`, or a budget fallback.
|
||
Zero-cost models are exempt from budget checks, so a key whose budget is spent can still be
|
||
given a "cheap model" fallback.
|
||
- **Spend writes stay off the request path:** usage records go to a channel and are batched into
|
||
SQLite, with hourly rollups. Nothing writes to the database on the request path.
|
||
- **Replica pinning (debug):** the `x-llmgw-replica` header, scoped to the key. If the pinned
|
||
replica is in cooldown or gone, the request falls through to normal routing.
|
||
- v1 is a single instance with in-memory limiters. With multiple instances this becomes Redis or
|
||
consistent-hash stickiness (v2).
|
||
|
||
---
|
||
|
||
## 8. Observability
|
||
|
||
Prometheus metrics (examples):
|
||
```
|
||
llmgw_requests_total{model,pool,status}
|
||
llmgw_request_duration_seconds{model,pool}
|
||
llmgw_ttft_seconds{model,pool}
|
||
llmgw_tokens_total{model,direction="prompt|completion"}
|
||
llmgw_endpoint_inflight{replica}
|
||
llmgw_endpoint_score{replica}
|
||
llmgw_replicas{model,state}
|
||
llmgw_gpu_free{node,gpu_class,fabric}
|
||
llmgw_schedule_failures_total{model,reason}
|
||
llmgw_agent_connected{node}
|
||
|
||
# admission / flow control (§7.2)
|
||
llmgw_admission_queue_depth{band,model}
|
||
llmgw_admission_wait_seconds{band,model} # histogram
|
||
llmgw_admission_rejected_total{band,reason="full|ttl|client_gone"}
|
||
llmgw_pool_saturation{pool} # 0..1, the detector's output
|
||
|
||
# routing decisions (§7.5) — prefix routing cannot be tuned without these
|
||
llmgw_cache_decision_total{model,decision="hit|miss_min_load|balance_override|stale|no_tree"}
|
||
llmgw_prefix_match_ratio{model} # histogram
|
||
llmgw_scorer_value{scorer,replica}
|
||
llmgw_endpoint_metrics_stale{replica}
|
||
llmgw_capacity_profile_stale{model,profile}
|
||
```
|
||
- OpenTelemetry traces: router span → upstream span (the traceparent header is forwarded to the engine).
|
||
- Structured logs (slog JSON) with `request_id` everywhere.
|
||
- `llmgw ctl top`: live queue depth and occupancy per pool.
|
||
|
||
---
|
||
|
||
## 9. Configuration Examples
|
||
|
||
### 9.1 server.yaml
|
||
```yaml
|
||
listen: ":8080"
|
||
admin_listen: ":9090" # metrics, admin API, pprof
|
||
agent_listen: ":7443"
|
||
tls:
|
||
ca: /etc/llmgw/ca.pem
|
||
cert: /etc/llmgw/server.pem
|
||
key: /etc/llmgw/server-key.pem
|
||
store:
|
||
path: /var/lib/llmgw/llmgw.db
|
||
|
||
fabrics:
|
||
- name: ib-a
|
||
gid_prefix: "fec0:0000:0000:000a" # optional automatic mapping
|
||
- name: ib-b
|
||
gid_prefix: "fec0:0000:0000:000b"
|
||
|
||
router:
|
||
scrape_interval: 2s
|
||
inject_usage: true
|
||
retries: 2
|
||
cooldown_seconds: 15 # how long a failing endpoint leaves the candidate pool
|
||
endpoint_cache_max_age: 5m # keep serving this long if the control plane goes down
|
||
|
||
# §7.2 admission — the saturation detector
|
||
admission:
|
||
target_waiting_per_replica: 4 # proposed by ctl bench
|
||
kv_ceiling: 0.90
|
||
queue_ttl: 30s
|
||
queue_capacity_per_band: 2000
|
||
bands:
|
||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 }
|
||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 }
|
||
|
||
# §7.3 routing profiles
|
||
default_profile: chat
|
||
profiles:
|
||
chat:
|
||
scorers: { prefix: 3.0, queue: 1.0, kv: 1.0, inflight: 1.0 }
|
||
picker: { type: p2c }
|
||
batch:
|
||
scorers: { queue: 3.0, kv: 1.0, load: 1.0, prefix: 0.5 }
|
||
picker: { type: max-score }
|
||
long-context:
|
||
scorers: { kv: 3.0, prefix: 2.0, queue: 1.0 }
|
||
picker: { type: softmax, temperature: 0.2 }
|
||
|
||
# §7.5 prefix awareness
|
||
prefix:
|
||
session_affinity: { enabled: true, header: x-session-id, load_factor: 1.25 } # tier 0
|
||
approx_tree: # tier 1
|
||
enabled: false # v1.1
|
||
cache_threshold: 0.30
|
||
balance_abs_threshold: 16
|
||
balance_rel_threshold: 1.5
|
||
max_tree_size: 2000000
|
||
eviction_interval: 60s
|
||
kv_events: { enabled: false } # tier 2 — needs a tokenizer, v2
|
||
|
||
models:
|
||
- name: llama-3.3-70b
|
||
source: /models/Llama-3.3-70B-Instruct
|
||
scaling: { min: 2, max: 6 }
|
||
fallbacks: [qwen2.5-32b]
|
||
routing_profile: chat
|
||
slo: { ttft_p99: 2s, tpot_p99: 50ms } # ctl bench measures capacity against this SLO
|
||
profiles:
|
||
- name: h100-tp4-fp8
|
||
gpu_class: h100-80g
|
||
engine: vllm
|
||
tp: 4
|
||
nodes: 1
|
||
priority: 100
|
||
weight: 100
|
||
args: ["--quantization", "fp8", "--max-model-len", "32768"]
|
||
- name: a100-tp8
|
||
gpu_class: a100-80g
|
||
engine: vllm
|
||
tp: 8
|
||
nodes: 1
|
||
priority: 50
|
||
weight: 45
|
||
|
||
- name: deepseek-large
|
||
source: /models/DeepSeek-V3
|
||
scaling: { min: 1, max: 2 }
|
||
profiles:
|
||
- name: h100-2node
|
||
gpu_class: h100-80g
|
||
engine: sglang
|
||
tp: 16
|
||
nodes: 2 # → must live on a single fabric
|
||
gpus_per_node: 8
|
||
priority: 100
|
||
weight: 100
|
||
startup_timeout: 40m
|
||
```
|
||
|
||
### 9.2 agent.yaml
|
||
```yaml
|
||
node_id: gpu-a-01
|
||
server: "llmgw-server.internal:7443"
|
||
mgmt_interface: eth0
|
||
tls: { ca: ..., cert: ..., key: ... }
|
||
labels: { rack: r12 }
|
||
fabrics:
|
||
- name: ib-a
|
||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
|
||
port_range: [20000, 20999] # engine HTTP and dist-init ports
|
||
engines:
|
||
vllm: { bin: /opt/venvs/vllm/bin/vllm, launch_mode: ray }
|
||
sglang: { python: /opt/venvs/sglang/bin/python }
|
||
work_dir: /var/lib/llmgw-agent
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Benchmarking and Weighting
|
||
|
||
`llmgw ctl bench <model> --profile <p>`:
|
||
- Drives a target replica at increasing concurrency with fixed prompt and output lengths.
|
||
- Measures TTFT p50/p95, TPOT, throughput (tok/s) and the maximum concurrency that still meets the SLO.
|
||
- The output is the `CapacityProfile` from §6.2.1: `MaxConcurrencyAtSLO`, throughput, TTFT/TPOT p99.
|
||
- `--apply` updates three things at once: the profile's `weight`, `target_waiting_per_replica`,
|
||
and the autoscaling target.
|
||
- This is the only sound way to compare GPU classes; never use an estimated weight.
|
||
- `--sweep-prefix` mode measures behaviour at 0/50/90% prefix cache hit rates, so routing
|
||
profile weights (§7.3) can be tuned from measurements instead of guesses.
|
||
- A profile goes stale when the engine version changes, so every `ctl bench` result is tagged
|
||
with the engine version.
|
||
|
||
---
|
||
|
||
## 11. Roadmap
|
||
|
||
### Phase 0 — Skeleton (1 week)
|
||
- [x] `go mod init`, cobra commands, koanf config, slog, Makefile, golangci-lint
|
||
- [x] `sim-engine`: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format `/metrics`)
|
||
- [x] CI: `go test ./...`, race detector
|
||
|
||
### Phase 1 — Static router (2 weeks)
|
||
- [ ] OpenAI handlers, streaming proxy, cancellation propagation
|
||
- [ ] Static endpoint list from configuration
|
||
- [x] **The `Filter`/`Scorer`/`Picker` pipeline (§7.3)** — a plugin pipeline from the start, not one "strategy"
|
||
- [x] `queue`, `kv`, `inflight` scorers; `max-score` and `p2c` pickers
|
||
- [x] Session affinity (§7.5 tier 0) — cheap, an early win
|
||
- [x] Metrics scraping (the §7.6 contract), staleness detection, retry, cooldown, breaker
|
||
- [ ] Prometheus metrics plus decision metrics (§8)
|
||
- **Exit criterion:** 1000 concurrent streams behind 3 sim-engines; upstream cancellation on
|
||
client disconnect verified; router overhead p99 < 2 ms.
|
||
|
||
### Phase 1.5 — Admission control (1 week) ⭐
|
||
- [ ] Saturation detector (§7.2.1)
|
||
- [ ] Priority bands, round-robin tenant fairness within a band, queue capacity and TTL
|
||
- [ ] `Admission` interface and lease, cancellation while queued
|
||
- [ ] `llmgw_admission_*` metrics
|
||
- **Exit criterion:** while 500 concurrent batch requests flow through the bronze band, the TTFT
|
||
p99 of interactive requests in the platinum band stays under 1.5× its idle value. **Do not
|
||
move on before writing this test — it is the entire reason phase 1.5 exists.**
|
||
|
||
### Phase 2 — Agent: discovery and single-node engines (2 weeks)
|
||
- [ ] GPU discovery (NVML plus fallback), IB discovery, topology
|
||
- [ ] Engine interface, single-node vLLM and SGLang
|
||
- [ ] Supervisor (process groups, log buffer, state machine, adoption)
|
||
- [ ] connect-go stream, register/heartbeat, mTLS
|
||
- **Exit criterion:** start and stop a model on a real node via `ctl`.
|
||
|
||
### Phase 3 — Control plane (2–3 weeks)
|
||
- [ ] SQLite store and migrations
|
||
- [ ] Catalog, inventory, scheduler (single-node profiles), reconciler
|
||
- [ ] The router's `EndpointSource` becomes dynamic
|
||
- [ ] `ctl`: nodes, models, replicas, cordon, drain, explain-placement, logs
|
||
- **Exit criterion:** shutting down a node moves its replicas elsewhere while the router keeps serving.
|
||
|
||
### Phase 4 — Multi-node and fabrics (2 weeks)
|
||
- [ ] Fabric identity (config plus GID prefix), verification
|
||
- [ ] Multi-node placement (the fabric constraint), port allocation, rank-ordered startup
|
||
- [ ] NCCL environment generation, HCA selection
|
||
- [ ] Replica-level failure handling (one dead shard kills the replica)
|
||
- **Exit criterion:** a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it to the other one.
|
||
|
||
### Phase 5 — Production features (2–3 weeks)
|
||
- [ ] Tenancy: org/team/project/key hierarchy, budgets, RPM/TPM, usage records (§7.9)
|
||
- [ ] Budget fallback, model allowlists, key→band mapping
|
||
- [ ] Routing profiles from configuration plus the `x-llmgw-profile` header
|
||
- [ ] Approximate radix tree (§7.5 tier 1), with its thresholds and decision metrics
|
||
- [ ] Fallback models and pool spillover
|
||
- [ ] `ctl bench` → `CapacityProfile` (§6.2.1), `ctl plan`
|
||
- [ ] OTel tracing
|
||
- [ ] systemd units, documentation
|
||
|
||
### Phase 6 — v2 ideas
|
||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||
- [ ] Admission displacement (dropping requests from lower bands)
|
||
- [ ] Autoscaling and priority-based preemption
|
||
- [ ] Control plane HA (Raft)
|
||
- [ ] Weight downloading and cache management
|
||
- [ ] Prefill/decode disaggregation (§12)
|
||
- [ ] LoRA adapter routing (affinity to a replica that already has the adapter)
|
||
- [ ] External provider support (as a fallback)
|
||
- [ ] Admin web UI
|
||
|
||
---
|
||
|
||
## 12. Prefill/Decode Disaggregation (v2 notes)
|
||
- Prefill and decode replicas must live on the **same fabric** (KV transfer goes over RDMA).
|
||
- A two-stage flow in the router: send to a prefill replica → KV handle → decode replica.
|
||
- Engine support varies sharply by version (vLLM KV connectors, SGLang PD mode, NVIDIA Dynamo/NIXL). Wrapping the engine's own built-in PD router as a single "endpoint" is less risky than implementing this ourselves.
|
||
- What llm-d and Dynamo show: PD's benefit lies mainly in **long context**. For short-prompt chat
|
||
traffic the KV transfer cost eats the gain. Measure your workload's ISL/OSL distribution with
|
||
the benchmark in §10 first, and enable PD only for long-context models.
|
||
- The scheduler gains a `Role: prefill|decode` field and a "paired placement on the same fabric" constraint.
|
||
|
||
---
|
||
|
||
## 13. Test Strategy
|
||
- **Unit:** the scheduler (table-driven: heterogeneous inventory plus expected placement or failure reason), balancer scores, SSE parsing, metric parsing (fixtures taken from real vLLM/SGLang `/metrics` output).
|
||
- **Property-based:** the invariants "a multi-node replica lives on one fabric" and "one GPU class per replica" always hold in scheduler output (`testing/quick` or `rapid`).
|
||
- **Integration:** server plus N fake agents (fake GPU/IB inventory) plus sim-engines, in one process.
|
||
- **Chaos:** kill agents, break the network (toxiproxy), crash engines, slow engines, and kill
|
||
the control plane to verify the router keeps serving from its cache.
|
||
- **Priority isolation:** phase 1.5's exit criterion as a permanent regression test — high-priority
|
||
TTFT is preserved under low-priority load. This test must stay in CI.
|
||
- **Routing quality:** give sim-engine a realistic prefix cache model (block-level LRU) and
|
||
measure the hit rate against synthetic multi-turn traffic. The comparison between round robin,
|
||
session affinity and the approximate tree must be numeric.
|
||
- **Load:** `k6` or our own Go load generator; measure the router's own overhead (target: p99 under 2 ms of added latency).
|
||
- **Real hardware:** at the end of every phase, a smoke test on at least one node with IB and one without.
|
||
|
||
---
|
||
|
||
## 14. Risks and Open Questions
|
||
| Topic | Note |
|
||
|---|---|
|
||
| Engine version differences | CLI flags and metric names change often → make engine adapters version-aware and overridable from configuration. |
|
||
| vLLM multi-node launch | Ray or built-in multiprocessing? Must be tested on the version actually deployed. |
|
||
| Fabric detection | Can a unique subnet prefix be set on each OpenSM? If not, manual labelling is mandatory. |
|
||
| Multiple fabrics per node | Does this occur? If so, HCA-to-GPU NUMA matching matters. |
|
||
| Model storage | Is there a shared filesystem, and what are the load times? Startup timeouts follow from this. |
|
||
| Mixed driver/CUDA versions | Should engine virtualenvs differ per GPU class? (The `engines` config must be overridable per profile.) |
|
||
| Single control plane | A SPOF in v1. The router caches the endpoint list (§6.4) and keeps working through a brief control plane outage. |
|
||
| Security | Engine ports must be reachable only by the router (firewall or bind address). |
|
||
| **Tokenizer consistency** | The precise KV index (§7.5 tier 2) forces the router to use **the same tokenizer** as the engine. In Go that means cgo or a separate tokenizer service, which is why it is deferred to v2; v1's prefix approach needs no tokenizer. The same limitation makes the TPM prompt estimate approximate in v1 (bytes/4). |
|
||
| **Prefix affinity backfiring** | Without the load-imbalance guard (the §7.5 tier 1 thresholds), prefix routing raises the cache hit rate while drowning a single replica and wrecking p99. Prefix affinity **must not** be enabled before those thresholds are. |
|
||
| **Tuning the admission thresholds** | Too low a `target_waiting_per_replica` starves the GPU; too high brings queueing delay back. This single number governs the system's behaviour, so measure it with `ctl bench` rather than guessing. |
|
||
| KV event loss | ZMQ is fire-and-forget; a lost event makes the router silently misroute. Sequence-gap detection and falling back to tier 1 are mandatory (§7.5). |
|
||
| Metric staleness | At 1000 requests/sec with a 2s scrape interval, decisions rest entirely on stale data. That is why the `inflight` scorer is not optional. |
|
||
|
||
---
|
||
|
||
## 15. First Steps
|
||
1. `mkdir llmgw && cd llmgw && go mod init <module-path>/llmgw`
|
||
2. `cmd/llmgw/main.go`: an empty cobra skeleton with the `server`, `agent`, `ctl` and `sim-engine` subcommands.
|
||
3. `internal/sim`: the fake engine — make this the first real code, because everything else is tested against it.
|
||
4. `internal/router/proxy`: streaming proxy plus a cancellation test.
|
||
5. `internal/router/scheduler`: the `Filter`/`Scorer`/`Picker` interfaces plus the `queue`, `kv`
|
||
and `inflight` scorers. **Do not write a single "least-load" function** — build the pipeline
|
||
from the start; coming back to it later is expensive.
|
||
6. `internal/router/admission`: saturation detector and band queues. Phase 1.5.
|
||
7. Write the end-to-end tests for the phase 1 and phase 1.5 exit criteria, then move to phase 2.
|
||
|
||
> Use `internal/router/scheduler/` rather than `internal/router/balancer/` (the tree in §4
|
||
> reflects this): "balancer" understates the job — this component does more than balance load.
|