Ilker ManapandClaude Opus 5.5
cf53904cc5
A model's adapters are fetched by each node with its weights and loaded unapplied at start; a request naming an adapter reaches the model's replicas with llama.cpp's per-request lora field. Adapter names share the model namespace, are listed, and follow allowlists on their own. The router's config seed and pushed catalog now build the name map in one place. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1581 lines
82 KiB
Markdown
1581 lines
82 KiB
Markdown
# llmgw — A Single-Binary LLM Gateway for Multi-Fabric InfiniBand and Heterogeneous GPUs
|
||
|
||
> Status: Design / Plan v0.2
|
||
> What changed in v0.2: existing LLM gateway and router projects were surveyed (§2.5) and
|
||
> their critical designs folded in — above all **separating admission from scheduling** (§7.2),
|
||
> the plugin-based **filter → score → pick** pipeline (§7.3), and the **three-tier model of
|
||
> prefix/KV cache awareness** (§7.5).
|
||
> Goal: one Go binary (running under different roles) that serves the LLMs on our own GPU
|
||
> servers behind a single OpenAI-compatible endpoint. Behind it sit several independent
|
||
> InfiniBand fabrics, nodes without IB, and different GPU types.
|
||
|
||
---
|
||
|
||
## 1. Scope
|
||
|
||
### 1.1 In scope
|
||
- OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`), including SSE streaming.
|
||
- An **agent** on every GPU node: GPU and IB discovery; starting, supervising and stopping inference engine processes (vLLM / SGLang).
|
||
- **Control plane**: node inventory, model catalog, replica placement, state tracking.
|
||
- **Router**, in two layers: **admission** (priority bands, tenant fairness, queueing under
|
||
saturation) and **scheduling** (plugin-based filter/score/pick with queue, KV-cache and
|
||
prefix awareness), plus fallback, retry, cooldown and circuit breaking.
|
||
- Fabric awareness: a multi-node replica **never** spans two fabrics; a TP group **never** mixes GPU types.
|
||
- API keys, rate limits, token accounting and budgets, Prometheus metrics. Persistence for these is
|
||
the only thing the store exists for (§6.5, §7.9).
|
||
|
||
### 1.2 Out of scope (at least for v1)
|
||
- Inference itself. Go only orchestrates and proxies; vLLM/SGLang run the model.
|
||
- A Kubernetes operator (bare metal / systemd is the target; K8s can come later).
|
||
- Prefill/decode disaggregation (a v2 goal, see §12).
|
||
- External providers (OpenAI, Anthropic) — the architecture stays open to them, but not in v1.
|
||
|
||
---
|
||
|
||
## 2. Terminology
|
||
|
||
| Term | Meaning |
|
||
|---|---|
|
||
| **Node** | A physical server running the agent. |
|
||
| **Fabric / Island** | An independent IB fabric. Nodes without IB belong to the `none` island and each counts as its own single-node island. |
|
||
| **GPU Class** | A group of GPUs with the same model and VRAM (e.g. `h100-80g`, `a100-40g`, `l40s-45g`). |
|
||
| **Pool** | A (fabric, gpu class) pair. The unit of placement. |
|
||
| **Model** | A logical model in the catalog (e.g. `llama-3.3-70b`). This is the name users see. |
|
||
| **Profile** | How a model runs on a particular GPU class (TP, PP, quantization, max length, engine arguments). |
|
||
| **Replica** | One running copy of a model. May span 1..N nodes. This is the unit the router sees. |
|
||
| **Shard** | A replica's part on a single node (one engine process). |
|
||
| **Routing Profile** | The scorer set and weights the router uses for a class of request (chat / batch / long-context). Not to be confused with the model **Profile** above. |
|
||
| **Band** | A priority class (platinum/gold/silver/bronze). The unit of admission decisions. |
|
||
| **Block** | The unit of KV cache accounting; a fixed number of tokens (16 in vLLM, 64 by default in the GIE indexer). |
|
||
| **Unit** | The billing unit budgets are counted in. Self-hosted models have no per-token price, so an operator defines what a unit is: GPU-seconds, a notional rate per million tokens, or 1.0 everywhere, which turns budgets into token quotas (§7.9.4). |
|
||
|
||
---
|
||
|
||
## 2.5 Reference Implementations and What We Took
|
||
|
||
Rather than design from scratch, we inherited the problems already solved by mature projects
|
||
in this space.
|
||
|
||
| Project | What it is | What llmgw takes from it |
|
||
|---|---|---|
|
||
| **Gateway API Inference Extension (EPP)** | The K8s standard; an Endpoint Picker running over Envoy `ext_proc` | Admission/scheduling separation, the plugin-based `filter → score → pick` pipeline, the **model server metric contract** (§7.6) |
|
||
| **llm-d** | "Well-lit paths" built on GIE | KV-event-based **precise** prefix index, priority bands with tenant fairness, the saturation detector |
|
||
| **AIBrix** (a vLLM project) | An Envoy ext_proc gateway plus a heterogeneous GPU optimizer | The **heterogeneous GPU SLO/cost model** (§6.2.1), queueing at the gateway (`slo-least-load-pulling`), disabling prefix affinity under load imbalance |
|
||
| **NVIDIA Dynamo** | Radix-tree KV router plus SLA planner | The shape of the cost function (§7.4) and `router_temperature` softmax sampling against herding |
|
||
| **SGLang router** | Cache-aware load balancing over an approximate radix tree | v1's prefix answer: approximate tree plus threshold logic (§7.5 tier 1) |
|
||
| **vLLM production-stack / vllm-project/router** | vLLM's own routers | Session affinity (measured 96% vs 93% cache hit), consistent hashing |
|
||
| **LiteLLM** | The most widely deployed multi-provider proxy | Virtual key → team/project → budget hierarchy, cooldowns, keeping spend writes off the request path, budget-triggered fallback |
|
||
| **Envoy AI Gateway** | An Envoy-based AI gateway (v1.0, June 2026) | Token-based (not merely request-based) rate limiting |
|
||
|
||
What we deliberately **did not** take: a Kubernetes dependency (bare metal is the target), the
|
||
Envoy `ext_proc` architecture (single binary is the goal — we write the router directly in Go),
|
||
semantic routing / LLM-based model selection (out of scope), and external provider management
|
||
(not in v1).
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
┌───────────────────────────────────────────┐
|
||
Clients ────────► │ llmgw server │
|
||
(OpenAI API) │ ┌────────────┐ ┌──────────────────────┐ │
|
||
│ │ Router │ │ Control Plane │ │
|
||
│ │ (data) │◄─┤ - Inventory │ │
|
||
│ │ │ │ - Catalog │ │
|
||
│ └─────┬──────┘ │ - Scheduler │ │
|
||
│ │ │ - Store (phase 5) │ │
|
||
│ │ └─────────▲────────────┘ │
|
||
└────────┼───────────────────┼──────────────┘
|
||
HTTP (eth) │ │ gRPC stream (mTLS, agent dials out)
|
||
┌────────────────────┼───────────────────┼─────────────────────┐
|
||
▼ ▼ │ ▼
|
||
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
|
||
│ llmgw │ IB-A │ llmgw │ │ llmgw │ IB-B │ llmgw │
|
||
│ agent │◄──────►│ agent │ │ agent │◄───────►│ agent │ ...
|
||
│ vLLM rank0│ NCCL │ vLLM rank1│ │ SGLang │ │ (no IB) │
|
||
└───────────┘ └───────────┘ └───────────┘ └───────────┘
|
||
Fabric A island Fabric B island none island
|
||
```
|
||
|
||
### 3.1 One binary, several roles
|
||
|
||
```
|
||
llmgw server --config server.yaml # router + control plane (same process in v1)
|
||
llmgw agent --config agent.yaml # on every GPU node
|
||
llmgw ctl ... # CLI: list nodes, deploy models, drain, ...
|
||
llmgw sim-engine # fake OpenAI-compatible engine for testing
|
||
```
|
||
|
||
`router` and `control` must eventually be able to run as separate processes
|
||
(`llmgw server --roles=router`), so the dependency between them goes through an interface
|
||
(see §6.4).
|
||
|
||
### 3.2 Build note
|
||
`go-nvml` requires cgo. Put the agent's NVML code behind a `//go:build nvml` tag and use a
|
||
fake / `nvidia-smi`-based discovery path in untagged builds. That keeps the server side
|
||
cgo-free and lets development machines without GPUs run the tests.
|
||
|
||
---
|
||
|
||
## 4. Project Layout
|
||
|
||
```
|
||
llmgw/
|
||
├── cmd/llmgw/main.go # cobra root; subcommands
|
||
├── api/
|
||
│ └── proto/llmgw/v1/*.proto # agent <-> control plane
|
||
├── internal/
|
||
│ ├── config/ # YAML + env via koanf
|
||
│ ├── agent/
|
||
│ │ ├── agent.go # main loop, control plane connection
|
||
│ │ ├── discovery/
|
||
│ │ │ ├── gpu_nvml.go # //go:build nvml
|
||
│ │ │ ├── gpu_fallback.go
|
||
│ │ │ ├── ib.go # reads /sys/class/infiniband
|
||
│ │ │ └── topology.go # NVLink, PCIe, NUMA, GPU<->HCA affinity
|
||
│ │ ├── supervisor/ # engine process management (exec, restart, logs)
|
||
│ │ ├── engines/
|
||
│ │ │ ├── engine.go # Engine interface
|
||
│ │ │ ├── vllm.go
|
||
│ │ │ └── sglang.go
|
||
│ │ └── health/ # polls engine /health and /metrics
|
||
│ ├── control/
|
||
│ │ ├── inventory/ # node and GPU state
|
||
│ │ ├── catalog/ # model and profile definitions
|
||
│ │ ├── scheduler/ # placement algorithm
|
||
│ │ ├── reconciler/ # desired vs actual loop
|
||
│ │ └── agentsvc/ # gRPC server
|
||
│ ├── router/ # HTTP handlers, dispatch, metrics
|
||
│ │ ├── proxy/ # streaming proxy to upstream
|
||
│ │ ├── admission/ # saturation detector, band queues (§7.2)
|
||
│ │ ├── scheduler/ # Filter/Scorer/Picker pipeline (§7.3)
|
||
│ │ ├── prefix/ # tier 0 session hash, tier 1 approx tree, tier 2 kv index
|
||
│ │ ├── endpoint/ # live endpoint state, inflight, staleness
|
||
│ │ ├── breaker/ # circuit breaker + cooldown
|
||
│ │ └── scrape/ # reads engine /metrics (the §7.6 contract)
|
||
│ ├── auth/ # API keys, org/team/project, budgets
|
||
│ ├── ratelimit/ # token bucket (requests + tokens)
|
||
│ ├── usage/ # token counting and recording
|
||
│ ├── store/ # SQLite (modernc.org/sqlite), migrations
|
||
│ ├── obs/ # slog, prometheus, otel
|
||
│ └── sim/ # fake engine
|
||
├── deploy/
|
||
│ ├── systemd/
|
||
│ └── examples/
|
||
├── test/e2e/
|
||
└── docs/
|
||
```
|
||
|
||
> The OpenAI handlers live in `internal/router` itself rather than an `openai/`
|
||
> subpackage: they need the dispatch loop, and splitting them apart would have
|
||
> meant an interface whose only implementation is the router.
|
||
|
||
### 4.1 Libraries
|
||
|
||
| Need | Choice |
|
||
|---|---|
|
||
| CLI | `github.com/spf13/cobra` |
|
||
| Config | `github.com/knadh/koanf/v2` |
|
||
| RPC | `connectrpc.com/connect` (gRPC compatible, HTTP/2, easy to debug) |
|
||
| GPU | `github.com/NVIDIA/go-nvml` |
|
||
| DB | `modernc.org/sqlite` (cgo-free) + `pressly/goose` migrations |
|
||
| Metrics | `github.com/prometheus/client_golang`, parsing via `github.com/prometheus/common/expfmt` |
|
||
| Tracing | `go.opentelemetry.io/otel` |
|
||
| Logging | `log/slog` (stdlib) |
|
||
| Rate limiting | `golang.org/x/time/rate` (plus our own bucket for token limits) |
|
||
| Hashing | `github.com/cespare/xxhash/v2` |
|
||
| HA (v2) | `github.com/hashicorp/raft` or etcd |
|
||
| Testing | stdlib `testing`, `github.com/stretchr/testify` (optional) |
|
||
|
||
---
|
||
|
||
## 5. Agent
|
||
|
||
### 5.1 GPU discovery
|
||
For every GPU:
|
||
- index, UUID, name, VRAM, compute capability (for FP8 and similar support), driver/CUDA version
|
||
- NUMA node, PCIe bus id
|
||
- NVLink peers (NVML `GetNvLinkRemotePciInfo`) → which GPUs pair efficiently for TP within a node
|
||
- health: XID errors, ECC, temperature, memory in use (are there non-engine processes?)
|
||
|
||
`GPUClass` is derived automatically: `normalize(name) + "-" + round(vram_gib) + "g"` → `h100-80g`.
|
||
It can be overridden in configuration.
|
||
|
||
The size comes from what the driver reports, in GiB, which does not always match the marketing
|
||
name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is `l40s-45g`. Consistency
|
||
matters more than matching the box here, because the number is what decides whether a model
|
||
fits. The model token is kept in full for the same reason — a 4090 and a 4080 are not
|
||
interchangeable, so the class is `4090-24g`, not `rtx-24g`.
|
||
|
||
### 5.2 IB discovery and fabric identity
|
||
From `/sys/class/infiniband/<hca>/ports/<n>/`:
|
||
- `state` (ACTIVE?), `phys_state`, `rate`, `link_layer` (InfiniBand / Ethernet → RoCE)
|
||
- `gids/0` → subnet prefix
|
||
- `sm_lid`, `lid`
|
||
- the HCA's PCIe address and NUMA node → GPU affinity
|
||
|
||
**How is fabric identity determined?** Automatic detection is not reliable, because OpenSM's
|
||
default subnet prefix (`fe80::`) is the same on every fabric. In order of preference:
|
||
|
||
1. An explicit label in the agent config: `fabrics: [{name: ib-a, hcas: [mlx5_0, mlx5_1]}]` **(recommended)**
|
||
2. If every OpenSM has a unique `subnet_prefix`, automatic mapping from the GID prefix (a `prefix → fabric` table in the server config)
|
||
3. Otherwise the node is marked `fabric=unknown` and **excluded from multi-node placement**.
|
||
|
||
Bonus verification (v1.1): for two nodes claiming the same fabric, the agents verify
|
||
reachability with a short `ibv_rc_pingpong`-style test or `ibping`. A mismatch raises an alert.
|
||
|
||
A node may have HCAs attached to more than one fabric, and the model must allow it
|
||
(`Node.Fabrics []FabricAttachment`). A replica, however, picks exactly one fabric and uses only
|
||
that fabric's HCAs.
|
||
|
||
### 5.3 Engine interface
|
||
|
||
```go
|
||
type Engine interface {
|
||
Name() string // "vllm", "sglang"
|
||
Validate(p Profile, gpus []GPU) error // compute capability, quantization fit, ...
|
||
BuildCommand(spec ShardSpec) (*exec.Cmd, error) // builds arguments and environment
|
||
HealthURL(spec ShardSpec) string
|
||
MetricsURL(spec ShardSpec) string
|
||
ParseMetrics(r io.Reader) (EngineStats, error) // engine-specific metric names
|
||
}
|
||
|
||
type ShardSpec struct {
|
||
ReplicaID string
|
||
Rank int // 0 = leader (the one serving HTTP)
|
||
NNodes int
|
||
LeaderAddr string // dist init address (ethernet IP)
|
||
LeaderPort int
|
||
GPUIndices []int
|
||
HCAs []string // HCAs of the chosen fabric
|
||
HTTPPort int
|
||
Profile Profile
|
||
ModelPath string
|
||
}
|
||
|
||
type EngineStats struct {
|
||
// The mandatory contract from §7.6 — without these the router cannot route.
|
||
Running, Waiting int
|
||
KVCacheUsage float64 // 0..1 (vllm:kv_cache_usage_perc)
|
||
// Optional.
|
||
BlockSize, NumGPUBlocks int
|
||
PrefixHitRate float64
|
||
TTFTp50, TPOTp50 time.Duration // from histograms
|
||
DrainRate float64 // observed completions/sec; for the `load` scorer
|
||
LoRA LoRAState // v2
|
||
ScrapedAt time.Time // for staleness detection — the router uses this
|
||
}
|
||
```
|
||
|
||
Supporting a new engine is defined as: being able to fill in the mandatory three from §7.6.
|
||
|
||
**Multi-node launch:**
|
||
- SGLang supports it natively: `--nnodes N --node-rank R --dist-init-addr host:port --tp T`.
|
||
- vLLM: depending on the version, either Ray (`ray start --head` on the leader, `ray start --address` on the workers) or the newer multi-node multiprocessing flags. **Check the documentation for the vLLM version you deploy**; `vllm.go` should offer both paths behind a `launchMode`.
|
||
|
||
**Environment generation (critical):**
|
||
```
|
||
CUDA_VISIBLE_DEVICES=<GPUIndices>
|
||
NCCL_IB_HCA=<HCAs, comma separated> # the chosen fabric only
|
||
NCCL_SOCKET_IFNAME=<mgmt interface> # bootstrap
|
||
NCCL_IB_DISABLE=1 # single node, or no IB
|
||
NCCL_CROSS_NIC / NCCL_NET_GDR_LEVEL # optional, from the profile
|
||
GLOO_SOCKET_IFNAME=<mgmt interface>
|
||
```
|
||
|
||
### 5.4 Supervisor
|
||
- Start processes in their own process group (`SysProcAttr.Setpgid`) and kill the whole group when stopping (SIGTERM → timeout → SIGKILL).
|
||
- stdout/stderr → line-oriented slog plus a ring buffer (the last N lines, for `llmgw ctl logs`).
|
||
- State machine: `Pending → Starting → Loading → Ready → Draining → Stopped | Failed`.
|
||
- `Ready` only once `/health` returns 200 **and** `/v1/models` lists the model.
|
||
- Model loading takes a long time; `startupTimeout` comes from the profile (e.g. 20 min).
|
||
- On a crash: report to the control plane and let **the control plane** decide about restarting. Restarting a single shard of a multi-node replica accomplishes nothing; the whole replica must be rebuilt.
|
||
- If the agent restarts, try to adopt still-running engine processes (pid file plus cmdline check) and clean up whatever cannot be adopted.
|
||
|
||
### 5.5 Model weights
|
||
- v1: a shared filesystem (NFS/Lustre/Weka) or a path pre-populated on the node. The agent only checks existence and size.
|
||
- v1.1: the agent downloads from HF/S3 into a local cache, reports progress, and enforces a disk quota.
|
||
|
||
### 5.6 Agent ↔ control plane protocol
|
||
The agent dials out (NAT- and firewall-friendly) over a bidirectional stream:
|
||
|
||
```proto
|
||
service AgentService {
|
||
rpc Connect(stream AgentMessage) returns (stream ControlMessage);
|
||
}
|
||
|
||
message AgentMessage {
|
||
oneof msg {
|
||
Register register = 1; // node info, GPUs, fabrics, versions
|
||
Heartbeat heartbeat = 2; // every 5s: GPU health, shard states
|
||
ShardStatus shard = 3; // sent on every state change
|
||
EngineStats stats = 4; // every 2s (the router may also scrape directly)
|
||
}
|
||
}
|
||
|
||
message ControlMessage {
|
||
oneof msg {
|
||
StartShard start = 1;
|
||
StopShard stop = 2;
|
||
DrainShard drain = 3;
|
||
Ack ack = 4;
|
||
}
|
||
}
|
||
```
|
||
|
||
- Commands are idempotent: a repeated `StartShard` with the same `shard_id` is a no-op.
|
||
- mTLS, with node identity taken from the certificate CN. The control plane
|
||
compares the CN against the node id the agent claims and refuses the stream
|
||
when they disagree, which is what makes a node's identity something it proves
|
||
rather than something it picks.
|
||
- The CA is internal, and has to be: public authorities stopped issuing the
|
||
`clientAuth` usage, so no Let's Encrypt certificate can authenticate an agent
|
||
however convenient it would be. `llmgw ctl cert init-ca | issue-server |
|
||
issue-client` runs a small authority whose key lives on one machine and whose
|
||
root is trusted everywhere.
|
||
- The server certificate must carry every address agents dial as a subject
|
||
alternative name, bare IP addresses included. Hostname verification ignores
|
||
the common name entirely, so a certificate that names the host only in its
|
||
subject is rejected by every client.
|
||
- This covers the fleet-facing port only. The client-facing API is a public
|
||
surface and is fronted by a publicly trusted certificate instead.
|
||
- Three missed heartbeats mark the node `Unreachable`; after 60s it becomes `Lost` and its replicas are rescheduled.
|
||
|
||
---
|
||
|
||
## 6. Control Plane
|
||
|
||
### 6.1 Data model (abridged)
|
||
|
||
```go
|
||
type Node struct {
|
||
ID, Hostname, MgmtAddr string
|
||
Labels map[string]string
|
||
GPUs []GPU
|
||
Fabrics []FabricAttachment // empty means no IB
|
||
State NodeState // Ready, Cordoned, Draining, Unreachable, Lost
|
||
LastSeen time.Time
|
||
}
|
||
|
||
type FabricAttachment struct {
|
||
Fabric string // "ib-a"
|
||
HCAs []string
|
||
Rate int // Gbps
|
||
}
|
||
|
||
type Model struct {
|
||
Name string // what users see
|
||
Aliases []string
|
||
Source string // path or HF id
|
||
Profiles []Profile
|
||
Scaling Scaling // min/max replicas, target queue, ...
|
||
Fallbacks []string // other model names
|
||
}
|
||
|
||
type Profile struct {
|
||
Name string
|
||
GPUClass string // "h100-80g"
|
||
Engine string // "vllm"
|
||
TP, PP int
|
||
GPUsPerNode int // usually TP
|
||
Nodes int // PP*... ; a fabric is mandatory when > 1
|
||
Args []string // --quantization fp8 --max-model-len 32768 ...
|
||
Env map[string]string
|
||
Weight int // capacity weight for the router (from benchmarks)
|
||
Priority int // placement preference
|
||
StartupTimeout time.Duration
|
||
}
|
||
|
||
type Replica struct {
|
||
ID, Model, Profile string
|
||
Fabric string // set for multi-node replicas
|
||
Shards []Shard
|
||
State ReplicaState
|
||
Endpoint string // rank0's HTTP address
|
||
}
|
||
```
|
||
|
||
### 6.2 Scheduler (placement)
|
||
|
||
Input: the desired replica count, the profile list, and the inventory.
|
||
|
||
**Hard** constraints:
|
||
1. All shards share one `GPUClass`.
|
||
2. When `Nodes > 1`, every node must attach to the same `Fabric` and have ACTIVE HCAs on it.
|
||
3. `GPUsPerNode` free GPUs, preferably in the same NVLink domain and on the same NUMA node as the HCA.
|
||
4. The node is `Ready` and not `Cordoned`.
|
||
5. The engine's `Validate` must pass (e.g. an FP8 profile never lands on a card with compute capability < 8.9).
|
||
|
||
**Soft** preferences (scored):
|
||
- the profile's `Priority`
|
||
- bin packing: fill partially used nodes first, keeping empty nodes free for large models
|
||
- spread a model's replicas across fabrics and nodes (failure domain distribution)
|
||
- GPU-to-HCA NUMA affinity
|
||
|
||
Algorithm (v1, simple and deterministic):
|
||
|
||
```
|
||
for each profile in model.Profiles sorted by Priority desc:
|
||
pools := groupNodes(by GPUClass == profile.GPUClass, by Fabric if profile.Nodes > 1)
|
||
for each pool sorted by score:
|
||
nodes := pickNodes(pool, profile.Nodes, profile.GPUsPerNode) // greedy + NVLink group selection
|
||
if nodes != nil:
|
||
return Placement{profile, pool.Fabric, nodes, gpuAssignments}
|
||
return ErrUnschedulable (with reasons — essential for debugging)
|
||
```
|
||
|
||
`groupNodes`: nodes without IB join the pool for `Nodes == 1` profiles, and never for `Nodes > 1`.
|
||
|
||
`llmgw ctl explain-placement <model>` must show why each pool was rejected.
|
||
|
||
### 6.2.1 Heterogeneous GPUs: the SLO/capacity model
|
||
|
||
Comparing GPU classes through a single scalar "weight" is too crude; the model taken from
|
||
AIBrix's heterogeneous GPU optimizer is more accurate. For each (model, profile) pair,
|
||
`ctl bench` (§10) produces a **capacity profile**:
|
||
|
||
```go
|
||
type CapacityProfile struct {
|
||
Model, Profile string
|
||
GPUClass string
|
||
// Maximum concurrency that still meets the SLO; the real capacity measure.
|
||
MaxConcurrencyAtSLO int
|
||
ThroughputTokPerSec float64
|
||
TTFTp99, TPOTp99 time.Duration
|
||
RelativeCost float64 // relative GPU-hour cost, from configuration
|
||
MeasuredAt time.Time
|
||
EngineVersion string // the profile expires when the version changes
|
||
}
|
||
```
|
||
|
||
An example SLO: `TPOT p99 ≤ 50ms && TTFT p99 ≤ 2s`.
|
||
|
||
The profile is used in three places:
|
||
1. **Router weight:** `Profile.Weight` is no longer hand-written; it derives from `MaxConcurrencyAtSLO`.
|
||
2. **Admission thresholds:** `TargetWaitingPerReplica` (§7.2.1) is suggested from the measurement.
|
||
3. **Placement and autoscaling objective:** "minimize total cost while meeting the SLO". For a
|
||
predicted load, the replica count needed per GPU class is `ceil(load / MaxConcurrencyAtSLO)`,
|
||
and the cheapest combination wins. In v1 this only produces advice via
|
||
`ctl plan <model> --target-rps N`; automatic application arrives with the v1.1 autoscaler.
|
||
|
||
When `EngineVersion` or the profile arguments change the measurement goes stale. llmgw must
|
||
surface that (`llmgw_capacity_profile_stale{model,profile}`) rather than silently reusing an
|
||
obsolete number.
|
||
|
||
### 6.3 Reconciler
|
||
- Every 5s, or on an event: compare desired (model scaling) with actual (replica states).
|
||
- Missing replica → schedule → send `StartShard` for each shard (workers first, then rank0, or whatever order the engine requires).
|
||
- Any shard in a replica `Failed` → stop the whole replica and reschedule with backoff.
|
||
- Excess replicas → `Drain` first (the router stops sending new requests, existing ones finish), then `Stop`.
|
||
- Node `Lost` → treat its replicas as `Failed`.
|
||
|
||
Autoscaling (v1.1): a per-model `targetWaitingPerReplica`, averaged from router metrics, with
|
||
cooldown-guarded scaling. Shrinking lower-priority models when no GPUs are free (preemption) is v2.
|
||
|
||
### 6.4 Interface to the router
|
||
|
||
```go
|
||
type EndpointSource interface {
|
||
// model name -> Ready replicas (+ pool, weight, fabric, gpuclass, capacity profile)
|
||
Endpoints(model string) []Endpoint
|
||
Subscribe() <-chan struct{} // change notification
|
||
}
|
||
```
|
||
In-memory in v1; once the router and control plane split apart, the same interface is backed by
|
||
a watch stream.
|
||
|
||
The router **always caches** this list: even if the control plane goes down it keeps routing to
|
||
the replicas it knows about, which mitigates the single-point-of-failure risk in §14. When the
|
||
cache ages past `endpoint_cache_max_age` (say 5 min) it only raises a warning metric; it never
|
||
stops traffic.
|
||
|
||
### 6.5 What is stored, and what is not
|
||
|
||
v0.1 of this plan listed eight tables. Most of them were a mistake, and the
|
||
reason is worth stating because it applies to every piece of state here:
|
||
|
||
> **A database that describes the cluster can be wrong. The cluster cannot.**
|
||
|
||
An engine that crashes during a restart leaves a row saying "running". A node
|
||
whose GPU developed an ECC fault leaves a row saying the card is fine. Every
|
||
such row is a trap: the scheduler believes it, places work accordingly, and the
|
||
failure surfaces somewhere else entirely. Rebuilding the same state from the
|
||
fleet on every reconnect cannot go stale, because the fleet is the thing being
|
||
described.
|
||
|
||
So:
|
||
|
||
| State | Persisted | Where it actually comes from |
|
||
|---|---|---|
|
||
| Nodes, GPUs, fabrics | no | Every agent sends its full inventory on registration (§5.6) |
|
||
| Replicas, shards | no | Reported by agents; see the restart hole below |
|
||
| Models, profiles | no | The YAML catalog is the source of truth |
|
||
| API keys and the tenancy tree | **yes** | Created at runtime, secret, rotated (§7.9) |
|
||
| Usage records and rollups | **yes** | Accounting must survive a restart (§7.9) |
|
||
| Capacity profiles from `ctl bench` | **yes** | Measured, not declared (§6.2.1) |
|
||
| Audit events | **yes** | An audit trail that forgets is not one |
|
||
|
||
Everything in the "yes" column belongs to tenancy and benchmarking, which is
|
||
phase 5. **The store lands there, not in phase 3**, so its schema is shaped by
|
||
real requirements instead of guesses.
|
||
|
||
#### The control plane restart hole
|
||
|
||
Not persisting replica state has one consequence that must be closed before it
|
||
matters. If the control plane restarts while engines are running:
|
||
|
||
1. The engines keep serving — correct, and the reason the router talks to them
|
||
directly.
|
||
2. Agents reconnect and re-register their hardware.
|
||
3. The control plane has no memory of any replica, because `Register` does not
|
||
carry one.
|
||
4. The reconciler sees zero replicas and places new ones.
|
||
5. The old engines keep holding GPUs that the inventory now believes are free,
|
||
so the new engines fail to allocate — with an error pointing at the new
|
||
process rather than the old one.
|
||
|
||
The fix is on the agent side, not in a database: `Register` carries the shards
|
||
the agent is currently supervising, and the control plane rebuilds replica state
|
||
from what the fleet reports. The supervisor already knows its live shards and
|
||
already finds, through pid files, the ones that survived an agent restart
|
||
(`FindOrphans`). A shard nothing claims after one reconciliation pass is an
|
||
orphan and is killed.
|
||
|
||
This is strictly better than a database would be: it cannot disagree with
|
||
reality, and it covers agent restarts and control plane restarts with one
|
||
mechanism.
|
||
|
||
#### Store choice
|
||
|
||
SQLite in WAL mode, via `modernc.org/sqlite` so the server stays cgo-free.
|
||
|
||
The deciding argument is not throughput. One row per request at 1000 req/s is 86
|
||
million rows a day, which sounds like a reason to reach for Postgres, but the
|
||
write path is batched anyway (§7.9.3) and a transaction of 500 rows every 200ms
|
||
is nothing for either engine. Batching is required for a different reason, and
|
||
once it exists the volume argument favours neither.
|
||
|
||
What decides it is the failure mode. Postgres is a second service, and the
|
||
question it raises is whether the gateway still serves traffic when that service
|
||
is unreachable. An inference gateway that stops serving because its accounting
|
||
database is down is worse than one that buffers some accounting. SQLite is
|
||
embedded, so the store shares the gateway's fate rather than adding an
|
||
independent way to be down — and the single-binary goal falls out of the same
|
||
choice.
|
||
|
||
It stops being enough at the point where there is more than one gateway
|
||
instance, because budgets and rate-limit counters then have to be counted
|
||
somewhere shared; per-instance counters mean N instances allow N times the
|
||
configured limit (§7.9.6). Postgres arrives with that, not before.
|
||
|
||
Three rules keep the swap cheap, and they are worth following from the first
|
||
commit:
|
||
|
||
1. **Nothing on the request path touches the database.** Keys are cached in
|
||
memory and read through on a miss, usage goes to a batch writer, budgets are
|
||
in-memory counters reconciled against the store. This matters more than the
|
||
engine choice and is what makes either engine survivable.
|
||
2. **Portable SQL.** No `AUTOINCREMENT` and no rowid tricks, `ON CONFLICT … DO
|
||
UPDATE` rather than `INSERT OR REPLACE`, timestamps as integer unix millis,
|
||
money and tokens as integers.
|
||
3. **Migrations per dialect, sharing version numbers.** A single migration set
|
||
for both engines would be a promise that cannot be kept — SQLite's
|
||
`ALTER TABLE` barely exists. Same numbering, same meaning, different files.
|
||
|
||
---
|
||
|
||
## 7. Router
|
||
|
||
### 7.1 Request lifecycle
|
||
|
||
```
|
||
Request
|
||
├─ 1. Auth : key → project/team, permitted models, priority band, routing profile
|
||
├─ 2. Quota : RPM/TPM and budget (in-memory counters; no DB on the request path)
|
||
├─ 3. Parse : model, stream, messages/prompt, max_tokens, user.
|
||
│ The rest of the body passes through as untouched JSON.
|
||
├─ 4. Resolve : model/alias → candidate pools (fabric × gpu_class)
|
||
├─ 5. ADMISSION : "now?" §7.2 ← waits here while the pool is saturated
|
||
├─ 6. SCHEDULING : "where?" §7.3 filter → score → pick
|
||
├─ 7. Proxy : stream to upstream, propagate cancellation §7.8
|
||
└─ 8. After the request (async): usage, spend, metrics, logs
|
||
```
|
||
|
||
**The critical separation (taken from GIE / llm-d).** *When* a request proceeds and *where* it
|
||
goes are two distinct decisions and must stay distinct. Kept apart, the router can hold requests
|
||
in its own central queue and apply priority and tenant fairness **before they enter an engine
|
||
queue**.
|
||
|
||
Why this matters: an engine's own queue is FIFO and has no notion of tenants or priority. Once a
|
||
request is in there, no prioritization is possible. When a batch job fills the vLLM queue with
|
||
500 requests, the TTFT of an interactive chat request arriving behind them collapses and there
|
||
is nothing the router can do. v0.1 had no such layer; **for a shared cluster this was the most
|
||
serious omission.**
|
||
|
||
### 7.2 Admission control (flow control)
|
||
|
||
#### 7.2.1 Saturation detector
|
||
|
||
One saturation signal is produced per pool, combining two metrics (llm-d's approach — queue
|
||
depth alone is not enough, KV pressure is a genuine exhaustion signal too):
|
||
|
||
```go
|
||
// per replica, then aggregated across the pool as p90 or mean
|
||
func saturation(e EngineStats, t Thresholds) float64 {
|
||
q := float64(e.Waiting) / t.TargetWaitingPerReplica // e.g. target 4
|
||
kv := e.KVCacheUsage / t.KVCeiling // e.g. ceiling 0.90
|
||
return math.Max(q, kv)
|
||
}
|
||
```
|
||
|
||
The goal is to keep a **small but healthy buffer** on the engine: enough requests for continuous
|
||
batching to stay efficient, but not enough for the queue to balloon. Set
|
||
`TargetWaitingPerReplica` too low and the GPU starves; too high and queueing delay returns. Both
|
||
thresholds must be tunable per model/profile, and `ctl bench` (§10) should propose them.
|
||
|
||
**Two lags have to be closed, and they need different answers.** The engine's queue is scraped,
|
||
so it is up to one interval old; the router's own inflight count is not. But inflight includes
|
||
*running* requests, and treating those as queue makes a perfectly healthy engine look overloaded
|
||
every interval. What is actually queued is `inflight − batch capacity`, and batch capacity is not
|
||
in the metric contract — so llmgw learns it as the high-water mark of observed `running`. The
|
||
second lag is shorter and subtler: between admission granting a request and the router
|
||
dispatching it, the request is invisible everywhere. A simultaneous burst can therefore pass the
|
||
check en masse, which is exactly the batch-job arrival pattern this layer exists to survive. The
|
||
controller closes it by counting its own outstanding leases against dispatched inflight.
|
||
|
||
#### 7.2.2 Priority bands
|
||
|
||
Every API key maps to a band, and the band sets the saturation ceiling:
|
||
|
||
```yaml
|
||
bands:
|
||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 } # flows to the healthy limit
|
||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 } # batch jobs
|
||
```
|
||
|
||
A ceiling of 1.00 means "hold only once the pool is past its healthy limit",
|
||
not "never hold". Holding even the top band is deliberate: there is nobody above
|
||
it to protect, so the benefit is a bounded engine queue. A flood dumped straight
|
||
into a FIFO engine queue makes every request in that band slow, while queueing it
|
||
at the gateway preserves ordering, TTL and tenant fairness. A band that must
|
||
never wait — a health probe, say — gets a ceiling above 1 (`1000`).
|
||
|
||
The dispatch loop scans bands from highest to lowest. Once pool saturation exceeds a band's
|
||
ceiling, that band's requests wait in the queue. The result: as the cluster fills, batch traffic
|
||
slows down first and interactive traffic is protected. (In llm-d's measurements platinum, gold
|
||
and silver stayed under a second while bronze batch fell far behind — exactly the desired
|
||
behaviour.)
|
||
|
||
#### 7.2.3 Fairness and queue management
|
||
|
||
- **Fairness within a band:** when several tenants share a band, dispatch turns are distributed
|
||
round-robin. One tenant's traffic spike cannot consume the whole band.
|
||
- **Capacity:** each band's queue is bounded. When it is full, new requests get an immediate
|
||
`503` — far better than accumulating silently.
|
||
- **TTL:** a request that waits longer than `queue_ttl` is dropped (`504`). There is no point
|
||
sending the GPU work the client has already given up on.
|
||
- **Client disconnect:** if the connection drops while queued, the request is discarded quietly
|
||
(context cancellation).
|
||
- **Displacement (v1.1):** to make room for a higher band, drop the **newest** requests from the
|
||
lowest band — the oldest ones are already near their TTL and have invested the most waiting.
|
||
|
||
```go
|
||
type Admission interface {
|
||
// Blocks until the request may proceed; errors on cancellation or TTL.
|
||
Acquire(ctx context.Context, req *RequestInfo) (Lease, error)
|
||
}
|
||
```
|
||
|
||
`Lease.Release()` is called once the response completes; it also maintains the router's inflight
|
||
counter.
|
||
|
||
### 7.3 Scheduler: filter → score → pick
|
||
|
||
The list of "strategies" from v0.1 is replaced by a plugin pipeline (the GIE EPP model). No
|
||
strategy is a separate monolith; scorers are combined by weight.
|
||
|
||
```go
|
||
type Filter interface { // drops candidates (hard constraints)
|
||
Name() string
|
||
Filter(ctx *SchedCtx, eps []Endpoint) []Endpoint
|
||
}
|
||
|
||
type Scorer interface { // 0..1 score; 1 is best
|
||
Name() string
|
||
Score(ctx *SchedCtx, eps []Endpoint) map[EndpointID]float64
|
||
}
|
||
|
||
type Picker interface { // chooses from the scores
|
||
Pick(ctx *SchedCtx, scores map[EndpointID]float64) Endpoint
|
||
}
|
||
|
||
type RoutingProfile struct {
|
||
Name string
|
||
Filters []Filter
|
||
Scorers []WeightedScorer // {Scorer, Weight}
|
||
Picker Picker
|
||
}
|
||
```
|
||
|
||
Total score: `score(ep) = Σ wᵢ · scoreᵢ(ep)`, with each scorer normalizing to 0..1 internally.
|
||
|
||
**A two-stage decision.** Because the topology is heterogeneous, the pool is chosen first and
|
||
the replica second:
|
||
|
||
1. **Pool selection:** the pool with the highest `Priority`; if every replica there is saturated
|
||
(§7.2.1), spill over to the next pool. This prevents falling back to the A100 pool before the
|
||
H100 pool is actually full.
|
||
2. **Replica selection:** filter/score/pick within the chosen pool.
|
||
|
||
**Routing profiles**, from configuration (a per-model default, overridable per request via the
|
||
`x-llmgw-profile` header):
|
||
|
||
| Profile | Weights | Use |
|
||
|---|---|---|
|
||
| `chat` | prefix 3.0, queue 1.0, kv 1.0 | Multi-turn chat and agents — high cache reuse |
|
||
| `batch` | queue 3.0, kv 1.0, prefix 0.5 | Bulk work — throughput matters, TTFT does not |
|
||
| `long-context` | kv 3.0, prefix 2.0, queue 1.0 | Long prompts — KV pressure is the real bottleneck |
|
||
| `debug` | round-robin picker | Troubleshooting |
|
||
|
||
### 7.4 Scorers and their formulas
|
||
|
||
| Scorer | Formula | Notes |
|
||
|---|---|---|
|
||
| `queue` | `1 − min(1, waiting / target_waiting)` | Queue depth |
|
||
| `kv` | `1 − kv_cache_usage` | KV occupancy; a hard penalty above the ceiling |
|
||
| `prefix` | `matched_blocks / prompt_blocks` | §7.5 |
|
||
| `load` | `1 − norm(running / drain_rate)` | AIBrix: `drain_rate` is the observed completion rate. `running` alone cannot spot a slow replica. |
|
||
| `inflight` | `1 − norm(router_inflight)` | The router's own count; compensates for scrape lag. **At a 2s scrape interval, herding is unavoidable without it.** |
|
||
| `pool-priority` | constant, from the profile's `Priority` | Pool preference |
|
||
| `session` | session hash match | §7.5 tier 0 |
|
||
| `lora` (v2) | is the adapter loaded | `vllm:lora_requests_info` |
|
||
|
||
**An alternative cost formulation (Dynamo).** Modelling prefill and decode cost directly in
|
||
blocks may be more accurate than normalized scores:
|
||
|
||
```
|
||
cost(ep) = w_overlap · (prompt_blocks − cached_blocks(ep)) + decode_blocks(ep)
|
||
choice = argmin cost
|
||
```
|
||
|
||
A high `w_overlap` favours cache reuse (better TTFT); a low one evens out load (better ITL).
|
||
Both can be expressed as a `Scorer`, so the pipeline survives either way: weighted scores in v1,
|
||
with this cost form added as a scorer in v1.1 and compared using `ctl bench`.
|
||
|
||
**Pickers:**
|
||
- `max-score`: the highest score. Deterministic but prone to herding.
|
||
- `p2c` (power of two choices): two random candidates, the better one. The default once there are many replicas.
|
||
- `softmax(temperature)`: Dynamo's `router_temperature`. Treats scores as logits and samples. `temperature=0` degenerates to argmax. A small temperature breaks the oscillation that synchronized decisions cause among equally scored replicas.
|
||
|
||
### 7.5 Prefix / KV cache awareness — three tiers
|
||
|
||
This is the single largest win the router can deliver, but the cost gap between the tiers is
|
||
enormous. They stay separate so they can be enabled per model.
|
||
|
||
#### Tier 0 — session affinity (v1, cheap, high return)
|
||
The client-supplied `x-session-id` header (falling back to OpenAI's `user` field) maps to a
|
||
replica via bounded-load consistent hashing. If that replica is overloaded — beyond
|
||
`load_factor` times the average — the next one on the ring is used. Roughly 100 lines of code;
|
||
production-stack measured cache hits going from about 93% to 96% versus round robin. For
|
||
multi-turn chat and agent workloads this is the best cost/benefit ratio available.
|
||
|
||
#### Tier 1 — approximate radix tree (v1.1, moderate cost)
|
||
The router maintains a tree that **estimates** each worker's cache from the request prefixes it
|
||
has seen (the SGLang router's approach). It does not know the true cache state; it updates
|
||
lazily and costs little.
|
||
|
||
The thresholds — the part that matters in real implementations:
|
||
- `cache_threshold`: below this match ratio, treat it as "no cache" and let load balancing decide.
|
||
- `balance_abs_threshold` **and** `balance_rel_threshold`: if `max_load − min_load > abs`
|
||
**and** `max_load / min_load > rel`, **ignore the prefix entirely** and send to the least
|
||
loaded replica. (AIBrix adds the same guard.) Without it, prefix affinity drowns one replica:
|
||
the cache hit rate rises while p99 latency collapses.
|
||
- `eviction_interval` plus `max_tree_size`: keep the tree from growing without bound.
|
||
|
||
Which branch a decision took must be recorded as a metric: `cache_hit | cache_miss_min_load |
|
||
load_balance_override | stale | no_tree | no_healthy_worker`. Without that, tuning prefix routing
|
||
is guesswork.
|
||
|
||
#### Tier 2 — precise KV index (v2, expensive, optional)
|
||
vLLM publishes block events over ZMQ with `--kv-events-config` (`BlockStored`, `BlockRemoved`).
|
||
The router subscribes and maintains a `block_hash → {replica}` map, giving it a true global view.
|
||
|
||
**The trap — a cost the plan must acknowledge:** block hashes must be computed identically to
|
||
vLLM's **token-id-based** scheme. That forces the router to tokenize requests with **the same
|
||
tokenizer as the engine**. On the Go side that means loading a tokenizer per model (HF
|
||
`tokenizers` via cgo, or a pure-Go implementation), tracking version compatibility, and paying
|
||
the tokenization CPU cost per request. This is why AIBrix requires a "remote tokenizer".
|
||
**Tiers 0 and 1 need no tokenizer** — hence they are v1 and this is v2.
|
||
|
||
Event loss: if a gap appears in the event sequence numbers, request the missing range from the
|
||
replay endpoint; if that fails, drop the index for that replica and fall back to **tier 1** for
|
||
it. Returning to approximate information beats silently routing on wrong data. If a replica
|
||
restarts, its index is reset.
|
||
|
||
### 7.6 The engine metric contract
|
||
|
||
This is the formal definition of "adding a new engine". If an engine exposes these metrics
|
||
in Prometheus format, llmgw can route to it (GIE's model server protocol):
|
||
|
||
| Contract field | vLLM metric | llama.cpp metric | Mandatory |
|
||
|---|---|---|---|
|
||
| `Waiting` | `vllm:num_requests_waiting` | `llamacpp:requests_deferred` | yes |
|
||
| `Running` | `vllm:num_requests_running` | `llamacpp:requests_processing` | yes |
|
||
| `KVCacheUsage` | `vllm:kv_cache_usage_perc` | — | where the engine has it |
|
||
| `PromptTokens` | `vllm:prompt_tokens_total` | `llamacpp:prompt_tokens_total` | no |
|
||
| `GenerationTokens` | `vllm:generation_tokens_total` | `llamacpp:tokens_predicted_total` | no |
|
||
| `BlockSize`, `NumGPUBlocks` | `vllm:cache_config_info` | — | no |
|
||
| LoRA state | `vllm:lora_requests_info` (`max_lora`, `running_lora_adapters`, `waiting_lora_adapters`) | — | no |
|
||
|
||
Mappings are **per engine, not per cluster**. A cluster running llama.cpp on its CPU nodes and
|
||
vLLM on its GPU nodes has two vocabularies for the same numbers, and one global mapping would
|
||
leave half the fleet unmeasurable. An endpoint that names no engine is read with vLLM's names.
|
||
|
||
`Running` and `Waiting` are the two that cannot be omitted: an engine that reports neither how
|
||
much it is doing nor how much is queued cannot be scheduled against at all. `KVCacheUsage` is
|
||
required *of engines that publish it* — a mapping that names it and does not get it is a contract
|
||
violation, while a mapping that leaves it unnamed is a statement that this engine has no such
|
||
metric. Current llama.cpp is the second case, and saturation there falls back to queue depth
|
||
alone. That is a real loss of signal, mitigated by llama.cpp serving a fixed number of slots, so
|
||
its queue depth already says when it is full.
|
||
|
||
> Correction: an earlier version of this plan gave llama.cpp's KV metric as
|
||
> `llamacpp:kv_cache_usage_ratio`. Build b10991 does not publish it — it serves fifteen metric
|
||
> families and none is a cache ratio. Naming it failed the contract on every scrape and left the
|
||
> router with no numbers at all rather than two out of three.
|
||
|
||
> Correction: v0.1 said `vllm:gpu_cache_usage_perc`; the correct name is
|
||
> `vllm:kv_cache_usage_perc`. Metric names drift between versions, so the `ParseMetrics` mapping
|
||
> must be overridable from configuration, and the agent must report the engine version when it
|
||
> registers.
|
||
|
||
Scrape rules:
|
||
- Every Ready endpoint every 1–2s, one goroutine per endpoint, with jitter.
|
||
- A failed scrape marks the endpoint stale; its score drops but it is not removed.
|
||
- Three consecutive failures trip the breaker (§7.7).
|
||
- A **contract violation is reported at warn level, once**, and again when it clears. It is a
|
||
compatibility problem rather than a blip: every scrape fails, the router schedules to that
|
||
endpoint blind, and per-scrape debug logging meant a cluster could look entirely healthy while
|
||
none of its routing decisions were informed by anything.
|
||
- **So that decisions are never made purely on stale data**, the `inflight` scorer (§7.4) stays
|
||
enabled at all times.
|
||
|
||
### 7.7 Reliability
|
||
|
||
- **Retry:** only before the first byte has been written upstream, only on connection errors,
|
||
503 or 429, always against a different endpoint, at most N times. No retries once the stream
|
||
has started.
|
||
- **Cooldown (the LiteLLM lesson):** an endpoint that errors leaves the candidate pool for
|
||
`cooldown_seconds`. A long fallback list plus retries without cooldown equals a retry storm:
|
||
one endpoint's rate limit spreads across the whole cluster.
|
||
- **Circuit breaker:** a closed/open/half-open state machine per endpoint, with exponential backoff.
|
||
- **Fallback:** when a model has no usable endpoints, fall back to a model in its `Fallbacks`
|
||
list (if the key permits it). The response carries an `x-llmgw-served-model` header.
|
||
**Budget fallback:** when a key exceeds its budget, optionally downgrade to a cheaper model
|
||
rather than rejecting the request (§7.9).
|
||
- **Timeouts:** separate limits for connection, first token (TTFT) and total, plus an
|
||
inter-token idle timeout while streaming.
|
||
- **Cancellation:** when the client disconnects, cancel the upstream request too (context
|
||
propagation) so the GPU stops working. This applies to queued requests as well.
|
||
- **Draining:** a `Draining` replica receives no new requests; active ones run to completion or
|
||
time out.
|
||
- **Control plane loss:** the router keeps serving from its last known endpoint list (§14).
|
||
|
||
### 7.8 Streaming proxy details
|
||
|
||
- `http.Transport` with a high `MaxIdleConnsPerHost` and HTTP/1.1 keep-alive (engines rarely
|
||
speak h2c).
|
||
- Read lines with `bufio.Reader`, pass `data:` lines through, and call `http.Flusher.Flush()`
|
||
after every event.
|
||
- If the client did not send `stream_options.include_usage`, the router may add it
|
||
(configurable) to capture usage from the final chunk, and then withhold that chunk from a
|
||
client that did not ask for it.
|
||
- Normalize upstream error JSON into OpenAI's format.
|
||
- Response headers: `x-llmgw-replica`, `x-llmgw-pool`, `x-llmgw-served-model`,
|
||
`x-llmgw-queue-ms` (time spent in admission) and `x-llmgw-cache-decision` (§7.5).
|
||
|
||
### 7.9 Identity, quota and usage (tenancy)
|
||
|
||
LiteLLM's production-proven model:
|
||
|
||
```
|
||
Organization ─► Team ─► Project ─► API Key
|
||
```
|
||
Each level carries a budget (in tokens or notional cost units), RPM/TPM limits and a permitted
|
||
model list. A key additionally carries its **priority band** (§7.2.2) and its **default routing
|
||
profile** (§7.3).
|
||
|
||
- API keys look like `sk-llmgw-...`; only the SHA-256 hash is stored. A leaked key is rotated
|
||
without touching upstream credentials.
|
||
- Rate limiting: RPM via `x/time/rate` plus **TPM**. The token limit is reserved at request
|
||
start as `max_tokens + estimated prompt` and reconciled against the real figure when the
|
||
response completes. (The prompt estimate is bytes/4 in v1; it becomes exact once the tier 2
|
||
tokenizer arrives.)
|
||
- **Budgets:** periodic (daily/monthly) spend. On exhaustion: `429`, or a budget fallback.
|
||
Zero-cost models are exempt from budget checks, so a key whose budget is spent can still be
|
||
given a "cheap model" fallback.
|
||
- **Spend writes stay off the request path:** usage records go to a channel and are batched into
|
||
SQLite, with hourly rollups. Nothing writes to the database on the request path.
|
||
- **Replica pinning (debug):** the `x-llmgw-replica` header, scoped to the key. If the pinned
|
||
replica is in cooldown or gone, the request falls through to normal routing.
|
||
- v1 is a single instance with in-memory limiters. With multiple instances this becomes Redis or
|
||
consistent-hash stickiness (v2).
|
||
|
||
#### 7.9.1 How keys are stored
|
||
|
||
A key is a 256-bit random value, shown once at creation and never again:
|
||
|
||
```
|
||
sk-llmgw-<base62(32 random bytes)>
|
||
```
|
||
|
||
Only `sha256(key)` is stored, indexed. Authentication hashes what the client
|
||
presented and looks it up. Nothing can reconstruct the key from the database,
|
||
so a database leak does not hand over working credentials.
|
||
|
||
**SHA-256 rather than bcrypt or argon2, deliberately.** Slow password hashing
|
||
exists because humans pick guessable passwords, and the cost buys time against
|
||
an offline dictionary attack. A 256-bit random value has no dictionary to
|
||
attack: at any hash rate, guessing it is infeasible. What a slow hash would buy
|
||
here is nothing, and what it would cost is tens of milliseconds on every single
|
||
request. Using the fast hash is the correct choice precisely *because* the input
|
||
is high entropy — and that is the assumption to re-examine if the key format
|
||
ever changes.
|
||
|
||
Alongside the hash, a `key_prefix` (`sk-llmgw-4f2a…`) is stored in clear so the
|
||
UI and audit log can identify a key without revealing it.
|
||
|
||
```sql
|
||
CREATE TABLE api_keys (
|
||
id TEXT PRIMARY KEY,
|
||
key_hash BLOB NOT NULL UNIQUE, -- sha256 of the key; the only copy
|
||
key_prefix TEXT NOT NULL, -- for display, never for auth
|
||
name TEXT NOT NULL,
|
||
project_id TEXT NOT NULL REFERENCES projects(id),
|
||
band TEXT, -- §7.2.2; NULL inherits the project's
|
||
routing_profile TEXT, -- §7.3
|
||
created_at INTEGER NOT NULL,
|
||
expires_at INTEGER, -- NULL never expires
|
||
revoked_at INTEGER, -- soft delete, so usage rows keep a valid FK
|
||
last_used_at INTEGER -- written lazily; see below
|
||
);
|
||
CREATE INDEX api_keys_by_project ON api_keys(project_id);
|
||
```
|
||
|
||
Revocation is a soft delete. Usage rows reference the key, and an audit trail
|
||
that loses its subject when the key is deleted is not an audit trail.
|
||
|
||
`last_used_at` is written at most once a minute per key, from the same batch
|
||
writer as usage. Writing it per request would put a database write back on the
|
||
request path for a field nobody reads in real time.
|
||
|
||
#### 7.9.2 The tenancy tree and how limits resolve
|
||
|
||
```sql
|
||
CREATE TABLE organizations (id TEXT PRIMARY KEY, name TEXT NOT NULL);
|
||
CREATE TABLE teams (id TEXT PRIMARY KEY, org_id TEXT NOT NULL REFERENCES organizations(id), name TEXT NOT NULL);
|
||
CREATE TABLE projects (id TEXT PRIMARY KEY, team_id TEXT NOT NULL REFERENCES teams(id), name TEXT NOT NULL);
|
||
|
||
-- Limits hang off any level rather than being duplicated onto each one.
|
||
CREATE TABLE limits (
|
||
scope_type TEXT NOT NULL, -- 'org' | 'team' | 'project' | 'key'
|
||
scope_id TEXT NOT NULL,
|
||
rpm INTEGER, -- NULL means no limit at this level
|
||
tpm INTEGER,
|
||
budget_micros INTEGER, -- per period, in micro-units; NULL means no budget
|
||
budget_period TEXT, -- 'day' | 'month'
|
||
allowed_models TEXT, -- JSON array; NULL means all
|
||
PRIMARY KEY (scope_type, scope_id)
|
||
);
|
||
```
|
||
|
||
A request's effective limit is the **most restrictive** value along
|
||
key → project → team → org. A team cannot raise a limit its organization set,
|
||
which is the only composition rule that makes a budget mean anything. Model
|
||
allowlists compose the same way, by intersection: a key cannot add a model its
|
||
team withheld. An **empty** allowlist is not an absent one — absent means every
|
||
model, empty means none.
|
||
|
||
Budgets are counted in integer micro-units rather than a float. A budget is
|
||
accumulated by repeated addition, and repeated addition of floats drifts.
|
||
|
||
Resolving four levels per request would be four lookups on the hot path, so the
|
||
resolved view is computed once and cached per key hash, invalidated whenever
|
||
anything on that key's chain changes. Authentication is then a single map
|
||
lookup, with the database consulted only on a cache miss.
|
||
|
||
#### 7.9.3 How accounting is written
|
||
|
||
The volume decides the design. At 1000 requests/sec, one row per request is
|
||
86 million rows a day, and SQLite has a single writer. Inserting per request
|
||
would serialize every request behind that writer.
|
||
|
||
So the path is:
|
||
|
||
```
|
||
request completes
|
||
→ usage record onto a buffered channel (nanoseconds, never blocks)
|
||
→ batch writer: every 200ms or 500 records, one transaction
|
||
→ hourly rollup job collapses raw rows
|
||
→ raw rows older than the retention window are deleted
|
||
```
|
||
|
||
```sql
|
||
CREATE TABLE usage (
|
||
ts INTEGER NOT NULL, -- unix millis
|
||
key_id TEXT NOT NULL,
|
||
project_id TEXT NOT NULL,
|
||
model TEXT NOT NULL,
|
||
served_model TEXT NOT NULL, -- differs from model when a fallback served it
|
||
pool TEXT,
|
||
status INTEGER NOT NULL,
|
||
prompt_tokens INTEGER NOT NULL,
|
||
completion_tokens INTEGER NOT NULL,
|
||
cached_tokens INTEGER NOT NULL, -- prefix cache hit, from the engine
|
||
ttft_ms INTEGER,
|
||
duration_ms INTEGER NOT NULL,
|
||
queue_ms INTEGER -- time spent in admission (§7.2)
|
||
);
|
||
CREATE INDEX usage_by_key_ts ON usage(key_id, ts);
|
||
|
||
-- What survives retention, and what budgets are enforced against.
|
||
CREATE TABLE usage_rollup (
|
||
hour_ts INTEGER NOT NULL,
|
||
key_id TEXT NOT NULL,
|
||
project_id TEXT NOT NULL,
|
||
model TEXT NOT NULL,
|
||
requests INTEGER NOT NULL,
|
||
prompt_tokens INTEGER NOT NULL,
|
||
completion_tokens INTEGER NOT NULL,
|
||
cached_tokens INTEGER NOT NULL,
|
||
units REAL NOT NULL, -- billable cost, per §7.9.4
|
||
PRIMARY KEY (hour_ts, key_id, model)
|
||
);
|
||
```
|
||
|
||
**The honest tradeoff:** records sitting in the channel when the process is
|
||
killed are lost — at most a few hundred milliseconds of accounting. Making that
|
||
impossible means a durable write before each response, which puts a disk fsync
|
||
on every request. For usage that informs budgets and invoices, losing a fraction
|
||
of a second of records to a hard kill is the better trade. **If this ever has to
|
||
feed external billing, that trade has to be revisited**, and the answer then is
|
||
an append-only journal the batch writer drains, not a synchronous insert.
|
||
|
||
#### 7.9.4 How budgets are enforced without reading the database
|
||
|
||
Budgets are enforced per request, so they cannot involve a query. Instead:
|
||
|
||
- On startup, the current period's spend is loaded per key from `usage_rollup`
|
||
plus the un-rolled tail of `usage`. **A restart must not reset a budget**, which
|
||
it would if the counter lived only in memory.
|
||
- From then on an in-memory counter is incremented as responses complete. The
|
||
database catches up through the batch writer.
|
||
- At the period boundary the counter resets and the new period starts empty.
|
||
|
||
"Units" are deliberately not dollars. Self-hosted models have no per-token
|
||
price, so a unit is whatever the operator configures — GPU-seconds, a notional
|
||
rate per million tokens, or 1.0 for everything, which reduces budgets to token
|
||
quotas. This is also what makes the plan's budget fallback work: a model priced
|
||
at zero stays available to a key whose budget is spent (§7.7).
|
||
|
||
#### 7.9.5 Token counting
|
||
|
||
Two numbers exist and they are not the same:
|
||
|
||
- **The estimate**, computed before dispatch as `bytes/4`, used to reserve TPM
|
||
quota and to score prefix affinity. It is a guess and is labelled as one.
|
||
- **The engine's report**, taken from the response's `usage` block. This is
|
||
authoritative and is what gets recorded.
|
||
|
||
The router asks engines for usage even when the client did not, then drops that
|
||
chunk on the way back (§7.8), so accounting does not depend on client behaviour.
|
||
TPM is reserved at request start as `max_tokens + estimate` and reconciled
|
||
against the real figure on completion — reserving optimistically would let a
|
||
key exceed its limit by a whole response.
|
||
|
||
The estimate stops being a guess only when a real tokenizer arrives, which the
|
||
plan defers to v2 along with tier 2 prefix routing (§7.5, §14).
|
||
|
||
#### 7.9.6 Multiple router instances
|
||
|
||
Everything above assumes one instance. Two would each hold their own rate limit
|
||
counters and each allow the configured limit, so the effective limit doubles.
|
||
|
||
v1 is single-instance and this is a documented constraint rather than a bug. The
|
||
answer when it arrives is either shared counters in Redis, or consistent hashing
|
||
on the key so every request for one key reaches the same instance. The second is
|
||
cheaper and degrades better, but rebalances counters whenever the instance set
|
||
changes.
|
||
|
||
---
|
||
|
||
## 8. Observability
|
||
|
||
Prometheus metrics (examples):
|
||
```
|
||
llmgw_requests_total{model,pool,status}
|
||
llmgw_request_duration_seconds{model,pool}
|
||
llmgw_ttft_seconds{model,pool}
|
||
llmgw_tokens_total{model,direction="prompt|completion"}
|
||
llmgw_endpoint_inflight{replica}
|
||
llmgw_endpoint_score{replica}
|
||
llmgw_replicas{model,state}
|
||
llmgw_gpu_free{node,gpu_class,fabric}
|
||
llmgw_schedule_failures_total{model,reason}
|
||
llmgw_agent_connected{node}
|
||
|
||
# admission / flow control (§7.2)
|
||
llmgw_admission_queue_depth{band,model}
|
||
llmgw_admission_wait_seconds{band,model} # histogram
|
||
llmgw_admission_rejected_total{band,reason="full|ttl|client_gone"}
|
||
llmgw_pool_saturation{pool} # 0..1, the detector's output
|
||
|
||
# routing decisions (§7.5) — prefix routing cannot be tuned without these
|
||
llmgw_cache_decision_total{model,decision="hit|miss_min_load|balance_override|stale|no_tree"}
|
||
llmgw_prefix_match_ratio{model} # histogram
|
||
llmgw_scorer_value{scorer,replica}
|
||
llmgw_endpoint_metrics_stale{replica}
|
||
llmgw_capacity_profile_stale{model,profile}
|
||
```
|
||
- OpenTelemetry traces: router span → upstream span (the traceparent header is forwarded to the engine).
|
||
- Structured logs (slog JSON) with `request_id` everywhere.
|
||
- `llmgw ctl top`: live queue depth and occupancy per pool.
|
||
|
||
---
|
||
|
||
## 9. Configuration Examples
|
||
|
||
### 9.1 server.yaml
|
||
```yaml
|
||
listen: ":8080"
|
||
admin_listen: ":9090" # metrics, admin API, pprof
|
||
agent_listen: ":7443"
|
||
tls:
|
||
ca: /etc/llmgw/ca.pem
|
||
cert: /etc/llmgw/server.pem
|
||
key: /etc/llmgw/server-key.pem
|
||
store:
|
||
path: /var/lib/llmgw/llmgw.db
|
||
|
||
fabrics:
|
||
- name: ib-a
|
||
gid_prefix: "fec0:0000:0000:000a" # optional automatic mapping
|
||
- name: ib-b
|
||
gid_prefix: "fec0:0000:0000:000b"
|
||
|
||
router:
|
||
scrape_interval: 2s
|
||
inject_usage: true
|
||
retries: 2
|
||
cooldown_seconds: 15 # how long a failing endpoint leaves the candidate pool
|
||
endpoint_cache_max_age: 5m # keep serving this long if the control plane goes down
|
||
|
||
# §7.2 admission — the saturation detector
|
||
admission:
|
||
target_waiting_per_replica: 4 # proposed by ctl bench
|
||
kv_ceiling: 0.90
|
||
queue_ttl: 30s
|
||
queue_capacity_per_band: 2000
|
||
bands:
|
||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 }
|
||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 }
|
||
|
||
# §7.3 routing profiles
|
||
default_profile: chat
|
||
profiles:
|
||
chat:
|
||
scorers: { prefix: 3.0, queue: 1.0, kv: 1.0, inflight: 1.0 }
|
||
picker: { type: p2c }
|
||
batch:
|
||
scorers: { queue: 3.0, kv: 1.0, load: 1.0, prefix: 0.5 }
|
||
picker: { type: max-score }
|
||
long-context:
|
||
scorers: { kv: 3.0, prefix: 2.0, queue: 1.0 }
|
||
picker: { type: softmax, temperature: 0.2 }
|
||
|
||
# §7.5 prefix awareness
|
||
prefix:
|
||
session_affinity: { enabled: true, header: x-session-id, load_factor: 1.25 } # tier 0
|
||
approx_tree: # tier 1
|
||
enabled: false # v1.1
|
||
cache_threshold: 0.30
|
||
balance_abs_threshold: 16
|
||
balance_rel_threshold: 1.5
|
||
max_tree_size: 2000000
|
||
eviction_interval: 60s
|
||
kv_events: { enabled: false } # tier 2 — needs a tokenizer, v2
|
||
|
||
models:
|
||
- name: llama-3.3-70b
|
||
source: /models/Llama-3.3-70B-Instruct
|
||
scaling: { min: 2, max: 6 }
|
||
fallbacks: [qwen2.5-32b]
|
||
routing_profile: chat
|
||
slo: { ttft_p99: 2s, tpot_p99: 50ms } # ctl bench measures capacity against this SLO
|
||
profiles:
|
||
- name: h100-tp4-fp8
|
||
gpu_class: h100-80g
|
||
engine: vllm
|
||
tp: 4
|
||
nodes: 1
|
||
priority: 100
|
||
weight: 100
|
||
args: ["--quantization", "fp8", "--max-model-len", "32768"]
|
||
- name: a100-tp8
|
||
gpu_class: a100-80g
|
||
engine: vllm
|
||
tp: 8
|
||
nodes: 1
|
||
priority: 50
|
||
weight: 45
|
||
|
||
- name: deepseek-large
|
||
source: /models/DeepSeek-V3
|
||
scaling: { min: 1, max: 2 }
|
||
profiles:
|
||
- name: h100-2node
|
||
gpu_class: h100-80g
|
||
engine: sglang
|
||
tp: 16
|
||
nodes: 2 # → must live on a single fabric
|
||
gpus_per_node: 8
|
||
priority: 100
|
||
weight: 100
|
||
startup_timeout: 40m
|
||
```
|
||
|
||
### 9.2 agent.yaml
|
||
```yaml
|
||
node_id: gpu-a-01
|
||
server: "llmgw-server.internal:7443"
|
||
mgmt_interface: eth0
|
||
tls: { ca: ..., cert: ..., key: ... }
|
||
labels: { rack: r12 }
|
||
fabrics:
|
||
- name: ib-a
|
||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
|
||
port_range: [20000, 20999] # engine HTTP and dist-init ports
|
||
engines:
|
||
vllm: { bin: /opt/venvs/vllm/bin/vllm, launch_mode: ray }
|
||
sglang: { python: /opt/venvs/sglang/bin/python }
|
||
work_dir: /var/lib/llmgw-agent
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Benchmarking and Weighting
|
||
|
||
`llmgw ctl bench <model> --profile <p>`:
|
||
- Drives a target replica at increasing concurrency with fixed prompt and output lengths.
|
||
- Measures TTFT p50/p95, TPOT, throughput (tok/s) and the maximum concurrency that still meets the SLO.
|
||
- The output is the `CapacityProfile` from §6.2.1: `MaxConcurrencyAtSLO`, throughput, TTFT/TPOT p99.
|
||
- `--apply` updates three things at once: the profile's `weight`, `target_waiting_per_replica`,
|
||
and the autoscaling target.
|
||
- This is the only sound way to compare GPU classes; never use an estimated weight.
|
||
- `--sweep-prefix` mode measures behaviour at 0/50/90% prefix cache hit rates, so routing
|
||
profile weights (§7.3) can be tuned from measurements instead of guesses.
|
||
- A profile goes stale when the engine version changes, so every `ctl bench` result is tagged
|
||
with the engine version.
|
||
|
||
---
|
||
|
||
## 11. Roadmap
|
||
|
||
### Phase 0 — Skeleton (1 week)
|
||
- [x] `go mod init`, cobra commands, koanf config, slog, Makefile, golangci-lint
|
||
- [x] `sim-engine`: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format `/metrics`)
|
||
- [x] CI: `go test ./...`, race detector
|
||
|
||
### Phase 1 — Static router (2 weeks)
|
||
- [x] OpenAI handlers, streaming proxy, cancellation propagation
|
||
- [x] Static endpoint list from configuration
|
||
- [x] **The `Filter`/`Scorer`/`Picker` pipeline (§7.3)** — a plugin pipeline from the start, not one "strategy"
|
||
- [x] `queue`, `kv`, `inflight` scorers; `max-score` and `p2c` pickers
|
||
- [x] Session affinity (§7.5 tier 0) — cheap, an early win
|
||
- [x] Metrics scraping (the §7.6 contract), staleness detection, retry, cooldown, breaker
|
||
- [x] Prometheus metrics plus decision metrics (§8) — `llmgw_requests_total`,
|
||
`llmgw_ttft_seconds`, `llmgw_scorer_value`, `llmgw_candidates_considered`,
|
||
`llmgw_candidates_filtered_total`, `llmgw_tokens_total`
|
||
- **Exit criterion:** 1000 concurrent streams behind 3 sim-engines; upstream cancellation on
|
||
client disconnect verified; router overhead p99 < 2 ms.
|
||
|
||
### Phase 1.5 — Admission control (1 week) ⭐
|
||
- [x] Saturation detector (§7.2.1)
|
||
- [x] Priority bands, round-robin tenant fairness within a band, queue capacity and TTL
|
||
- [x] `Admission` interface and lease, cancellation while queued
|
||
- [x] `llmgw_admission_*` metrics
|
||
- **Exit criterion:** while 500 concurrent batch requests flow through the bronze band, the TTFT
|
||
p99 of interactive requests in the platinum band stays under 1.5× its idle value. **Do not
|
||
move on before writing this test — it is the entire reason phase 1.5 exists.**
|
||
|
||
### Phase 2 — Agent: discovery and single-node engines (2 weeks)
|
||
- [x] GPU discovery (nvidia-smi path), IB discovery from sysfs
|
||
- [x] Engine interface, vLLM and SGLang command and environment construction
|
||
- [x] llama.cpp adapter, and with it CPU profiles: a profile may omit its GPU class,
|
||
and only an engine that can serve without a card may do so. This is what
|
||
makes the whole path testable on machines that have no GPUs — verified
|
||
end to end on four VMs serving Qwen2.5-0.5B on CPU at ~50 tok/s, twelve
|
||
concurrent requests balancing 4/4/4 across three replicas.
|
||
- [x] Supervisor (process groups, log buffer, state machine, orphan adoption)
|
||
- [x] NVML backend behind the `nvml` build tag, with NVLink topology. It compiles
|
||
in CI but has not run against a driver: **verify on a GPU node before
|
||
relying on the NVLink domains it reports.**
|
||
- [x] connect-go stream, register/heartbeat, shard status
|
||
- [x] mTLS: a private CA (`llmgw ctl cert`), certificates on both ends, and the
|
||
control plane refusing any agent whose claimed node id does not match its
|
||
certificate. Plaintext HTTP/2 remains the default and still must stay on a
|
||
trusted network; `agent_tls`/`tls` in the configuration turns it on.
|
||
- **Exit criterion (met, via configuration rather than `ctl`):** a model starts
|
||
and stops on a node. Demonstrated with real binaries: `llmgw server` and
|
||
`llmgw agent` as separate processes, the agent registering, a replica being
|
||
placed, an engine launched through the vLLM adapter, traffic served, and the
|
||
node's loss detected when the agent is killed. `ctl` itself is phase 3's
|
||
remaining item.
|
||
|
||
### Phase 3 — Control plane (2–3 weeks)
|
||
- [x] `Register` carries the agent's running shards, so a control plane restart
|
||
rebuilds replica state from the fleet instead of orphaning live engines (§6.5)
|
||
- [x] Orphan adoption: an agent takes back engines a previous agent left running,
|
||
and kills whatever it cannot identify or the catalog will not claim
|
||
- [x] Inventory: nodes, hardware, fabric attachments, allocations
|
||
- [x] Placement: profile priority, bin packing, NVLink-aware GPU selection,
|
||
`explain-placement` rejection reasons, property-based invariant tests
|
||
- [x] Catalog: models, profiles, alias and fallback resolution, validation
|
||
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
|
||
- [x] `EndpointSource` wired from the reconciler to the router
|
||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||
- [x] `ctl logs`: a request/response pair on the agent stream (LogsRequest,
|
||
LogsReply) reads a node's agent log and any shard's engine output on
|
||
demand, from the CLI and from the node's page under /admin.
|
||
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
|
||
moves its replicas elsewhere while the router keeps serving. Measured in
|
||
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
|
||
rebuilt on a survivor and ~370 requests complete with zero failures.
|
||
- **Since verified against real agents and engines**, which turned up what the
|
||
simulated fleet could not: replica ids were reused after a control plane
|
||
restart, so a restarted control plane collided with engines still running from
|
||
its previous life and every placement landed on a port an orphan was holding.
|
||
Ids now carry random bytes, adopted state wins over merely-placed state, and a
|
||
replica that never comes up is failed rather than left in `starting` forever.
|
||
|
||
### Phase 4 — Multi-node and fabrics (2 weeks)
|
||
- [x] Fabric identity from the agent's explicit labels
|
||
- [x] Multi-node placement (the fabric constraint), port allocation, rank ordering
|
||
- [x] NCCL environment generation, HCA selection
|
||
- [x] Replica-level failure handling (one dead shard kills the replica)
|
||
|
||
The launch path is now built too:
|
||
|
||
- [x] **Only the leader is health-checked.** `Engine.ServesHTTP(spec)` says which
|
||
ranks answer the API; a worker is judged by staying alive, and the
|
||
replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined. Health-checking a worker's port used to time out
|
||
and fail a replica that was coming up perfectly well.
|
||
- [x] **A shard can be several processes.** `Engine.Commands(spec)` returns them
|
||
in start order: every command but the last is a sidecar kept alive for the
|
||
shard's lifetime and torn down with it, and the last one's exit is the
|
||
shard's. vLLM under Ray is the reason — the leader runs a Ray head and
|
||
then the server, a worker runs only the join and no vLLM process at all.
|
||
- [x] **The leader starts first.** Both engines put the rendezvous point on rank
|
||
0, so a worker started first has nothing to connect to. An earlier version
|
||
had this backwards; vLLM's server blocks until the Ray cluster has the
|
||
GPUs it needs, so the leader can start immediately and waits on its own.
|
||
- [x] **Rendezvous is waited for, not assumed.** A command can require that
|
||
`LeaderAddr:LeaderPort` accepts a connection before the next one starts,
|
||
bounded by `rendezvous_timeout` so a head that never appears fails the
|
||
shard instead of hanging it.
|
||
|
||
- [x] **Verified against a simulated cluster**, which is not the same as real
|
||
hardware and covers more than stub processes did. Four nodes across two
|
||
fabrics, eight GPUs each; `sim` is a real engine adapter whose leader
|
||
binds a rendezvous port and does not serve until every worker has joined,
|
||
and whose workers serve nothing at all.
|
||
|
||
What that proves: placement honours the fabric constraint, the leader
|
||
starts first and workers wait for its port, only rank 0 is health-checked,
|
||
a 2-node replica comes up on each fabric, and losing half of one fails the
|
||
replica and rebuilds it **within a single fabric** rather than spanning
|
||
two.
|
||
|
||
What it does not prove: that a given vLLM version wants `ray` or `mp`,
|
||
that NCCL picks the right HCA, or that the generated environment makes a
|
||
real collective run. Those need hardware. The GPUs here are an inventory,
|
||
the fabric is a sysfs-shaped directory, and the engine speaks no NCCL.
|
||
- [x] `ctl engine-check`, which asks an engine whether it still accepts the
|
||
flags llmgw builds. No GPU needed — a CLI prints its help anywhere, and
|
||
vLLM prints its own once told to assume a device.
|
||
- [x] **Checked against real engines.** vLLM 0.29.0: all five `vllm serve` flags
|
||
accepted (`--host`, `--port`, `--tensor-parallel-size`,
|
||
`--served-model-name`, `--distributed-executor-backend`). llama.cpp
|
||
b10991: all seven accepted.
|
||
- [ ] **The Ray sidecar is still unchecked.** vLLM 0.29 does not install Ray, so
|
||
`ray start --head --block --port --num-gpus` has never been run against
|
||
Ray's own help. `launch_mode: mp` avoids it entirely and is worth
|
||
preferring until someone checks.
|
||
- [ ] **SGLang is unchecked**, for the same reason as Ray: not installed
|
||
anywhere yet.
|
||
|
||
- **Exit criterion (met on the simulated cluster):** a 2-node replica on each
|
||
fabric; killing a node rebuilds the replica within one fabric and never
|
||
spreads it across two. Measured: node02 lost at t+60s, the ib-a replica failed
|
||
and was reaped, a replacement was ready at t+80s on ib-b — the fabric that
|
||
could still host it — while node01 sat idle rather than being paired across
|
||
fabrics with a node from ib-b.
|
||
|
||
### Phase 5 — Production features (3 weeks)
|
||
- [x] **SQLite store and migrations**, behind an interface, with per-dialect
|
||
migration directories so Postgres is a driver swap rather than a rewrite.
|
||
It arrived here rather than in phase 3 because everything that genuinely
|
||
needs persisting is a phase 5 feature; writing it earlier would have meant
|
||
designing a schema against guesses (§6.5).
|
||
- [x] Tenancy: org/team/project/key hierarchy, key storage and authentication,
|
||
limit resolution (most restrictive along the chain), `llmgw keys`
|
||
- [x] Budgets, RPM/TPM enforcement and the usage batch writer (§7.9.3, §7.9.4).
|
||
Rate limits are a sliding window, not a fixed one: a fixed window lets a
|
||
caller send its whole allowance at the end of one window and again at the
|
||
start of the next, so the observed rate doubles exactly when the system is
|
||
least able to take it.
|
||
- [x] Accounts: signup, operator approval, sessions, password changes, and a
|
||
profile page where a person issues and revokes their own keys. Approval stays on the CLI —
|
||
having a shell on the control plane is what makes someone an
|
||
administrator, and there are no roles to get wrong.
|
||
- [x] Onboarding: one-time claim links (the key is sealed with the link's own
|
||
token, which is never stored), `GET /me` for a user's own limits and
|
||
spend, and `docs/onboarding.md`. Sending the key itself was the weak link
|
||
— a key pasted into a chat window stays there forever.
|
||
- [x] Model allowlists and key→band mapping. With auth on, the band comes from
|
||
the key and the header is ignored — it was trusted only while there was
|
||
no identity to consult, which let any client promote itself.
|
||
- [x] Budget fallback
|
||
- [x] Routing profiles from configuration plus the `x-llmgw-profile` header
|
||
- [x] Approximate radix tree (§7.5 tier 1), with its thresholds and the
|
||
`llmgw_prefix_decisions_total` breakdown. Measured against sim-engines
|
||
whose caches were too small for the working set: 22.5% → 81.7% cache
|
||
hits. It does nothing when the working set fits in one replica's cache,
|
||
which is worth knowing rather than assuming.
|
||
- [x] Fallback models
|
||
- [x] Pool spillover: a saturated pool passes requests to the next one that
|
||
is not, by admission's own measure of saturation
|
||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||
- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
|
||
and the autoscaler and profile choice use what `ctl bench --save` measured
|
||
- [x] OTel tracing: an inference span with admission, schedule and each
|
||
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
|
||
set, and the traceparent carried on to the engine
|
||
- [x] systemd units and `docs/onboarding.md`
|
||
- [x] Release binaries: static linux/amd64, linux/arm64 and darwin builds
|
||
attached to each tag, so installing on a node does not need a Go toolchain
|
||
- [x] Reference documentation beyond the plan itself: `docs/`, with
|
||
`docs/reference.md` generated from the configuration structs
|
||
|
||
### Phase 6 — v2 ideas
|
||
- [ ] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
|
||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||
- [x] Admission displacement: a total queue bound, and the newest request of
|
||
the lowest band turned away (503, Retry-After) for a higher one
|
||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||
model's max, and taken back after ten quiet minutes
|
||
- [x] Priority-based preemption: capacity classes (critical, standard,
|
||
opportunistic), preemption tried on a copy of the inventory first
|
||
- [x] Capacity quotas per organization, team and project: guarantees kept,
|
||
borrowing taken back, limits enforced
|
||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
|
||
temperature and RAPL power, board inlet air; published and shown
|
||
- [ ] Rack power, temperature and cooling as placement signals: planned in
|
||
[docs/facility-aware-placement.md](docs/facility-aware-placement.md),
|
||
deferred — it needs each site's rack, PDU and power data first
|
||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||
- [ ] Control plane HA (Raft)
|
||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||
checked against their digest, and reclaiming what nothing uses
|
||
- [ ] Prefill/decode disaggregation (§12)
|
||
- [x] LoRA adapters on llama.cpp: every replica loads a model's adapters and
|
||
applies one per request, so any replica serves any adapter
|
||
- [ ] LoRA affinity for engines that load adapters on demand (vLLM)
|
||
- [x] External provider support: a catalog model served by an OpenAI-compatible
|
||
API elsewhere, usable as a fallback, under the same allowlists and budgets
|
||
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
||
placement failures, usage) and a public landing page
|
||
- [x] `/chat`: an in-browser playground for signed-in people, dispatched through
|
||
the ordinary request path so every limit still applies and the browser
|
||
never holds a key
|
||
- [x] `/admin/users`: approve, suspend and reinstate accounts from the browser;
|
||
promoting, resetting passwords and issuing keys stay on the CLI
|
||
- [x] The dashboard reachable over the web at `/admin`, behind a session and an
|
||
admin flag, without exposing the listener that carries pprof
|
||
- [ ] An admin UI that can *change* things, which needs roles this deliberately
|
||
does not have
|
||
|
||
---
|
||
|
||
## 12. Prefill/Decode Disaggregation (v2 notes)
|
||
- Prefill and decode replicas must live on the **same fabric** (KV transfer goes over RDMA).
|
||
- A two-stage flow in the router: send to a prefill replica → KV handle → decode replica.
|
||
- Engine support varies sharply by version (vLLM KV connectors, SGLang PD mode, NVIDIA Dynamo/NIXL). Wrapping the engine's own built-in PD router as a single "endpoint" is less risky than implementing this ourselves.
|
||
- What llm-d and Dynamo show: PD's benefit lies mainly in **long context**. For short-prompt chat
|
||
traffic the KV transfer cost eats the gain. Measure your workload's ISL/OSL distribution with
|
||
the benchmark in §10 first, and enable PD only for long-context models.
|
||
- The scheduler gains a `Role: prefill|decode` field and a "paired placement on the same fabric" constraint.
|
||
|
||
---
|
||
|
||
## 13. Test Strategy
|
||
- **Unit:** the scheduler (table-driven: heterogeneous inventory plus expected placement or failure reason), balancer scores, SSE parsing, metric parsing (fixtures taken from real vLLM/SGLang `/metrics` output).
|
||
- **Property-based:** the invariants "a multi-node replica lives on one fabric" and "one GPU class per replica" always hold in scheduler output (`testing/quick` or `rapid`).
|
||
- **Integration:** server plus N fake agents (fake GPU/IB inventory) plus sim-engines, in one process.
|
||
- **Chaos:** kill agents, break the network (toxiproxy), crash engines, slow engines, and kill
|
||
the control plane to verify the router keeps serving from its cache.
|
||
- **Priority isolation:** phase 1.5's exit criterion as a permanent regression test — high-priority
|
||
TTFT is preserved under low-priority load. This test must stay in CI.
|
||
- **Routing quality:** give sim-engine a realistic prefix cache model (block-level LRU) and
|
||
measure the hit rate against synthetic multi-turn traffic. The comparison between round robin,
|
||
session affinity and the approximate tree must be numeric.
|
||
- **Load:** `k6` or our own Go load generator; measure the router's own overhead (target: p99 under 2 ms of added latency).
|
||
- **Real hardware:** at the end of every phase, a smoke test on at least one node with IB and one without.
|
||
|
||
---
|
||
|
||
## 14. Risks and Open Questions
|
||
| Topic | Note |
|
||
|---|---|
|
||
| Engine version differences | CLI flags and metric names change often → make engine adapters version-aware and overridable from configuration. |
|
||
| vLLM multi-node launch | Ray or built-in multiprocessing? Must be tested on the version actually deployed. |
|
||
| Fabric detection | Can a unique subnet prefix be set on each OpenSM? If not, manual labelling is mandatory. |
|
||
| Multiple fabrics per node | Does this occur? If so, HCA-to-GPU NUMA matching matters. |
|
||
| Model storage | Is there a shared filesystem, and what are the load times? Startup timeouts follow from this. |
|
||
| Mixed driver/CUDA versions | Should engine virtualenvs differ per GPU class? (The `engines` config must be overridable per profile.) |
|
||
| Single control plane | A SPOF in v1. The router caches the endpoint list (§6.4) and keeps working through a brief control plane outage. |
|
||
| Security | Engine ports must be reachable only by the router (firewall or bind address). |
|
||
| **Tokenizer consistency** | The precise KV index (§7.5 tier 2) forces the router to use **the same tokenizer** as the engine. In Go that means cgo or a separate tokenizer service, which is why it is deferred to v2; v1's prefix approach needs no tokenizer. The same limitation makes the TPM prompt estimate approximate in v1 (bytes/4). |
|
||
| **Prefix affinity backfiring** | Without the load-imbalance guard (the §7.5 tier 1 thresholds), prefix routing raises the cache hit rate while drowning a single replica and wrecking p99. Prefix affinity **must not** be enabled before those thresholds are. |
|
||
| **Tuning the admission thresholds** | Too low a `target_waiting_per_replica` starves the GPU; too high brings queueing delay back. This single number governs the system's behaviour, so measure it with `ctl bench` rather than guessing. |
|
||
| KV event loss | ZMQ is fire-and-forget; a lost event makes the router silently misroute. Sequence-gap detection and falling back to tier 1 are mandatory (§7.5). |
|
||
| Metric staleness | At 1000 requests/sec with a 2s scrape interval, decisions rest entirely on stale data. That is why the `inflight` scorer is not optional. |
|
||
| **Usage lost on a hard kill** | Records buffered for the batch writer are lost if the process is killed — at most a few hundred milliseconds of accounting. Acceptable for budgets; **not acceptable if this ever feeds external billing**, and the answer then is an append-only journal rather than a synchronous insert (§7.9.3). |
|
||
| **Rate limits with several routers** | Counters are per instance, so N instances allow N times the configured limit. v1 is single-instance by design; Redis or consistent-hash stickiness is the fix (§7.9.6). |
|
||
| **Control plane restart** | Until `Register` carries running shards, restarting the control plane orphans live engines: they keep holding GPUs the inventory believes are free, and the next placement fails to allocate with an error pointing at the wrong process (§6.5). |
|
||
|
||
---
|
||
|
||
## 15. First Steps
|
||
1. `mkdir llmgw && cd llmgw && go mod init <module-path>/llmgw`
|
||
2. `cmd/llmgw/main.go`: an empty cobra skeleton with the `server`, `agent`, `ctl` and `sim-engine` subcommands.
|
||
3. `internal/sim`: the fake engine — make this the first real code, because everything else is tested against it.
|
||
4. `internal/router/proxy`: streaming proxy plus a cancellation test.
|
||
5. `internal/router/scheduler`: the `Filter`/`Scorer`/`Picker` interfaces plus the `queue`, `kv`
|
||
and `inflight` scorers. **Do not write a single "least-load" function** — build the pipeline
|
||
from the start; coming back to it later is expensive.
|
||
6. `internal/router/admission`: saturation detector and band queues. Phase 1.5.
|
||
7. Write the end-to-end tests for the phase 1 and phase 1.5 exit criteria, then move to phase 2.
|
||
|
||
> Use `internal/router/scheduler/` rather than `internal/router/balancer/` (the tree in §4
|
||
> reflects this): "balancer" understates the job — this component does more than balance load.
|