Ilker ManapandClaude Opus 5.5
41601bc5ef
CI / test (push) Failing after 6m22s
The precise KV index and vLLM's LoRA affinity were built in v0.11.2 and never run against a real vLLM; facility-aware placement in v0.11.2 less temperature and cooling: marked [~], a mark now explained at the top of the roadmap. Control-plane HA is done, by leader election on PostgreSQL rather than Raft, and the admin UI changes things since staff roles (v0.11.2) and the settings page (v0.14.1). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
1594 lines
83 KiB
Markdown
1594 lines
83 KiB
Markdown
# llmgw — A Single-Binary LLM Gateway for Multi-Fabric InfiniBand and Heterogeneous GPUs
|
||
|
||
> Status: Design / Plan v0.2
|
||
> What changed in v0.2: existing LLM gateway and router projects were surveyed (§2.5) and
|
||
> their critical designs folded in — above all **separating admission from scheduling** (§7.2),
|
||
> the plugin-based **filter → score → pick** pipeline (§7.3), and the **three-tier model of
|
||
> prefix/KV cache awareness** (§7.5).
|
||
> Goal: one Go binary (running under different roles) that serves the LLMs on our own GPU
|
||
> servers behind a single OpenAI-compatible endpoint. Behind it sit several independent
|
||
> InfiniBand fabrics, nodes without IB, and different GPU types.
|
||
|
||
---
|
||
|
||
## 1. Scope
|
||
|
||
### 1.1 In scope
|
||
- OpenAI-compatible API (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`, `/v1/models`), including SSE streaming.
|
||
- An **agent** on every GPU node: GPU and IB discovery; starting, supervising and stopping inference engine processes (vLLM / SGLang).
|
||
- **Control plane**: node inventory, model catalog, replica placement, state tracking.
|
||
- **Router**, in two layers: **admission** (priority bands, tenant fairness, queueing under
|
||
saturation) and **scheduling** (plugin-based filter/score/pick with queue, KV-cache and
|
||
prefix awareness), plus fallback, retry, cooldown and circuit breaking.
|
||
- Fabric awareness: a multi-node replica **never** spans two fabrics; a TP group **never** mixes GPU types.
|
||
- API keys, rate limits, token accounting and budgets, Prometheus metrics. Persistence for these is
|
||
the only thing the store exists for (§6.5, §7.9).
|
||
|
||
### 1.2 Out of scope (at least for v1)
|
||
- Inference itself. Go only orchestrates and proxies; vLLM/SGLang run the model.
|
||
- A Kubernetes operator (bare metal / systemd is the target; K8s can come later).
|
||
- Prefill/decode disaggregation (a v2 goal, see §12).
|
||
- External providers (OpenAI, Anthropic) — the architecture stays open to them, but not in v1.
|
||
|
||
---
|
||
|
||
## 2. Terminology
|
||
|
||
| Term | Meaning |
|
||
|---|---|
|
||
| **Node** | A physical server running the agent. |
|
||
| **Fabric / Island** | An independent IB fabric. Nodes without IB belong to the `none` island and each counts as its own single-node island. |
|
||
| **GPU Class** | A group of GPUs with the same model and VRAM (e.g. `h100-80g`, `a100-40g`, `l40s-45g`). |
|
||
| **Pool** | A (fabric, gpu class) pair. The unit of placement. |
|
||
| **Model** | A logical model in the catalog (e.g. `llama-3.3-70b`). This is the name users see. |
|
||
| **Profile** | How a model runs on a particular GPU class (TP, PP, quantization, max length, engine arguments). |
|
||
| **Replica** | One running copy of a model. May span 1..N nodes. This is the unit the router sees. |
|
||
| **Shard** | A replica's part on a single node (one engine process). |
|
||
| **Routing Profile** | The scorer set and weights the router uses for a class of request (chat / batch / long-context). Not to be confused with the model **Profile** above. |
|
||
| **Band** | A priority class (platinum/gold/silver/bronze). The unit of admission decisions. |
|
||
| **Block** | The unit of KV cache accounting; a fixed number of tokens (16 in vLLM, 64 by default in the GIE indexer). |
|
||
| **Unit** | The billing unit budgets are counted in. Self-hosted models have no per-token price, so an operator defines what a unit is: GPU-seconds, a notional rate per million tokens, or 1.0 everywhere, which turns budgets into token quotas (§7.9.4). |
|
||
|
||
---
|
||
|
||
## 2.5 Reference Implementations and What We Took
|
||
|
||
Rather than design from scratch, we inherited the problems already solved by mature projects
|
||
in this space.
|
||
|
||
| Project | What it is | What llmgw takes from it |
|
||
|---|---|---|
|
||
| **Gateway API Inference Extension (EPP)** | The K8s standard; an Endpoint Picker running over Envoy `ext_proc` | Admission/scheduling separation, the plugin-based `filter → score → pick` pipeline, the **model server metric contract** (§7.6) |
|
||
| **llm-d** | "Well-lit paths" built on GIE | KV-event-based **precise** prefix index, priority bands with tenant fairness, the saturation detector |
|
||
| **AIBrix** (a vLLM project) | An Envoy ext_proc gateway plus a heterogeneous GPU optimizer | The **heterogeneous GPU SLO/cost model** (§6.2.1), queueing at the gateway (`slo-least-load-pulling`), disabling prefix affinity under load imbalance |
|
||
| **NVIDIA Dynamo** | Radix-tree KV router plus SLA planner | The shape of the cost function (§7.4) and `router_temperature` softmax sampling against herding |
|
||
| **SGLang router** | Cache-aware load balancing over an approximate radix tree | v1's prefix answer: approximate tree plus threshold logic (§7.5 tier 1) |
|
||
| **vLLM production-stack / vllm-project/router** | vLLM's own routers | Session affinity (measured 96% vs 93% cache hit), consistent hashing |
|
||
| **LiteLLM** | The most widely deployed multi-provider proxy | Virtual key → team/project → budget hierarchy, cooldowns, keeping spend writes off the request path, budget-triggered fallback |
|
||
| **Envoy AI Gateway** | An Envoy-based AI gateway (v1.0, June 2026) | Token-based (not merely request-based) rate limiting |
|
||
|
||
What we deliberately **did not** take: a Kubernetes dependency (bare metal is the target), the
|
||
Envoy `ext_proc` architecture (single binary is the goal — we write the router directly in Go),
|
||
semantic routing / LLM-based model selection (out of scope), and external provider management
|
||
(not in v1).
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
```
|
||
┌───────────────────────────────────────────┐
|
||
Clients ────────► │ llmgw server │
|
||
(OpenAI API) │ ┌────────────┐ ┌──────────────────────┐ │
|
||
│ │ Router │ │ Control Plane │ │
|
||
│ │ (data) │◄─┤ - Inventory │ │
|
||
│ │ │ │ - Catalog │ │
|
||
│ └─────┬──────┘ │ - Scheduler │ │
|
||
│ │ │ - Store (phase 5) │ │
|
||
│ │ └─────────▲────────────┘ │
|
||
└────────┼───────────────────┼──────────────┘
|
||
HTTP (eth) │ │ gRPC stream (mTLS, agent dials out)
|
||
┌────────────────────┼───────────────────┼─────────────────────┐
|
||
▼ ▼ │ ▼
|
||
┌───────────┐ ┌───────────┐ ┌───────────┐ ┌───────────┐
|
||
│ llmgw │ IB-A │ llmgw │ │ llmgw │ IB-B │ llmgw │
|
||
│ agent │◄──────►│ agent │ │ agent │◄───────►│ agent │ ...
|
||
│ vLLM rank0│ NCCL │ vLLM rank1│ │ SGLang │ │ (no IB) │
|
||
└───────────┘ └───────────┘ └───────────┘ └───────────┘
|
||
Fabric A island Fabric B island none island
|
||
```
|
||
|
||
### 3.1 One binary, several roles
|
||
|
||
```
|
||
llmgw server --config server.yaml # router + control plane (same process in v1)
|
||
llmgw agent --config agent.yaml # on every GPU node
|
||
llmgw ctl ... # CLI: list nodes, deploy models, drain, ...
|
||
llmgw sim-engine # fake OpenAI-compatible engine for testing
|
||
```
|
||
|
||
`router` and `control` must eventually be able to run as separate processes
|
||
(`llmgw server --roles=router`), so the dependency between them goes through an interface
|
||
(see §6.4).
|
||
|
||
### 3.2 Build note
|
||
`go-nvml` requires cgo. Put the agent's NVML code behind a `//go:build nvml` tag and use a
|
||
fake / `nvidia-smi`-based discovery path in untagged builds. That keeps the server side
|
||
cgo-free and lets development machines without GPUs run the tests.
|
||
|
||
---
|
||
|
||
## 4. Project Layout
|
||
|
||
```
|
||
llmgw/
|
||
├── cmd/llmgw/main.go # cobra root; subcommands
|
||
├── api/
|
||
│ └── proto/llmgw/v1/*.proto # agent <-> control plane
|
||
├── internal/
|
||
│ ├── config/ # YAML + env via koanf
|
||
│ ├── agent/
|
||
│ │ ├── agent.go # main loop, control plane connection
|
||
│ │ ├── discovery/
|
||
│ │ │ ├── gpu_nvml.go # //go:build nvml
|
||
│ │ │ ├── gpu_fallback.go
|
||
│ │ │ ├── ib.go # reads /sys/class/infiniband
|
||
│ │ │ └── topology.go # NVLink, PCIe, NUMA, GPU<->HCA affinity
|
||
│ │ ├── supervisor/ # engine process management (exec, restart, logs)
|
||
│ │ ├── engines/
|
||
│ │ │ ├── engine.go # Engine interface
|
||
│ │ │ ├── vllm.go
|
||
│ │ │ └── sglang.go
|
||
│ │ └── health/ # polls engine /health and /metrics
|
||
│ ├── control/
|
||
│ │ ├── inventory/ # node and GPU state
|
||
│ │ ├── catalog/ # model and profile definitions
|
||
│ │ ├── scheduler/ # placement algorithm
|
||
│ │ ├── reconciler/ # desired vs actual loop
|
||
│ │ └── agentsvc/ # gRPC server
|
||
│ ├── router/ # HTTP handlers, dispatch, metrics
|
||
│ │ ├── proxy/ # streaming proxy to upstream
|
||
│ │ ├── admission/ # saturation detector, band queues (§7.2)
|
||
│ │ ├── scheduler/ # Filter/Scorer/Picker pipeline (§7.3)
|
||
│ │ ├── prefix/ # tier 0 session hash, tier 1 approx tree, tier 2 kv index
|
||
│ │ ├── endpoint/ # live endpoint state, inflight, staleness
|
||
│ │ ├── breaker/ # circuit breaker + cooldown
|
||
│ │ └── scrape/ # reads engine /metrics (the §7.6 contract)
|
||
│ ├── auth/ # API keys, org/team/project, budgets
|
||
│ ├── ratelimit/ # token bucket (requests + tokens)
|
||
│ ├── usage/ # token counting and recording
|
||
│ ├── store/ # SQLite (modernc.org/sqlite), migrations
|
||
│ ├── obs/ # slog, prometheus, otel
|
||
│ └── sim/ # fake engine
|
||
├── deploy/
|
||
│ ├── systemd/
|
||
│ └── examples/
|
||
├── test/e2e/
|
||
└── docs/
|
||
```
|
||
|
||
> The OpenAI handlers live in `internal/router` itself rather than an `openai/`
|
||
> subpackage: they need the dispatch loop, and splitting them apart would have
|
||
> meant an interface whose only implementation is the router.
|
||
|
||
### 4.1 Libraries
|
||
|
||
| Need | Choice |
|
||
|---|---|
|
||
| CLI | `github.com/spf13/cobra` |
|
||
| Config | `github.com/knadh/koanf/v2` |
|
||
| RPC | `connectrpc.com/connect` (gRPC compatible, HTTP/2, easy to debug) |
|
||
| GPU | `github.com/NVIDIA/go-nvml` |
|
||
| DB | `modernc.org/sqlite` (cgo-free) + `pressly/goose` migrations |
|
||
| Metrics | `github.com/prometheus/client_golang`, parsing via `github.com/prometheus/common/expfmt` |
|
||
| Tracing | `go.opentelemetry.io/otel` |
|
||
| Logging | `log/slog` (stdlib) |
|
||
| Rate limiting | `golang.org/x/time/rate` (plus our own bucket for token limits) |
|
||
| Hashing | `github.com/cespare/xxhash/v2` |
|
||
| HA (v2) | `github.com/hashicorp/raft` or etcd |
|
||
| Testing | stdlib `testing`, `github.com/stretchr/testify` (optional) |
|
||
|
||
---
|
||
|
||
## 5. Agent
|
||
|
||
### 5.1 GPU discovery
|
||
For every GPU:
|
||
- index, UUID, name, VRAM, compute capability (for FP8 and similar support), driver/CUDA version
|
||
- NUMA node, PCIe bus id
|
||
- NVLink peers (NVML `GetNvLinkRemotePciInfo`) → which GPUs pair efficiently for TP within a node
|
||
- health: XID errors, ECC, temperature, memory in use (are there non-engine processes?)
|
||
|
||
`GPUClass` is derived automatically: `normalize(name) + "-" + round(vram_gib) + "g"` → `h100-80g`.
|
||
It can be overridden in configuration.
|
||
|
||
The size comes from what the driver reports, in GiB, which does not always match the marketing
|
||
name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is `l40s-45g`. Consistency
|
||
matters more than matching the box here, because the number is what decides whether a model
|
||
fits. The model token is kept in full for the same reason — a 4090 and a 4080 are not
|
||
interchangeable, so the class is `4090-24g`, not `rtx-24g`.
|
||
|
||
### 5.2 IB discovery and fabric identity
|
||
From `/sys/class/infiniband/<hca>/ports/<n>/`:
|
||
- `state` (ACTIVE?), `phys_state`, `rate`, `link_layer` (InfiniBand / Ethernet → RoCE)
|
||
- `gids/0` → subnet prefix
|
||
- `sm_lid`, `lid`
|
||
- the HCA's PCIe address and NUMA node → GPU affinity
|
||
|
||
**How is fabric identity determined?** Automatic detection is not reliable, because OpenSM's
|
||
default subnet prefix (`fe80::`) is the same on every fabric. In order of preference:
|
||
|
||
1. An explicit label in the agent config: `fabrics: [{name: ib-a, hcas: [mlx5_0, mlx5_1]}]` **(recommended)**
|
||
2. If every OpenSM has a unique `subnet_prefix`, automatic mapping from the GID prefix (a `prefix → fabric` table in the server config)
|
||
3. Otherwise the node is marked `fabric=unknown` and **excluded from multi-node placement**.
|
||
|
||
Bonus verification (v1.1): for two nodes claiming the same fabric, the agents verify
|
||
reachability with a short `ibv_rc_pingpong`-style test or `ibping`. A mismatch raises an alert.
|
||
|
||
A node may have HCAs attached to more than one fabric, and the model must allow it
|
||
(`Node.Fabrics []FabricAttachment`). A replica, however, picks exactly one fabric and uses only
|
||
that fabric's HCAs.
|
||
|
||
### 5.3 Engine interface
|
||
|
||
```go
|
||
type Engine interface {
|
||
Name() string // "vllm", "sglang"
|
||
Validate(p Profile, gpus []GPU) error // compute capability, quantization fit, ...
|
||
BuildCommand(spec ShardSpec) (*exec.Cmd, error) // builds arguments and environment
|
||
HealthURL(spec ShardSpec) string
|
||
MetricsURL(spec ShardSpec) string
|
||
ParseMetrics(r io.Reader) (EngineStats, error) // engine-specific metric names
|
||
}
|
||
|
||
type ShardSpec struct {
|
||
ReplicaID string
|
||
Rank int // 0 = leader (the one serving HTTP)
|
||
NNodes int
|
||
LeaderAddr string // dist init address (ethernet IP)
|
||
LeaderPort int
|
||
GPUIndices []int
|
||
HCAs []string // HCAs of the chosen fabric
|
||
HTTPPort int
|
||
Profile Profile
|
||
ModelPath string
|
||
}
|
||
|
||
type EngineStats struct {
|
||
// The mandatory contract from §7.6 — without these the router cannot route.
|
||
Running, Waiting int
|
||
KVCacheUsage float64 // 0..1 (vllm:kv_cache_usage_perc)
|
||
// Optional.
|
||
BlockSize, NumGPUBlocks int
|
||
PrefixHitRate float64
|
||
TTFTp50, TPOTp50 time.Duration // from histograms
|
||
DrainRate float64 // observed completions/sec; for the `load` scorer
|
||
LoRA LoRAState // v2
|
||
ScrapedAt time.Time // for staleness detection — the router uses this
|
||
}
|
||
```
|
||
|
||
Supporting a new engine is defined as: being able to fill in the mandatory three from §7.6.
|
||
|
||
**Multi-node launch:**
|
||
- SGLang supports it natively: `--nnodes N --node-rank R --dist-init-addr host:port --tp T`.
|
||
- vLLM: depending on the version, either Ray (`ray start --head` on the leader, `ray start --address` on the workers) or the newer multi-node multiprocessing flags. **Check the documentation for the vLLM version you deploy**; `vllm.go` should offer both paths behind a `launchMode`.
|
||
|
||
**Environment generation (critical):**
|
||
```
|
||
CUDA_VISIBLE_DEVICES=<GPUIndices>
|
||
NCCL_IB_HCA=<HCAs, comma separated> # the chosen fabric only
|
||
NCCL_SOCKET_IFNAME=<mgmt interface> # bootstrap
|
||
NCCL_IB_DISABLE=1 # single node, or no IB
|
||
NCCL_CROSS_NIC / NCCL_NET_GDR_LEVEL # optional, from the profile
|
||
GLOO_SOCKET_IFNAME=<mgmt interface>
|
||
```
|
||
|
||
### 5.4 Supervisor
|
||
- Start processes in their own process group (`SysProcAttr.Setpgid`) and kill the whole group when stopping (SIGTERM → timeout → SIGKILL).
|
||
- stdout/stderr → line-oriented slog plus a ring buffer (the last N lines, for `llmgw ctl logs`).
|
||
- State machine: `Pending → Starting → Loading → Ready → Draining → Stopped | Failed`.
|
||
- `Ready` only once `/health` returns 200 **and** `/v1/models` lists the model.
|
||
- Model loading takes a long time; `startupTimeout` comes from the profile (e.g. 20 min).
|
||
- On a crash: report to the control plane and let **the control plane** decide about restarting. Restarting a single shard of a multi-node replica accomplishes nothing; the whole replica must be rebuilt.
|
||
- If the agent restarts, try to adopt still-running engine processes (pid file plus cmdline check) and clean up whatever cannot be adopted.
|
||
|
||
### 5.5 Model weights
|
||
- v1: a shared filesystem (NFS/Lustre/Weka) or a path pre-populated on the node. The agent only checks existence and size.
|
||
- v1.1: the agent downloads from HF/S3 into a local cache, reports progress, and enforces a disk quota.
|
||
|
||
### 5.6 Agent ↔ control plane protocol
|
||
The agent dials out (NAT- and firewall-friendly) over a bidirectional stream:
|
||
|
||
```proto
|
||
service AgentService {
|
||
rpc Connect(stream AgentMessage) returns (stream ControlMessage);
|
||
}
|
||
|
||
message AgentMessage {
|
||
oneof msg {
|
||
Register register = 1; // node info, GPUs, fabrics, versions
|
||
Heartbeat heartbeat = 2; // every 5s: GPU health, shard states
|
||
ShardStatus shard = 3; // sent on every state change
|
||
EngineStats stats = 4; // every 2s (the router may also scrape directly)
|
||
}
|
||
}
|
||
|
||
message ControlMessage {
|
||
oneof msg {
|
||
StartShard start = 1;
|
||
StopShard stop = 2;
|
||
DrainShard drain = 3;
|
||
Ack ack = 4;
|
||
}
|
||
}
|
||
```
|
||
|
||
- Commands are idempotent: a repeated `StartShard` with the same `shard_id` is a no-op.
|
||
- mTLS, with node identity taken from the certificate CN. The control plane
|
||
compares the CN against the node id the agent claims and refuses the stream
|
||
when they disagree, which is what makes a node's identity something it proves
|
||
rather than something it picks.
|
||
- The CA is internal, and has to be: public authorities stopped issuing the
|
||
`clientAuth` usage, so no Let's Encrypt certificate can authenticate an agent
|
||
however convenient it would be. `llmgw ctl cert init-ca | issue-server |
|
||
issue-client` runs a small authority whose key lives on one machine and whose
|
||
root is trusted everywhere.
|
||
- The server certificate must carry every address agents dial as a subject
|
||
alternative name, bare IP addresses included. Hostname verification ignores
|
||
the common name entirely, so a certificate that names the host only in its
|
||
subject is rejected by every client.
|
||
- This covers the fleet-facing port only. The client-facing API is a public
|
||
surface and is fronted by a publicly trusted certificate instead.
|
||
- Three missed heartbeats mark the node `Unreachable`; after 60s it becomes `Lost` and its replicas are rescheduled.
|
||
|
||
---
|
||
|
||
## 6. Control Plane
|
||
|
||
### 6.1 Data model (abridged)
|
||
|
||
```go
|
||
type Node struct {
|
||
ID, Hostname, MgmtAddr string
|
||
Labels map[string]string
|
||
GPUs []GPU
|
||
Fabrics []FabricAttachment // empty means no IB
|
||
State NodeState // Ready, Cordoned, Draining, Unreachable, Lost
|
||
LastSeen time.Time
|
||
}
|
||
|
||
type FabricAttachment struct {
|
||
Fabric string // "ib-a"
|
||
HCAs []string
|
||
Rate int // Gbps
|
||
}
|
||
|
||
type Model struct {
|
||
Name string // what users see
|
||
Aliases []string
|
||
Source string // path or HF id
|
||
Profiles []Profile
|
||
Scaling Scaling // min/max replicas, target queue, ...
|
||
Fallbacks []string // other model names
|
||
}
|
||
|
||
type Profile struct {
|
||
Name string
|
||
GPUClass string // "h100-80g"
|
||
Engine string // "vllm"
|
||
TP, PP int
|
||
GPUsPerNode int // usually TP
|
||
Nodes int // PP*... ; a fabric is mandatory when > 1
|
||
Args []string // --quantization fp8 --max-model-len 32768 ...
|
||
Env map[string]string
|
||
Weight int // capacity weight for the router (from benchmarks)
|
||
Priority int // placement preference
|
||
StartupTimeout time.Duration
|
||
}
|
||
|
||
type Replica struct {
|
||
ID, Model, Profile string
|
||
Fabric string // set for multi-node replicas
|
||
Shards []Shard
|
||
State ReplicaState
|
||
Endpoint string // rank0's HTTP address
|
||
}
|
||
```
|
||
|
||
### 6.2 Scheduler (placement)
|
||
|
||
Input: the desired replica count, the profile list, and the inventory.
|
||
|
||
**Hard** constraints:
|
||
1. All shards share one `GPUClass`.
|
||
2. When `Nodes > 1`, every node must attach to the same `Fabric` and have ACTIVE HCAs on it.
|
||
3. `GPUsPerNode` free GPUs, preferably in the same NVLink domain and on the same NUMA node as the HCA.
|
||
4. The node is `Ready` and not `Cordoned`.
|
||
5. The engine's `Validate` must pass (e.g. an FP8 profile never lands on a card with compute capability < 8.9).
|
||
|
||
**Soft** preferences (scored):
|
||
- the profile's `Priority`
|
||
- bin packing: fill partially used nodes first, keeping empty nodes free for large models
|
||
- spread a model's replicas across fabrics and nodes (failure domain distribution)
|
||
- GPU-to-HCA NUMA affinity
|
||
|
||
Algorithm (v1, simple and deterministic):
|
||
|
||
```
|
||
for each profile in model.Profiles sorted by Priority desc:
|
||
pools := groupNodes(by GPUClass == profile.GPUClass, by Fabric if profile.Nodes > 1)
|
||
for each pool sorted by score:
|
||
nodes := pickNodes(pool, profile.Nodes, profile.GPUsPerNode) // greedy + NVLink group selection
|
||
if nodes != nil:
|
||
return Placement{profile, pool.Fabric, nodes, gpuAssignments}
|
||
return ErrUnschedulable (with reasons — essential for debugging)
|
||
```
|
||
|
||
`groupNodes`: nodes without IB join the pool for `Nodes == 1` profiles, and never for `Nodes > 1`.
|
||
|
||
`llmgw ctl explain-placement <model>` must show why each pool was rejected.
|
||
|
||
### 6.2.1 Heterogeneous GPUs: the SLO/capacity model
|
||
|
||
Comparing GPU classes through a single scalar "weight" is too crude; the model taken from
|
||
AIBrix's heterogeneous GPU optimizer is more accurate. For each (model, profile) pair,
|
||
`ctl bench` (§10) produces a **capacity profile**:
|
||
|
||
```go
|
||
type CapacityProfile struct {
|
||
Model, Profile string
|
||
GPUClass string
|
||
// Maximum concurrency that still meets the SLO; the real capacity measure.
|
||
MaxConcurrencyAtSLO int
|
||
ThroughputTokPerSec float64
|
||
TTFTp99, TPOTp99 time.Duration
|
||
RelativeCost float64 // relative GPU-hour cost, from configuration
|
||
MeasuredAt time.Time
|
||
EngineVersion string // the profile expires when the version changes
|
||
}
|
||
```
|
||
|
||
An example SLO: `TPOT p99 ≤ 50ms && TTFT p99 ≤ 2s`.
|
||
|
||
The profile is used in three places:
|
||
1. **Router weight:** `Profile.Weight` is no longer hand-written; it derives from `MaxConcurrencyAtSLO`.
|
||
2. **Admission thresholds:** `TargetWaitingPerReplica` (§7.2.1) is suggested from the measurement.
|
||
3. **Placement and autoscaling objective:** "minimize total cost while meeting the SLO". For a
|
||
predicted load, the replica count needed per GPU class is `ceil(load / MaxConcurrencyAtSLO)`,
|
||
and the cheapest combination wins. In v1 this only produces advice via
|
||
`ctl plan <model> --target-rps N`; automatic application arrives with the v1.1 autoscaler.
|
||
|
||
When `EngineVersion` or the profile arguments change the measurement goes stale. llmgw must
|
||
surface that (`llmgw_capacity_profile_stale{model,profile}`) rather than silently reusing an
|
||
obsolete number.
|
||
|
||
### 6.3 Reconciler
|
||
- Every 5s, or on an event: compare desired (model scaling) with actual (replica states).
|
||
- Missing replica → schedule → send `StartShard` for each shard (workers first, then rank0, or whatever order the engine requires).
|
||
- Any shard in a replica `Failed` → stop the whole replica and reschedule with backoff.
|
||
- Excess replicas → `Drain` first (the router stops sending new requests, existing ones finish), then `Stop`.
|
||
- Node `Lost` → treat its replicas as `Failed`.
|
||
|
||
Autoscaling (v1.1): a per-model `targetWaitingPerReplica`, averaged from router metrics, with
|
||
cooldown-guarded scaling. Shrinking lower-priority models when no GPUs are free (preemption) is v2.
|
||
|
||
### 6.4 Interface to the router
|
||
|
||
```go
|
||
type EndpointSource interface {
|
||
// model name -> Ready replicas (+ pool, weight, fabric, gpuclass, capacity profile)
|
||
Endpoints(model string) []Endpoint
|
||
Subscribe() <-chan struct{} // change notification
|
||
}
|
||
```
|
||
In-memory in v1; once the router and control plane split apart, the same interface is backed by
|
||
a watch stream.
|
||
|
||
The router **always caches** this list: even if the control plane goes down it keeps routing to
|
||
the replicas it knows about, which mitigates the single-point-of-failure risk in §14. When the
|
||
cache ages past `endpoint_cache_max_age` (say 5 min) it only raises a warning metric; it never
|
||
stops traffic.
|
||
|
||
### 6.5 What is stored, and what is not
|
||
|
||
v0.1 of this plan listed eight tables. Most of them were a mistake, and the
|
||
reason is worth stating because it applies to every piece of state here:
|
||
|
||
> **A database that describes the cluster can be wrong. The cluster cannot.**
|
||
|
||
An engine that crashes during a restart leaves a row saying "running". A node
|
||
whose GPU developed an ECC fault leaves a row saying the card is fine. Every
|
||
such row is a trap: the scheduler believes it, places work accordingly, and the
|
||
failure surfaces somewhere else entirely. Rebuilding the same state from the
|
||
fleet on every reconnect cannot go stale, because the fleet is the thing being
|
||
described.
|
||
|
||
So:
|
||
|
||
| State | Persisted | Where it actually comes from |
|
||
|---|---|---|
|
||
| Nodes, GPUs, fabrics | no | Every agent sends its full inventory on registration (§5.6) |
|
||
| Replicas, shards | no | Reported by agents; see the restart hole below |
|
||
| Models, profiles | no | The YAML catalog is the source of truth |
|
||
| API keys and the tenancy tree | **yes** | Created at runtime, secret, rotated (§7.9) |
|
||
| Usage records and rollups | **yes** | Accounting must survive a restart (§7.9) |
|
||
| Capacity profiles from `ctl bench` | **yes** | Measured, not declared (§6.2.1) |
|
||
| Audit events | **yes** | An audit trail that forgets is not one |
|
||
|
||
Everything in the "yes" column belongs to tenancy and benchmarking, which is
|
||
phase 5. **The store lands there, not in phase 3**, so its schema is shaped by
|
||
real requirements instead of guesses.
|
||
|
||
#### The control plane restart hole
|
||
|
||
Not persisting replica state has one consequence that must be closed before it
|
||
matters. If the control plane restarts while engines are running:
|
||
|
||
1. The engines keep serving — correct, and the reason the router talks to them
|
||
directly.
|
||
2. Agents reconnect and re-register their hardware.
|
||
3. The control plane has no memory of any replica, because `Register` does not
|
||
carry one.
|
||
4. The reconciler sees zero replicas and places new ones.
|
||
5. The old engines keep holding GPUs that the inventory now believes are free,
|
||
so the new engines fail to allocate — with an error pointing at the new
|
||
process rather than the old one.
|
||
|
||
The fix is on the agent side, not in a database: `Register` carries the shards
|
||
the agent is currently supervising, and the control plane rebuilds replica state
|
||
from what the fleet reports. The supervisor already knows its live shards and
|
||
already finds, through pid files, the ones that survived an agent restart
|
||
(`FindOrphans`). A shard nothing claims after one reconciliation pass is an
|
||
orphan and is killed.
|
||
|
||
This is strictly better than a database would be: it cannot disagree with
|
||
reality, and it covers agent restarts and control plane restarts with one
|
||
mechanism.
|
||
|
||
#### Store choice
|
||
|
||
SQLite in WAL mode, via `modernc.org/sqlite` so the server stays cgo-free.
|
||
|
||
The deciding argument is not throughput. One row per request at 1000 req/s is 86
|
||
million rows a day, which sounds like a reason to reach for Postgres, but the
|
||
write path is batched anyway (§7.9.3) and a transaction of 500 rows every 200ms
|
||
is nothing for either engine. Batching is required for a different reason, and
|
||
once it exists the volume argument favours neither.
|
||
|
||
What decides it is the failure mode. Postgres is a second service, and the
|
||
question it raises is whether the gateway still serves traffic when that service
|
||
is unreachable. An inference gateway that stops serving because its accounting
|
||
database is down is worse than one that buffers some accounting. SQLite is
|
||
embedded, so the store shares the gateway's fate rather than adding an
|
||
independent way to be down — and the single-binary goal falls out of the same
|
||
choice.
|
||
|
||
It stops being enough at the point where there is more than one gateway
|
||
instance, because budgets and rate-limit counters then have to be counted
|
||
somewhere shared; per-instance counters mean N instances allow N times the
|
||
configured limit (§7.9.6). Postgres arrives with that, not before.
|
||
|
||
Three rules keep the swap cheap, and they are worth following from the first
|
||
commit:
|
||
|
||
1. **Nothing on the request path touches the database.** Keys are cached in
|
||
memory and read through on a miss, usage goes to a batch writer, budgets are
|
||
in-memory counters reconciled against the store. This matters more than the
|
||
engine choice and is what makes either engine survivable.
|
||
2. **Portable SQL.** No `AUTOINCREMENT` and no rowid tricks, `ON CONFLICT … DO
|
||
UPDATE` rather than `INSERT OR REPLACE`, timestamps as integer unix millis,
|
||
money and tokens as integers.
|
||
3. **Migrations per dialect, sharing version numbers.** A single migration set
|
||
for both engines would be a promise that cannot be kept — SQLite's
|
||
`ALTER TABLE` barely exists. Same numbering, same meaning, different files.
|
||
|
||
---
|
||
|
||
## 7. Router
|
||
|
||
### 7.1 Request lifecycle
|
||
|
||
```
|
||
Request
|
||
├─ 1. Auth : key → project/team, permitted models, priority band, routing profile
|
||
├─ 2. Quota : RPM/TPM and budget (in-memory counters; no DB on the request path)
|
||
├─ 3. Parse : model, stream, messages/prompt, max_tokens, user.
|
||
│ The rest of the body passes through as untouched JSON.
|
||
├─ 4. Resolve : model/alias → candidate pools (fabric × gpu_class)
|
||
├─ 5. ADMISSION : "now?" §7.2 ← waits here while the pool is saturated
|
||
├─ 6. SCHEDULING : "where?" §7.3 filter → score → pick
|
||
├─ 7. Proxy : stream to upstream, propagate cancellation §7.8
|
||
└─ 8. After the request (async): usage, spend, metrics, logs
|
||
```
|
||
|
||
**The critical separation (taken from GIE / llm-d).** *When* a request proceeds and *where* it
|
||
goes are two distinct decisions and must stay distinct. Kept apart, the router can hold requests
|
||
in its own central queue and apply priority and tenant fairness **before they enter an engine
|
||
queue**.
|
||
|
||
Why this matters: an engine's own queue is FIFO and has no notion of tenants or priority. Once a
|
||
request is in there, no prioritization is possible. When a batch job fills the vLLM queue with
|
||
500 requests, the TTFT of an interactive chat request arriving behind them collapses and there
|
||
is nothing the router can do. v0.1 had no such layer; **for a shared cluster this was the most
|
||
serious omission.**
|
||
|
||
### 7.2 Admission control (flow control)
|
||
|
||
#### 7.2.1 Saturation detector
|
||
|
||
One saturation signal is produced per pool, combining two metrics (llm-d's approach — queue
|
||
depth alone is not enough, KV pressure is a genuine exhaustion signal too):
|
||
|
||
```go
|
||
// per replica, then aggregated across the pool as p90 or mean
|
||
func saturation(e EngineStats, t Thresholds) float64 {
|
||
q := float64(e.Waiting) / t.TargetWaitingPerReplica // e.g. target 4
|
||
kv := e.KVCacheUsage / t.KVCeiling // e.g. ceiling 0.90
|
||
return math.Max(q, kv)
|
||
}
|
||
```
|
||
|
||
The goal is to keep a **small but healthy buffer** on the engine: enough requests for continuous
|
||
batching to stay efficient, but not enough for the queue to balloon. Set
|
||
`TargetWaitingPerReplica` too low and the GPU starves; too high and queueing delay returns. Both
|
||
thresholds must be tunable per model/profile, and `ctl bench` (§10) should propose them.
|
||
|
||
**Two lags have to be closed, and they need different answers.** The engine's queue is scraped,
|
||
so it is up to one interval old; the router's own inflight count is not. But inflight includes
|
||
*running* requests, and treating those as queue makes a perfectly healthy engine look overloaded
|
||
every interval. What is actually queued is `inflight − batch capacity`, and batch capacity is not
|
||
in the metric contract — so llmgw learns it as the high-water mark of observed `running`. The
|
||
second lag is shorter and subtler: between admission granting a request and the router
|
||
dispatching it, the request is invisible everywhere. A simultaneous burst can therefore pass the
|
||
check en masse, which is exactly the batch-job arrival pattern this layer exists to survive. The
|
||
controller closes it by counting its own outstanding leases against dispatched inflight.
|
||
|
||
#### 7.2.2 Priority bands
|
||
|
||
Every API key maps to a band, and the band sets the saturation ceiling:
|
||
|
||
```yaml
|
||
bands:
|
||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 } # flows to the healthy limit
|
||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 } # batch jobs
|
||
```
|
||
|
||
A ceiling of 1.00 means "hold only once the pool is past its healthy limit",
|
||
not "never hold". Holding even the top band is deliberate: there is nobody above
|
||
it to protect, so the benefit is a bounded engine queue. A flood dumped straight
|
||
into a FIFO engine queue makes every request in that band slow, while queueing it
|
||
at the gateway preserves ordering, TTL and tenant fairness. A band that must
|
||
never wait — a health probe, say — gets a ceiling above 1 (`1000`).
|
||
|
||
The dispatch loop scans bands from highest to lowest. Once pool saturation exceeds a band's
|
||
ceiling, that band's requests wait in the queue. The result: as the cluster fills, batch traffic
|
||
slows down first and interactive traffic is protected. (In llm-d's measurements platinum, gold
|
||
and silver stayed under a second while bronze batch fell far behind — exactly the desired
|
||
behaviour.)
|
||
|
||
#### 7.2.3 Fairness and queue management
|
||
|
||
- **Fairness within a band:** when several tenants share a band, dispatch turns are distributed
|
||
round-robin. One tenant's traffic spike cannot consume the whole band.
|
||
- **Capacity:** each band's queue is bounded. When it is full, new requests get an immediate
|
||
`503` — far better than accumulating silently.
|
||
- **TTL:** a request that waits longer than `queue_ttl` is dropped (`504`). There is no point
|
||
sending the GPU work the client has already given up on.
|
||
- **Client disconnect:** if the connection drops while queued, the request is discarded quietly
|
||
(context cancellation).
|
||
- **Displacement (v1.1):** to make room for a higher band, drop the **newest** requests from the
|
||
lowest band — the oldest ones are already near their TTL and have invested the most waiting.
|
||
|
||
```go
|
||
type Admission interface {
|
||
// Blocks until the request may proceed; errors on cancellation or TTL.
|
||
Acquire(ctx context.Context, req *RequestInfo) (Lease, error)
|
||
}
|
||
```
|
||
|
||
`Lease.Release()` is called once the response completes; it also maintains the router's inflight
|
||
counter.
|
||
|
||
### 7.3 Scheduler: filter → score → pick
|
||
|
||
The list of "strategies" from v0.1 is replaced by a plugin pipeline (the GIE EPP model). No
|
||
strategy is a separate monolith; scorers are combined by weight.
|
||
|
||
```go
|
||
type Filter interface { // drops candidates (hard constraints)
|
||
Name() string
|
||
Filter(ctx *SchedCtx, eps []Endpoint) []Endpoint
|
||
}
|
||
|
||
type Scorer interface { // 0..1 score; 1 is best
|
||
Name() string
|
||
Score(ctx *SchedCtx, eps []Endpoint) map[EndpointID]float64
|
||
}
|
||
|
||
type Picker interface { // chooses from the scores
|
||
Pick(ctx *SchedCtx, scores map[EndpointID]float64) Endpoint
|
||
}
|
||
|
||
type RoutingProfile struct {
|
||
Name string
|
||
Filters []Filter
|
||
Scorers []WeightedScorer // {Scorer, Weight}
|
||
Picker Picker
|
||
}
|
||
```
|
||
|
||
Total score: `score(ep) = Σ wᵢ · scoreᵢ(ep)`, with each scorer normalizing to 0..1 internally.
|
||
|
||
**A two-stage decision.** Because the topology is heterogeneous, the pool is chosen first and
|
||
the replica second:
|
||
|
||
1. **Pool selection:** the pool with the highest `Priority`; if every replica there is saturated
|
||
(§7.2.1), spill over to the next pool. This prevents falling back to the A100 pool before the
|
||
H100 pool is actually full.
|
||
2. **Replica selection:** filter/score/pick within the chosen pool.
|
||
|
||
**Routing profiles**, from configuration (a per-model default, overridable per request via the
|
||
`x-llmgw-profile` header):
|
||
|
||
| Profile | Weights | Use |
|
||
|---|---|---|
|
||
| `chat` | prefix 3.0, queue 1.0, kv 1.0 | Multi-turn chat and agents — high cache reuse |
|
||
| `batch` | queue 3.0, kv 1.0, prefix 0.5 | Bulk work — throughput matters, TTFT does not |
|
||
| `long-context` | kv 3.0, prefix 2.0, queue 1.0 | Long prompts — KV pressure is the real bottleneck |
|
||
| `debug` | round-robin picker | Troubleshooting |
|
||
|
||
### 7.4 Scorers and their formulas
|
||
|
||
| Scorer | Formula | Notes |
|
||
|---|---|---|
|
||
| `queue` | `1 − min(1, waiting / target_waiting)` | Queue depth |
|
||
| `kv` | `1 − kv_cache_usage` | KV occupancy; a hard penalty above the ceiling |
|
||
| `prefix` | `matched_blocks / prompt_blocks` | §7.5 |
|
||
| `load` | `1 − norm(running / drain_rate)` | AIBrix: `drain_rate` is the observed completion rate. `running` alone cannot spot a slow replica. |
|
||
| `inflight` | `1 − norm(router_inflight)` | The router's own count; compensates for scrape lag. **At a 2s scrape interval, herding is unavoidable without it.** |
|
||
| `pool-priority` | constant, from the profile's `Priority` | Pool preference |
|
||
| `session` | session hash match | §7.5 tier 0 |
|
||
| `lora` (v2) | is the adapter loaded | `vllm:lora_requests_info` |
|
||
|
||
**An alternative cost formulation (Dynamo).** Modelling prefill and decode cost directly in
|
||
blocks may be more accurate than normalized scores:
|
||
|
||
```
|
||
cost(ep) = w_overlap · (prompt_blocks − cached_blocks(ep)) + decode_blocks(ep)
|
||
choice = argmin cost
|
||
```
|
||
|
||
A high `w_overlap` favours cache reuse (better TTFT); a low one evens out load (better ITL).
|
||
Both can be expressed as a `Scorer`, so the pipeline survives either way: weighted scores in v1,
|
||
with this cost form added as a scorer in v1.1 and compared using `ctl bench`.
|
||
|
||
**Pickers:**
|
||
- `max-score`: the highest score. Deterministic but prone to herding.
|
||
- `p2c` (power of two choices): two random candidates, the better one. The default once there are many replicas.
|
||
- `softmax(temperature)`: Dynamo's `router_temperature`. Treats scores as logits and samples. `temperature=0` degenerates to argmax. A small temperature breaks the oscillation that synchronized decisions cause among equally scored replicas.
|
||
|
||
### 7.5 Prefix / KV cache awareness — three tiers
|
||
|
||
This is the single largest win the router can deliver, but the cost gap between the tiers is
|
||
enormous. They stay separate so they can be enabled per model.
|
||
|
||
#### Tier 0 — session affinity (v1, cheap, high return)
|
||
The client-supplied `x-session-id` header (falling back to OpenAI's `user` field) maps to a
|
||
replica via bounded-load consistent hashing. If that replica is overloaded — beyond
|
||
`load_factor` times the average — the next one on the ring is used. Roughly 100 lines of code;
|
||
production-stack measured cache hits going from about 93% to 96% versus round robin. For
|
||
multi-turn chat and agent workloads this is the best cost/benefit ratio available.
|
||
|
||
#### Tier 1 — approximate radix tree (v1.1, moderate cost)
|
||
The router maintains a tree that **estimates** each worker's cache from the request prefixes it
|
||
has seen (the SGLang router's approach). It does not know the true cache state; it updates
|
||
lazily and costs little.
|
||
|
||
The thresholds — the part that matters in real implementations:
|
||
- `cache_threshold`: below this match ratio, treat it as "no cache" and let load balancing decide.
|
||
- `balance_abs_threshold` **and** `balance_rel_threshold`: if `max_load − min_load > abs`
|
||
**and** `max_load / min_load > rel`, **ignore the prefix entirely** and send to the least
|
||
loaded replica. (AIBrix adds the same guard.) Without it, prefix affinity drowns one replica:
|
||
the cache hit rate rises while p99 latency collapses.
|
||
- `eviction_interval` plus `max_tree_size`: keep the tree from growing without bound.
|
||
|
||
Which branch a decision took must be recorded as a metric: `cache_hit | cache_miss_min_load |
|
||
load_balance_override | stale | no_tree | no_healthy_worker`. Without that, tuning prefix routing
|
||
is guesswork.
|
||
|
||
#### Tier 2 — precise KV index (v2, expensive, optional)
|
||
vLLM publishes block events over ZMQ with `--kv-events-config` (`BlockStored`, `BlockRemoved`).
|
||
The router subscribes and maintains a `block_hash → {replica}` map, giving it a true global view.
|
||
|
||
**The trap — a cost the plan must acknowledge:** block hashes must be computed identically to
|
||
vLLM's **token-id-based** scheme. That forces the router to tokenize requests with **the same
|
||
tokenizer as the engine**. On the Go side that means loading a tokenizer per model (HF
|
||
`tokenizers` via cgo, or a pure-Go implementation), tracking version compatibility, and paying
|
||
the tokenization CPU cost per request. This is why AIBrix requires a "remote tokenizer".
|
||
**Tiers 0 and 1 need no tokenizer** — hence they are v1 and this is v2.
|
||
|
||
Event loss: if a gap appears in the event sequence numbers, request the missing range from the
|
||
replay endpoint; if that fails, drop the index for that replica and fall back to **tier 1** for
|
||
it. Returning to approximate information beats silently routing on wrong data. If a replica
|
||
restarts, its index is reset.
|
||
|
||
### 7.6 The engine metric contract
|
||
|
||
This is the formal definition of "adding a new engine". If an engine exposes these metrics
|
||
in Prometheus format, llmgw can route to it (GIE's model server protocol):
|
||
|
||
| Contract field | vLLM metric | llama.cpp metric | Mandatory |
|
||
|---|---|---|---|
|
||
| `Waiting` | `vllm:num_requests_waiting` | `llamacpp:requests_deferred` | yes |
|
||
| `Running` | `vllm:num_requests_running` | `llamacpp:requests_processing` | yes |
|
||
| `KVCacheUsage` | `vllm:kv_cache_usage_perc` | — | where the engine has it |
|
||
| `PromptTokens` | `vllm:prompt_tokens_total` | `llamacpp:prompt_tokens_total` | no |
|
||
| `GenerationTokens` | `vllm:generation_tokens_total` | `llamacpp:tokens_predicted_total` | no |
|
||
| `BlockSize`, `NumGPUBlocks` | `vllm:cache_config_info` | — | no |
|
||
| LoRA state | `vllm:lora_requests_info` (`max_lora`, `running_lora_adapters`, `waiting_lora_adapters`) | — | no |
|
||
|
||
Mappings are **per engine, not per cluster**. A cluster running llama.cpp on its CPU nodes and
|
||
vLLM on its GPU nodes has two vocabularies for the same numbers, and one global mapping would
|
||
leave half the fleet unmeasurable. An endpoint that names no engine is read with vLLM's names.
|
||
|
||
`Running` and `Waiting` are the two that cannot be omitted: an engine that reports neither how
|
||
much it is doing nor how much is queued cannot be scheduled against at all. `KVCacheUsage` is
|
||
required *of engines that publish it* — a mapping that names it and does not get it is a contract
|
||
violation, while a mapping that leaves it unnamed is a statement that this engine has no such
|
||
metric. Current llama.cpp is the second case, and saturation there falls back to queue depth
|
||
alone. That is a real loss of signal, mitigated by llama.cpp serving a fixed number of slots, so
|
||
its queue depth already says when it is full.
|
||
|
||
> Correction: an earlier version of this plan gave llama.cpp's KV metric as
|
||
> `llamacpp:kv_cache_usage_ratio`. Build b10991 does not publish it — it serves fifteen metric
|
||
> families and none is a cache ratio. Naming it failed the contract on every scrape and left the
|
||
> router with no numbers at all rather than two out of three.
|
||
|
||
> Correction: v0.1 said `vllm:gpu_cache_usage_perc`; the correct name is
|
||
> `vllm:kv_cache_usage_perc`. Metric names drift between versions, so the `ParseMetrics` mapping
|
||
> must be overridable from configuration, and the agent must report the engine version when it
|
||
> registers.
|
||
|
||
Scrape rules:
|
||
- Every Ready endpoint every 1–2s, one goroutine per endpoint, with jitter.
|
||
- A failed scrape marks the endpoint stale; its score drops but it is not removed.
|
||
- Three consecutive failures trip the breaker (§7.7).
|
||
- A **contract violation is reported at warn level, once**, and again when it clears. It is a
|
||
compatibility problem rather than a blip: every scrape fails, the router schedules to that
|
||
endpoint blind, and per-scrape debug logging meant a cluster could look entirely healthy while
|
||
none of its routing decisions were informed by anything.
|
||
- **So that decisions are never made purely on stale data**, the `inflight` scorer (§7.4) stays
|
||
enabled at all times.
|
||
|
||
### 7.7 Reliability
|
||
|
||
- **Retry:** only before the first byte has been written upstream, only on connection errors,
|
||
503 or 429, always against a different endpoint, at most N times. No retries once the stream
|
||
has started.
|
||
- **Cooldown (the LiteLLM lesson):** an endpoint that errors leaves the candidate pool for
|
||
`cooldown_seconds`. A long fallback list plus retries without cooldown equals a retry storm:
|
||
one endpoint's rate limit spreads across the whole cluster.
|
||
- **Circuit breaker:** a closed/open/half-open state machine per endpoint, with exponential backoff.
|
||
- **Fallback:** when a model has no usable endpoints, fall back to a model in its `Fallbacks`
|
||
list (if the key permits it). The response carries an `x-llmgw-served-model` header.
|
||
**Budget fallback:** when a key exceeds its budget, optionally downgrade to a cheaper model
|
||
rather than rejecting the request (§7.9).
|
||
- **Timeouts:** separate limits for connection, first token (TTFT) and total, plus an
|
||
inter-token idle timeout while streaming.
|
||
- **Cancellation:** when the client disconnects, cancel the upstream request too (context
|
||
propagation) so the GPU stops working. This applies to queued requests as well.
|
||
- **Draining:** a `Draining` replica receives no new requests; active ones run to completion or
|
||
time out.
|
||
- **Control plane loss:** the router keeps serving from its last known endpoint list (§14).
|
||
|
||
### 7.8 Streaming proxy details
|
||
|
||
- `http.Transport` with a high `MaxIdleConnsPerHost` and HTTP/1.1 keep-alive (engines rarely
|
||
speak h2c).
|
||
- Read lines with `bufio.Reader`, pass `data:` lines through, and call `http.Flusher.Flush()`
|
||
after every event.
|
||
- If the client did not send `stream_options.include_usage`, the router may add it
|
||
(configurable) to capture usage from the final chunk, and then withhold that chunk from a
|
||
client that did not ask for it.
|
||
- Normalize upstream error JSON into OpenAI's format.
|
||
- Response headers: `x-llmgw-replica`, `x-llmgw-pool`, `x-llmgw-served-model`,
|
||
`x-llmgw-queue-ms` (time spent in admission) and `x-llmgw-cache-decision` (§7.5).
|
||
|
||
### 7.9 Identity, quota and usage (tenancy)
|
||
|
||
LiteLLM's production-proven model:
|
||
|
||
```
|
||
Organization ─► Team ─► Project ─► API Key
|
||
```
|
||
Each level carries a budget (in tokens or notional cost units), RPM/TPM limits and a permitted
|
||
model list. A key additionally carries its **priority band** (§7.2.2) and its **default routing
|
||
profile** (§7.3).
|
||
|
||
- API keys look like `sk-llmgw-...`; only the SHA-256 hash is stored. A leaked key is rotated
|
||
without touching upstream credentials.
|
||
- Rate limiting: RPM via `x/time/rate` plus **TPM**. The token limit is reserved at request
|
||
start as `max_tokens + estimated prompt` and reconciled against the real figure when the
|
||
response completes. (The prompt estimate is bytes/4 in v1; it becomes exact once the tier 2
|
||
tokenizer arrives.)
|
||
- **Budgets:** periodic (daily/monthly) spend. On exhaustion: `429`, or a budget fallback.
|
||
Zero-cost models are exempt from budget checks, so a key whose budget is spent can still be
|
||
given a "cheap model" fallback.
|
||
- **Spend writes stay off the request path:** usage records go to a channel and are batched into
|
||
SQLite, with hourly rollups. Nothing writes to the database on the request path.
|
||
- **Replica pinning (debug):** the `x-llmgw-replica` header, scoped to the key. If the pinned
|
||
replica is in cooldown or gone, the request falls through to normal routing.
|
||
- v1 is a single instance with in-memory limiters. With multiple instances this becomes Redis or
|
||
consistent-hash stickiness (v2).
|
||
|
||
#### 7.9.1 How keys are stored
|
||
|
||
A key is a 256-bit random value, shown once at creation and never again:
|
||
|
||
```
|
||
sk-llmgw-<base62(32 random bytes)>
|
||
```
|
||
|
||
Only `sha256(key)` is stored, indexed. Authentication hashes what the client
|
||
presented and looks it up. Nothing can reconstruct the key from the database,
|
||
so a database leak does not hand over working credentials.
|
||
|
||
**SHA-256 rather than bcrypt or argon2, deliberately.** Slow password hashing
|
||
exists because humans pick guessable passwords, and the cost buys time against
|
||
an offline dictionary attack. A 256-bit random value has no dictionary to
|
||
attack: at any hash rate, guessing it is infeasible. What a slow hash would buy
|
||
here is nothing, and what it would cost is tens of milliseconds on every single
|
||
request. Using the fast hash is the correct choice precisely *because* the input
|
||
is high entropy — and that is the assumption to re-examine if the key format
|
||
ever changes.
|
||
|
||
Alongside the hash, a `key_prefix` (`sk-llmgw-4f2a…`) is stored in clear so the
|
||
UI and audit log can identify a key without revealing it.
|
||
|
||
```sql
|
||
CREATE TABLE api_keys (
|
||
id TEXT PRIMARY KEY,
|
||
key_hash BLOB NOT NULL UNIQUE, -- sha256 of the key; the only copy
|
||
key_prefix TEXT NOT NULL, -- for display, never for auth
|
||
name TEXT NOT NULL,
|
||
project_id TEXT NOT NULL REFERENCES projects(id),
|
||
band TEXT, -- §7.2.2; NULL inherits the project's
|
||
routing_profile TEXT, -- §7.3
|
||
created_at INTEGER NOT NULL,
|
||
expires_at INTEGER, -- NULL never expires
|
||
revoked_at INTEGER, -- soft delete, so usage rows keep a valid FK
|
||
last_used_at INTEGER -- written lazily; see below
|
||
);
|
||
CREATE INDEX api_keys_by_project ON api_keys(project_id);
|
||
```
|
||
|
||
Revocation is a soft delete. Usage rows reference the key, and an audit trail
|
||
that loses its subject when the key is deleted is not an audit trail.
|
||
|
||
`last_used_at` is written at most once a minute per key, from the same batch
|
||
writer as usage. Writing it per request would put a database write back on the
|
||
request path for a field nobody reads in real time.
|
||
|
||
#### 7.9.2 The tenancy tree and how limits resolve
|
||
|
||
```sql
|
||
CREATE TABLE organizations (id TEXT PRIMARY KEY, name TEXT NOT NULL);
|
||
CREATE TABLE teams (id TEXT PRIMARY KEY, org_id TEXT NOT NULL REFERENCES organizations(id), name TEXT NOT NULL);
|
||
CREATE TABLE projects (id TEXT PRIMARY KEY, team_id TEXT NOT NULL REFERENCES teams(id), name TEXT NOT NULL);
|
||
|
||
-- Limits hang off any level rather than being duplicated onto each one.
|
||
CREATE TABLE limits (
|
||
scope_type TEXT NOT NULL, -- 'org' | 'team' | 'project' | 'key'
|
||
scope_id TEXT NOT NULL,
|
||
rpm INTEGER, -- NULL means no limit at this level
|
||
tpm INTEGER,
|
||
budget_micros INTEGER, -- per period, in micro-units; NULL means no budget
|
||
budget_period TEXT, -- 'day' | 'month'
|
||
allowed_models TEXT, -- JSON array; NULL means all
|
||
PRIMARY KEY (scope_type, scope_id)
|
||
);
|
||
```
|
||
|
||
A request's effective limit is the **most restrictive** value along
|
||
key → project → team → org. A team cannot raise a limit its organization set,
|
||
which is the only composition rule that makes a budget mean anything. Model
|
||
allowlists compose the same way, by intersection: a key cannot add a model its
|
||
team withheld. An **empty** allowlist is not an absent one — absent means every
|
||
model, empty means none.
|
||
|
||
Budgets are counted in integer micro-units rather than a float. A budget is
|
||
accumulated by repeated addition, and repeated addition of floats drifts.
|
||
|
||
Resolving four levels per request would be four lookups on the hot path, so the
|
||
resolved view is computed once and cached per key hash, invalidated whenever
|
||
anything on that key's chain changes. Authentication is then a single map
|
||
lookup, with the database consulted only on a cache miss.
|
||
|
||
#### 7.9.3 How accounting is written
|
||
|
||
The volume decides the design. At 1000 requests/sec, one row per request is
|
||
86 million rows a day, and SQLite has a single writer. Inserting per request
|
||
would serialize every request behind that writer.
|
||
|
||
So the path is:
|
||
|
||
```
|
||
request completes
|
||
→ usage record onto a buffered channel (nanoseconds, never blocks)
|
||
→ batch writer: every 200ms or 500 records, one transaction
|
||
→ hourly rollup job collapses raw rows
|
||
→ raw rows older than the retention window are deleted
|
||
```
|
||
|
||
```sql
|
||
CREATE TABLE usage (
|
||
ts INTEGER NOT NULL, -- unix millis
|
||
key_id TEXT NOT NULL,
|
||
project_id TEXT NOT NULL,
|
||
model TEXT NOT NULL,
|
||
served_model TEXT NOT NULL, -- differs from model when a fallback served it
|
||
pool TEXT,
|
||
status INTEGER NOT NULL,
|
||
prompt_tokens INTEGER NOT NULL,
|
||
completion_tokens INTEGER NOT NULL,
|
||
cached_tokens INTEGER NOT NULL, -- prefix cache hit, from the engine
|
||
ttft_ms INTEGER,
|
||
duration_ms INTEGER NOT NULL,
|
||
queue_ms INTEGER -- time spent in admission (§7.2)
|
||
);
|
||
CREATE INDEX usage_by_key_ts ON usage(key_id, ts);
|
||
|
||
-- What survives retention, and what budgets are enforced against.
|
||
CREATE TABLE usage_rollup (
|
||
hour_ts INTEGER NOT NULL,
|
||
key_id TEXT NOT NULL,
|
||
project_id TEXT NOT NULL,
|
||
model TEXT NOT NULL,
|
||
requests INTEGER NOT NULL,
|
||
prompt_tokens INTEGER NOT NULL,
|
||
completion_tokens INTEGER NOT NULL,
|
||
cached_tokens INTEGER NOT NULL,
|
||
units REAL NOT NULL, -- billable cost, per §7.9.4
|
||
PRIMARY KEY (hour_ts, key_id, model)
|
||
);
|
||
```
|
||
|
||
**The honest tradeoff:** records sitting in the channel when the process is
|
||
killed are lost — at most a few hundred milliseconds of accounting. Making that
|
||
impossible means a durable write before each response, which puts a disk fsync
|
||
on every request. For usage that informs budgets and invoices, losing a fraction
|
||
of a second of records to a hard kill is the better trade. **If this ever has to
|
||
feed external billing, that trade has to be revisited**, and the answer then is
|
||
an append-only journal the batch writer drains, not a synchronous insert.
|
||
|
||
#### 7.9.4 How budgets are enforced without reading the database
|
||
|
||
Budgets are enforced per request, so they cannot involve a query. Instead:
|
||
|
||
- On startup, the current period's spend is loaded per key from `usage_rollup`
|
||
plus the un-rolled tail of `usage`. **A restart must not reset a budget**, which
|
||
it would if the counter lived only in memory.
|
||
- From then on an in-memory counter is incremented as responses complete. The
|
||
database catches up through the batch writer.
|
||
- At the period boundary the counter resets and the new period starts empty.
|
||
|
||
"Units" are deliberately not dollars. Self-hosted models have no per-token
|
||
price, so a unit is whatever the operator configures — GPU-seconds, a notional
|
||
rate per million tokens, or 1.0 for everything, which reduces budgets to token
|
||
quotas. This is also what makes the plan's budget fallback work: a model priced
|
||
at zero stays available to a key whose budget is spent (§7.7).
|
||
|
||
#### 7.9.5 Token counting
|
||
|
||
Two numbers exist and they are not the same:
|
||
|
||
- **The estimate**, computed before dispatch as `bytes/4`, used to reserve TPM
|
||
quota and to score prefix affinity. It is a guess and is labelled as one.
|
||
- **The engine's report**, taken from the response's `usage` block. This is
|
||
authoritative and is what gets recorded.
|
||
|
||
The router asks engines for usage even when the client did not, then drops that
|
||
chunk on the way back (§7.8), so accounting does not depend on client behaviour.
|
||
TPM is reserved at request start as `max_tokens + estimate` and reconciled
|
||
against the real figure on completion — reserving optimistically would let a
|
||
key exceed its limit by a whole response.
|
||
|
||
The estimate stops being a guess only when a real tokenizer arrives, which the
|
||
plan defers to v2 along with tier 2 prefix routing (§7.5, §14).
|
||
|
||
#### 7.9.6 Multiple router instances
|
||
|
||
Everything above assumes one instance. Two would each hold their own rate limit
|
||
counters and each allow the configured limit, so the effective limit doubles.
|
||
|
||
v1 is single-instance and this is a documented constraint rather than a bug. The
|
||
answer when it arrives is either shared counters in Redis, or consistent hashing
|
||
on the key so every request for one key reaches the same instance. The second is
|
||
cheaper and degrades better, but rebalances counters whenever the instance set
|
||
changes.
|
||
|
||
---
|
||
|
||
## 8. Observability
|
||
|
||
Prometheus metrics (examples):
|
||
```
|
||
llmgw_requests_total{model,pool,status}
|
||
llmgw_request_duration_seconds{model,pool}
|
||
llmgw_ttft_seconds{model,pool}
|
||
llmgw_tokens_total{model,direction="prompt|completion"}
|
||
llmgw_endpoint_inflight{replica}
|
||
llmgw_endpoint_score{replica}
|
||
llmgw_replicas{model,state}
|
||
llmgw_gpu_free{node,gpu_class,fabric}
|
||
llmgw_schedule_failures_total{model,reason}
|
||
llmgw_agent_connected{node}
|
||
|
||
# admission / flow control (§7.2)
|
||
llmgw_admission_queue_depth{band,model}
|
||
llmgw_admission_wait_seconds{band,model} # histogram
|
||
llmgw_admission_rejected_total{band,reason="full|ttl|client_gone"}
|
||
llmgw_pool_saturation{pool} # 0..1, the detector's output
|
||
|
||
# routing decisions (§7.5) — prefix routing cannot be tuned without these
|
||
llmgw_cache_decision_total{model,decision="hit|miss_min_load|balance_override|stale|no_tree"}
|
||
llmgw_prefix_match_ratio{model} # histogram
|
||
llmgw_scorer_value{scorer,replica}
|
||
llmgw_endpoint_metrics_stale{replica}
|
||
llmgw_capacity_profile_stale{model,profile}
|
||
```
|
||
- OpenTelemetry traces: router span → upstream span (the traceparent header is forwarded to the engine).
|
||
- Structured logs (slog JSON) with `request_id` everywhere.
|
||
- `llmgw ctl top`: live queue depth and occupancy per pool.
|
||
|
||
---
|
||
|
||
## 9. Configuration Examples
|
||
|
||
### 9.1 server.yaml
|
||
```yaml
|
||
listen: ":8080"
|
||
admin_listen: ":9090" # metrics, admin API, pprof
|
||
agent_listen: ":7443"
|
||
tls:
|
||
ca: /etc/llmgw/ca.pem
|
||
cert: /etc/llmgw/server.pem
|
||
key: /etc/llmgw/server-key.pem
|
||
store:
|
||
path: /var/lib/llmgw/llmgw.db
|
||
|
||
fabrics:
|
||
- name: ib-a
|
||
gid_prefix: "fec0:0000:0000:000a" # optional automatic mapping
|
||
- name: ib-b
|
||
gid_prefix: "fec0:0000:0000:000b"
|
||
|
||
router:
|
||
scrape_interval: 2s
|
||
inject_usage: true
|
||
retries: 2
|
||
cooldown_seconds: 15 # how long a failing endpoint leaves the candidate pool
|
||
endpoint_cache_max_age: 5m # keep serving this long if the control plane goes down
|
||
|
||
# §7.2 admission — the saturation detector
|
||
admission:
|
||
target_waiting_per_replica: 4 # proposed by ctl bench
|
||
kv_ceiling: 0.90
|
||
queue_ttl: 30s
|
||
queue_capacity_per_band: 2000
|
||
bands:
|
||
- { name: platinum, priority: 400, saturation_ceiling: 1.00 }
|
||
- { name: gold, priority: 300, saturation_ceiling: 0.85 }
|
||
- { name: silver, priority: 200, saturation_ceiling: 0.60 }
|
||
- { name: bronze, priority: 100, saturation_ceiling: 0.30 }
|
||
|
||
# §7.3 routing profiles
|
||
default_profile: chat
|
||
profiles:
|
||
chat:
|
||
scorers: { prefix: 3.0, queue: 1.0, kv: 1.0, inflight: 1.0 }
|
||
picker: { type: p2c }
|
||
batch:
|
||
scorers: { queue: 3.0, kv: 1.0, load: 1.0, prefix: 0.5 }
|
||
picker: { type: max-score }
|
||
long-context:
|
||
scorers: { kv: 3.0, prefix: 2.0, queue: 1.0 }
|
||
picker: { type: softmax, temperature: 0.2 }
|
||
|
||
# §7.5 prefix awareness
|
||
prefix:
|
||
session_affinity: { enabled: true, header: x-session-id, load_factor: 1.25 } # tier 0
|
||
approx_tree: # tier 1
|
||
enabled: false # v1.1
|
||
cache_threshold: 0.30
|
||
balance_abs_threshold: 16
|
||
balance_rel_threshold: 1.5
|
||
max_tree_size: 2000000
|
||
eviction_interval: 60s
|
||
kv_events: { enabled: false } # tier 2 — needs a tokenizer, v2
|
||
|
||
models:
|
||
- name: llama-3.3-70b
|
||
source: /models/Llama-3.3-70B-Instruct
|
||
scaling: { min: 2, max: 6 }
|
||
fallbacks: [qwen2.5-32b]
|
||
routing_profile: chat
|
||
slo: { ttft_p99: 2s, tpot_p99: 50ms } # ctl bench measures capacity against this SLO
|
||
profiles:
|
||
- name: h100-tp4-fp8
|
||
gpu_class: h100-80g
|
||
engine: vllm
|
||
tp: 4
|
||
nodes: 1
|
||
priority: 100
|
||
weight: 100
|
||
args: ["--quantization", "fp8", "--max-model-len", "32768"]
|
||
- name: a100-tp8
|
||
gpu_class: a100-80g
|
||
engine: vllm
|
||
tp: 8
|
||
nodes: 1
|
||
priority: 50
|
||
weight: 45
|
||
|
||
- name: deepseek-large
|
||
source: /models/DeepSeek-V3
|
||
scaling: { min: 1, max: 2 }
|
||
profiles:
|
||
- name: h100-2node
|
||
gpu_class: h100-80g
|
||
engine: sglang
|
||
tp: 16
|
||
nodes: 2 # → must live on a single fabric
|
||
gpus_per_node: 8
|
||
priority: 100
|
||
weight: 100
|
||
startup_timeout: 40m
|
||
```
|
||
|
||
### 9.2 agent.yaml
|
||
```yaml
|
||
node_id: gpu-a-01
|
||
server: "llmgw-server.internal:7443"
|
||
mgmt_interface: eth0
|
||
tls: { ca: ..., cert: ..., key: ... }
|
||
labels: { rack: r12 }
|
||
fabrics:
|
||
- name: ib-a
|
||
hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
|
||
port_range: [20000, 20999] # engine HTTP and dist-init ports
|
||
engines:
|
||
vllm: { bin: /opt/venvs/vllm/bin/vllm, launch_mode: ray }
|
||
sglang: { python: /opt/venvs/sglang/bin/python }
|
||
work_dir: /var/lib/llmgw-agent
|
||
```
|
||
|
||
---
|
||
|
||
## 10. Benchmarking and Weighting
|
||
|
||
`llmgw ctl bench <model> --profile <p>`:
|
||
- Drives a target replica at increasing concurrency with fixed prompt and output lengths.
|
||
- Measures TTFT p50/p95, TPOT, throughput (tok/s) and the maximum concurrency that still meets the SLO.
|
||
- The output is the `CapacityProfile` from §6.2.1: `MaxConcurrencyAtSLO`, throughput, TTFT/TPOT p99.
|
||
- `--apply` updates three things at once: the profile's `weight`, `target_waiting_per_replica`,
|
||
and the autoscaling target.
|
||
- This is the only sound way to compare GPU classes; never use an estimated weight.
|
||
- `--sweep-prefix` mode measures behaviour at 0/50/90% prefix cache hit rates, so routing
|
||
profile weights (§7.3) can be tuned from measurements instead of guesses.
|
||
- A profile goes stale when the engine version changes, so every `ctl bench` result is tagged
|
||
with the engine version.
|
||
|
||
---
|
||
|
||
## 11. Roadmap
|
||
|
||
`[x]` is built and shown to work, `[ ]` is not built, and `[~]` is built
|
||
but not yet run where it matters — a real engine, real hardware.
|
||
|
||
### Phase 0 — Skeleton (1 week)
|
||
- [x] `go mod init`, cobra commands, koanf config, slog, Makefile, golangci-lint
|
||
- [x] `sim-engine`: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format `/metrics`)
|
||
- [x] CI: `go test ./...`, race detector
|
||
|
||
### Phase 1 — Static router (2 weeks)
|
||
- [x] OpenAI handlers, streaming proxy, cancellation propagation
|
||
- [x] Static endpoint list from configuration
|
||
- [x] **The `Filter`/`Scorer`/`Picker` pipeline (§7.3)** — a plugin pipeline from the start, not one "strategy"
|
||
- [x] `queue`, `kv`, `inflight` scorers; `max-score` and `p2c` pickers
|
||
- [x] Session affinity (§7.5 tier 0) — cheap, an early win
|
||
- [x] Metrics scraping (the §7.6 contract), staleness detection, retry, cooldown, breaker
|
||
- [x] Prometheus metrics plus decision metrics (§8) — `llmgw_requests_total`,
|
||
`llmgw_ttft_seconds`, `llmgw_scorer_value`, `llmgw_candidates_considered`,
|
||
`llmgw_candidates_filtered_total`, `llmgw_tokens_total`
|
||
- **Exit criterion:** 1000 concurrent streams behind 3 sim-engines; upstream cancellation on
|
||
client disconnect verified; router overhead p99 < 2 ms.
|
||
|
||
### Phase 1.5 — Admission control (1 week) ⭐
|
||
- [x] Saturation detector (§7.2.1)
|
||
- [x] Priority bands, round-robin tenant fairness within a band, queue capacity and TTL
|
||
- [x] `Admission` interface and lease, cancellation while queued
|
||
- [x] `llmgw_admission_*` metrics
|
||
- **Exit criterion:** while 500 concurrent batch requests flow through the bronze band, the TTFT
|
||
p99 of interactive requests in the platinum band stays under 1.5× its idle value. **Do not
|
||
move on before writing this test — it is the entire reason phase 1.5 exists.**
|
||
|
||
### Phase 2 — Agent: discovery and single-node engines (2 weeks)
|
||
- [x] GPU discovery (nvidia-smi path), IB discovery from sysfs
|
||
- [x] Engine interface, vLLM and SGLang command and environment construction
|
||
- [x] llama.cpp adapter, and with it CPU profiles: a profile may omit its GPU class,
|
||
and only an engine that can serve without a card may do so. This is what
|
||
makes the whole path testable on machines that have no GPUs — verified
|
||
end to end on four VMs serving Qwen2.5-0.5B on CPU at ~50 tok/s, twelve
|
||
concurrent requests balancing 4/4/4 across three replicas.
|
||
- [x] Supervisor (process groups, log buffer, state machine, orphan adoption)
|
||
- [x] NVML backend behind the `nvml` build tag, with NVLink topology. It compiles
|
||
in CI but has not run against a driver: **verify on a GPU node before
|
||
relying on the NVLink domains it reports.**
|
||
- [x] connect-go stream, register/heartbeat, shard status
|
||
- [x] mTLS: a private CA (`llmgw ctl cert`), certificates on both ends, and the
|
||
control plane refusing any agent whose claimed node id does not match its
|
||
certificate. Plaintext HTTP/2 remains the default and still must stay on a
|
||
trusted network; `agent_tls`/`tls` in the configuration turns it on.
|
||
- **Exit criterion (met, via configuration rather than `ctl`):** a model starts
|
||
and stops on a node. Demonstrated with real binaries: `llmgw server` and
|
||
`llmgw agent` as separate processes, the agent registering, a replica being
|
||
placed, an engine launched through the vLLM adapter, traffic served, and the
|
||
node's loss detected when the agent is killed. `ctl` itself is phase 3's
|
||
remaining item.
|
||
|
||
### Phase 3 — Control plane (2–3 weeks)
|
||
- [x] `Register` carries the agent's running shards, so a control plane restart
|
||
rebuilds replica state from the fleet instead of orphaning live engines (§6.5)
|
||
- [x] Orphan adoption: an agent takes back engines a previous agent left running,
|
||
and kills whatever it cannot identify or the catalog will not claim
|
||
- [x] Inventory: nodes, hardware, fabric attachments, allocations
|
||
- [x] Placement: profile priority, bin packing, NVLink-aware GPU selection,
|
||
`explain-placement` rejection reasons, property-based invariant tests
|
||
- [x] Catalog: models, profiles, alias and fallback resolution, validation
|
||
- [x] Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
|
||
- [x] `EndpointSource` wired from the reconciler to the router
|
||
- [x] `ctl`: nodes, models, replicas, cordon, uncordon, drain, explain-placement
|
||
- [x] `ctl logs`: a request/response pair on the agent stream (LogsRequest,
|
||
LogsReply) reads a node's agent log and any shard's engine output on
|
||
demand, from the CLI and from the node's page under /admin.
|
||
- **Exit criterion (met against a simulated agent fleet):** shutting down a node
|
||
moves its replicas elsewhere while the router keeps serving. Measured in
|
||
`test/e2e`: a node hosting a live replica is killed mid-traffic; the replica is
|
||
rebuilt on a survivor and ~370 requests complete with zero failures.
|
||
- **Since verified against real agents and engines**, which turned up what the
|
||
simulated fleet could not: replica ids were reused after a control plane
|
||
restart, so a restarted control plane collided with engines still running from
|
||
its previous life and every placement landed on a port an orphan was holding.
|
||
Ids now carry random bytes, adopted state wins over merely-placed state, and a
|
||
replica that never comes up is failed rather than left in `starting` forever.
|
||
|
||
### Phase 4 — Multi-node and fabrics (2 weeks)
|
||
- [x] Fabric identity from the agent's explicit labels
|
||
- [x] Multi-node placement (the fabric constraint), port allocation, rank ordering
|
||
- [x] NCCL environment generation, HCA selection
|
||
- [x] Replica-level failure handling (one dead shard kills the replica)
|
||
|
||
The launch path is now built too:
|
||
|
||
- [x] **Only the leader is health-checked.** `Engine.ServesHTTP(spec)` says which
|
||
ranks answer the API; a worker is judged by staying alive, and the
|
||
replica's real readiness is the leader answering, which only happens once
|
||
every worker has joined. Health-checking a worker's port used to time out
|
||
and fail a replica that was coming up perfectly well.
|
||
- [x] **A shard can be several processes.** `Engine.Commands(spec)` returns them
|
||
in start order: every command but the last is a sidecar kept alive for the
|
||
shard's lifetime and torn down with it, and the last one's exit is the
|
||
shard's. vLLM under Ray is the reason — the leader runs a Ray head and
|
||
then the server, a worker runs only the join and no vLLM process at all.
|
||
- [x] **The leader starts first.** Both engines put the rendezvous point on rank
|
||
0, so a worker started first has nothing to connect to. An earlier version
|
||
had this backwards; vLLM's server blocks until the Ray cluster has the
|
||
GPUs it needs, so the leader can start immediately and waits on its own.
|
||
- [x] **Rendezvous is waited for, not assumed.** A command can require that
|
||
`LeaderAddr:LeaderPort` accepts a connection before the next one starts,
|
||
bounded by `rendezvous_timeout` so a head that never appears fails the
|
||
shard instead of hanging it.
|
||
|
||
- [x] **Verified against a simulated cluster**, which is not the same as real
|
||
hardware and covers more than stub processes did. Four nodes across two
|
||
fabrics, eight GPUs each; `sim` is a real engine adapter whose leader
|
||
binds a rendezvous port and does not serve until every worker has joined,
|
||
and whose workers serve nothing at all.
|
||
|
||
What that proves: placement honours the fabric constraint, the leader
|
||
starts first and workers wait for its port, only rank 0 is health-checked,
|
||
a 2-node replica comes up on each fabric, and losing half of one fails the
|
||
replica and rebuilds it **within a single fabric** rather than spanning
|
||
two.
|
||
|
||
What it does not prove: that a given vLLM version wants `ray` or `mp`,
|
||
that NCCL picks the right HCA, or that the generated environment makes a
|
||
real collective run. Those need hardware. The GPUs here are an inventory,
|
||
the fabric is a sysfs-shaped directory, and the engine speaks no NCCL.
|
||
- [x] `ctl engine-check`, which asks an engine whether it still accepts the
|
||
flags llmgw builds. No GPU needed — a CLI prints its help anywhere, and
|
||
vLLM prints its own once told to assume a device.
|
||
- [x] **Checked against real engines.** vLLM 0.29.0: all five `vllm serve` flags
|
||
accepted (`--host`, `--port`, `--tensor-parallel-size`,
|
||
`--served-model-name`, `--distributed-executor-backend`). llama.cpp
|
||
b10991: all seven accepted.
|
||
- [ ] **The Ray sidecar is still unchecked.** vLLM 0.29 does not install Ray, so
|
||
`ray start --head --block --port --num-gpus` has never been run against
|
||
Ray's own help. `launch_mode: mp` avoids it entirely and is worth
|
||
preferring until someone checks.
|
||
- [ ] **SGLang is unchecked**, for the same reason as Ray: not installed
|
||
anywhere yet.
|
||
|
||
- **Exit criterion (met on the simulated cluster):** a 2-node replica on each
|
||
fabric; killing a node rebuilds the replica within one fabric and never
|
||
spreads it across two. Measured: node02 lost at t+60s, the ib-a replica failed
|
||
and was reaped, a replacement was ready at t+80s on ib-b — the fabric that
|
||
could still host it — while node01 sat idle rather than being paired across
|
||
fabrics with a node from ib-b.
|
||
|
||
### Phase 5 — Production features (3 weeks)
|
||
- [x] **SQLite store and migrations**, behind an interface, with per-dialect
|
||
migration directories so Postgres is a driver swap rather than a rewrite.
|
||
It arrived here rather than in phase 3 because everything that genuinely
|
||
needs persisting is a phase 5 feature; writing it earlier would have meant
|
||
designing a schema against guesses (§6.5).
|
||
- [x] Tenancy: org/team/project/key hierarchy, key storage and authentication,
|
||
limit resolution (most restrictive along the chain), `llmgw keys`
|
||
- [x] Budgets, RPM/TPM enforcement and the usage batch writer (§7.9.3, §7.9.4).
|
||
Rate limits are a sliding window, not a fixed one: a fixed window lets a
|
||
caller send its whole allowance at the end of one window and again at the
|
||
start of the next, so the observed rate doubles exactly when the system is
|
||
least able to take it.
|
||
- [x] Accounts: signup, operator approval, sessions, password changes, and a
|
||
profile page where a person issues and revokes their own keys. Approval stays on the CLI —
|
||
having a shell on the control plane is what makes someone an
|
||
administrator, and there are no roles to get wrong.
|
||
- [x] Onboarding: one-time claim links (the key is sealed with the link's own
|
||
token, which is never stored), `GET /me` for a user's own limits and
|
||
spend, and `docs/onboarding.md`. Sending the key itself was the weak link
|
||
— a key pasted into a chat window stays there forever.
|
||
- [x] Model allowlists and key→band mapping. With auth on, the band comes from
|
||
the key and the header is ignored — it was trusted only while there was
|
||
no identity to consult, which let any client promote itself.
|
||
- [x] Budget fallback
|
||
- [x] Routing profiles from configuration plus the `x-llmgw-profile` header
|
||
- [x] Approximate radix tree (§7.5 tier 1), with its thresholds and the
|
||
`llmgw_prefix_decisions_total` breakdown. Measured against sim-engines
|
||
whose caches were too small for the working set: 22.5% → 81.7% cache
|
||
hits. It does nothing when the working set fits in one replica's cache,
|
||
which is worth knowing rather than assuming.
|
||
- [x] Fallback models
|
||
- [x] Pool spillover: a saturated pool passes requests to the next one that
|
||
is not, by admission's own measure of saturation
|
||
- [x] `ctl bench`: a load ramp that reports capacity as the last concurrency
|
||
level meeting the SLO, plus the queue depth and KV usage observed there,
|
||
so `weight`, `target_waiting` and `kv_ceiling` stop being guesses
|
||
- [x] Sizing from stored capacity: `ctl model target <model> --tokens-per-sec N`,
|
||
and the autoscaler and profile choice use what `ctl bench --save` measured
|
||
- [x] OTel tracing: an inference span with admission, schedule and each
|
||
upstream attempt under it, over OTLP/HTTP when `tracing.endpoint` is
|
||
set, and the traceparent carried on to the engine
|
||
- [x] systemd units and `docs/onboarding.md`
|
||
- [x] Release binaries: static linux/amd64, linux/arm64 and darwin builds
|
||
attached to each tag, so installing on a node does not need a Go toolchain
|
||
- [x] Reference documentation beyond the plan itself: `docs/`, with
|
||
`docs/reference.md` generated from the configuration structs
|
||
|
||
### Phase 6 — v2 ideas
|
||
- [~] Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration.
|
||
Built in v0.11.2 (`kv_events` on a profile, the agent forwarding block
|
||
changes, the router's index and scorer); never run against a real vLLM,
|
||
which is what is left (BACKLOG C24)
|
||
- [ ] Dynamo-style block cost function, tuning the `softmax` picker
|
||
- [x] Admission displacement: a total queue bound, and the newest request of
|
||
the lowest band turned away (503, Retry-After) for a higher one
|
||
- [x] Autoscaling on load: a replica added while requests wait, up to a
|
||
model's max, and taken back after ten quiet minutes
|
||
- [x] Priority-based preemption: capacity classes (critical, standard,
|
||
opportunistic), preemption tried on a copy of the inventory first
|
||
- [x] Capacity quotas per organization, team and project: guarantees kept,
|
||
borrowing taken back, limits enforced
|
||
- [x] CPU cores reserved like memory, and memory required of CPU models
|
||
- [x] Node sensors collected: GPU temperature, power and throttling, CPU
|
||
temperature and RAPL power, board inlet air; published and shown
|
||
- [~] Rack power, temperature and cooling as placement signals: planned in
|
||
[docs/facility-aware-placement.md](docs/facility-aware-placement.md).
|
||
Built in v0.11.2: location labels, spreading across racks and PDUs,
|
||
`keep_within`, held domains, power caps from `facility.budgets`. Left:
|
||
temperature and cooling, which need each site's sensor data
|
||
- [ ] Defragmentation for large tensor-parallel replicas across NVLink domains
|
||
- [x] Control plane HA, by leader election on PostgreSQL (v0.8.0) and from
|
||
setup (v0.11.0) rather than by Raft: the store is replicated already, so
|
||
a second consensus layer would only have to agree with it
|
||
- [x] Weight downloading and cache management: the mirror, agent fetches
|
||
checked against their digest, and reclaiming what nothing uses
|
||
- [ ] Prefill/decode disaggregation (§12)
|
||
- [x] LoRA adapters on llama.cpp: every replica loads a model's adapters and
|
||
applies one per request, so any replica serves any adapter
|
||
- [~] LoRA affinity for engines that load adapters on demand (vLLM): built in
|
||
v0.11.2 (`max_loaded_adapters`, the router's adapter tracker, the
|
||
affinity filter); never run against a real vLLM
|
||
- [x] External provider support: a catalog model served by an OpenAI-compatible
|
||
API elsewhere, usable as a fallback, under the same allowlists and budgets
|
||
- [x] Read-only dashboard on the admin listener (nodes, replicas, models,
|
||
placement failures, usage) and a public landing page
|
||
- [x] `/chat`: an in-browser playground for signed-in people, dispatched through
|
||
the ordinary request path so every limit still applies and the browser
|
||
never holds a key
|
||
- [x] `/admin/users`: approve, suspend and reinstate accounts from the browser;
|
||
promoting, resetting passwords and issuing keys stay on the CLI
|
||
- [x] The dashboard reachable over the web at `/admin`, behind a session and an
|
||
admin flag, without exposing the listener that carries pprof
|
||
- [x] An admin UI that can *change* things: staff roles (viewer, operator,
|
||
billing, admin) and the audit log in v0.11.2, the cluster's settings on
|
||
the portal in v0.14.1
|
||
|
||
---
|
||
|
||
## 12. Prefill/Decode Disaggregation (v2 notes)
|
||
- Prefill and decode replicas must live on the **same fabric** (KV transfer goes over RDMA).
|
||
- A two-stage flow in the router: send to a prefill replica → KV handle → decode replica.
|
||
- Engine support varies sharply by version (vLLM KV connectors, SGLang PD mode, NVIDIA Dynamo/NIXL). Wrapping the engine's own built-in PD router as a single "endpoint" is less risky than implementing this ourselves.
|
||
- What llm-d and Dynamo show: PD's benefit lies mainly in **long context**. For short-prompt chat
|
||
traffic the KV transfer cost eats the gain. Measure your workload's ISL/OSL distribution with
|
||
the benchmark in §10 first, and enable PD only for long-context models.
|
||
- The scheduler gains a `Role: prefill|decode` field and a "paired placement on the same fabric" constraint.
|
||
|
||
---
|
||
|
||
## 13. Test Strategy
|
||
- **Unit:** the scheduler (table-driven: heterogeneous inventory plus expected placement or failure reason), balancer scores, SSE parsing, metric parsing (fixtures taken from real vLLM/SGLang `/metrics` output).
|
||
- **Property-based:** the invariants "a multi-node replica lives on one fabric" and "one GPU class per replica" always hold in scheduler output (`testing/quick` or `rapid`).
|
||
- **Integration:** server plus N fake agents (fake GPU/IB inventory) plus sim-engines, in one process.
|
||
- **Chaos:** kill agents, break the network (toxiproxy), crash engines, slow engines, and kill
|
||
the control plane to verify the router keeps serving from its cache.
|
||
- **Priority isolation:** phase 1.5's exit criterion as a permanent regression test — high-priority
|
||
TTFT is preserved under low-priority load. This test must stay in CI.
|
||
- **Routing quality:** give sim-engine a realistic prefix cache model (block-level LRU) and
|
||
measure the hit rate against synthetic multi-turn traffic. The comparison between round robin,
|
||
session affinity and the approximate tree must be numeric.
|
||
- **Load:** `k6` or our own Go load generator; measure the router's own overhead (target: p99 under 2 ms of added latency).
|
||
- **Real hardware:** at the end of every phase, a smoke test on at least one node with IB and one without.
|
||
|
||
---
|
||
|
||
## 14. Risks and Open Questions
|
||
| Topic | Note |
|
||
|---|---|
|
||
| Engine version differences | CLI flags and metric names change often → make engine adapters version-aware and overridable from configuration. |
|
||
| vLLM multi-node launch | Ray or built-in multiprocessing? Must be tested on the version actually deployed. |
|
||
| Fabric detection | Can a unique subnet prefix be set on each OpenSM? If not, manual labelling is mandatory. |
|
||
| Multiple fabrics per node | Does this occur? If so, HCA-to-GPU NUMA matching matters. |
|
||
| Model storage | Is there a shared filesystem, and what are the load times? Startup timeouts follow from this. |
|
||
| Mixed driver/CUDA versions | Should engine virtualenvs differ per GPU class? (The `engines` config must be overridable per profile.) |
|
||
| Single control plane | A SPOF in v1. The router caches the endpoint list (§6.4) and keeps working through a brief control plane outage. |
|
||
| Security | Engine ports must be reachable only by the router (firewall or bind address). |
|
||
| **Tokenizer consistency** | The precise KV index (§7.5 tier 2) forces the router to use **the same tokenizer** as the engine. In Go that means cgo or a separate tokenizer service, which is why it is deferred to v2; v1's prefix approach needs no tokenizer. The same limitation makes the TPM prompt estimate approximate in v1 (bytes/4). |
|
||
| **Prefix affinity backfiring** | Without the load-imbalance guard (the §7.5 tier 1 thresholds), prefix routing raises the cache hit rate while drowning a single replica and wrecking p99. Prefix affinity **must not** be enabled before those thresholds are. |
|
||
| **Tuning the admission thresholds** | Too low a `target_waiting_per_replica` starves the GPU; too high brings queueing delay back. This single number governs the system's behaviour, so measure it with `ctl bench` rather than guessing. |
|
||
| KV event loss | ZMQ is fire-and-forget; a lost event makes the router silently misroute. Sequence-gap detection and falling back to tier 1 are mandatory (§7.5). |
|
||
| Metric staleness | At 1000 requests/sec with a 2s scrape interval, decisions rest entirely on stale data. That is why the `inflight` scorer is not optional. |
|
||
| **Usage lost on a hard kill** | Records buffered for the batch writer are lost if the process is killed — at most a few hundred milliseconds of accounting. Acceptable for budgets; **not acceptable if this ever feeds external billing**, and the answer then is an append-only journal rather than a synchronous insert (§7.9.3). |
|
||
| **Rate limits with several routers** | Counters are per instance, so N instances allow N times the configured limit. v1 is single-instance by design; Redis or consistent-hash stickiness is the fix (§7.9.6). |
|
||
| **Control plane restart** | Until `Register` carries running shards, restarting the control plane orphans live engines: they keep holding GPUs the inventory believes are free, and the next placement fails to allocate with an error pointing at the wrong process (§6.5). |
|
||
|
||
---
|
||
|
||
## 15. First Steps
|
||
1. `mkdir llmgw && cd llmgw && go mod init <module-path>/llmgw`
|
||
2. `cmd/llmgw/main.go`: an empty cobra skeleton with the `server`, `agent`, `ctl` and `sim-engine` subcommands.
|
||
3. `internal/sim`: the fake engine — make this the first real code, because everything else is tested against it.
|
||
4. `internal/router/proxy`: streaming proxy plus a cancellation test.
|
||
5. `internal/router/scheduler`: the `Filter`/`Scorer`/`Picker` interfaces plus the `queue`, `kv`
|
||
and `inflight` scorers. **Do not write a single "least-load" function** — build the pipeline
|
||
from the start; coming back to it later is expensive.
|
||
6. `internal/router/admission`: saturation detector and band queues. Phase 1.5.
|
||
7. Write the end-to-end tests for the phase 1 and phase 1.5 exit criteria, then move to phase 2.
|
||
|
||
> Use `internal/router/scheduler/` rather than `internal/router/balancer/` (the tree in §4
|
||
> reflects this): "balancer" understates the job — this component does more than balance load.
|