llmgw/llmgw-plan.md

llmgw — A Single-Binary LLM Gateway for Multi-Fabric InfiniBand and Heterogeneous GPUs

Status: Design / Plan v0.2 What changed in v0.2: existing LLM gateway and router projects were surveyed (§2.5) and their critical designs folded in — above all separating admission from scheduling (§7.2), the plugin-based filter → score → pick pipeline (§7.3), and the three-tier model of prefix/KV cache awareness (§7.5). Goal: one Go binary (running under different roles) that serves the LLMs on our own GPU servers behind a single OpenAI-compatible endpoint. Behind it sit several independent InfiniBand fabrics, nodes without IB, and different GPU types.

---

1. Scope

1.1 In scope

• OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models), including SSE streaming.
• An agent on every GPU node: GPU and IB discovery; starting, supervising and stopping inference engine processes (vLLM / SGLang).
• Control plane: node inventory, model catalog, replica placement, state tracking.
• Router, in two layers: admission (priority bands, tenant fairness, queueing under saturation) and scheduling (plugin-based filter/score/pick with queue, KV-cache and prefix awareness), plus fallback, retry, cooldown and circuit breaking.
• Fabric awareness: a multi-node replica never spans two fabrics; a TP group never mixes GPU types.
• API keys, rate limits, token accounting and budgets, Prometheus metrics. Persistence for these is the only thing the store exists for (§6.5, §7.9).

1.2 Out of scope (at least for v1)

• Inference itself. Go only orchestrates and proxies; vLLM/SGLang run the model.
• A Kubernetes operator (bare metal / systemd is the target; K8s can come later).
• Prefill/decode disaggregation (a v2 goal, see §12).
• External providers (OpenAI, Anthropic) — the architecture stays open to them, but not in v1.

---

2. Terminology

Term	Meaning
Node	A physical server running the agent.
Fabric / Island	An independent IB fabric. Nodes without IB belong to the none island and each counts as its own single-node island.
GPU Class	A group of GPUs with the same model and VRAM (e.g. h100-80g, a100-40g, l40s-45g).
Pool	A (fabric, gpu class) pair. The unit of placement.
Model	A logical model in the catalog (e.g. llama-3.3-70b). This is the name users see.
Profile	How a model runs on a particular GPU class (TP, PP, quantization, max length, engine arguments).
Replica	One running copy of a model. May span 1..N nodes. This is the unit the router sees.
Shard	A replica's part on a single node (one engine process).
Routing Profile	The scorer set and weights the router uses for a class of request (chat / batch / long-context). Not to be confused with the model Profile above.
Band	A priority class (platinum/gold/silver/bronze). The unit of admission decisions.
Block	The unit of KV cache accounting; a fixed number of tokens (16 in vLLM, 64 by default in the GIE indexer).
Unit	The billing unit budgets are counted in. Self-hosted models have no per-token price, so an operator defines what a unit is: GPU-seconds, a notional rate per million tokens, or 1.0 everywhere, which turns budgets into token quotas (§7.9.4).

---

2.5 Reference Implementations and What We Took

Rather than design from scratch, we inherited the problems already solved by mature projects in this space.

Project	What it is	What llmgw takes from it
Gateway API Inference Extension (EPP)	The K8s standard; an Endpoint Picker running over Envoy ext_proc	Admission/scheduling separation, the plugin-based filter → score → pick pipeline, the model server metric contract (§7.6)
llm-d	"Well-lit paths" built on GIE	KV-event-based precise prefix index, priority bands with tenant fairness, the saturation detector
AIBrix (a vLLM project)	An Envoy ext_proc gateway plus a heterogeneous GPU optimizer	The heterogeneous GPU SLO/cost model (§6.2.1), queueing at the gateway (slo-least-load-pulling), disabling prefix affinity under load imbalance
NVIDIA Dynamo	Radix-tree KV router plus SLA planner	The shape of the cost function (§7.4) and router_temperature softmax sampling against herding
SGLang router	Cache-aware load balancing over an approximate radix tree	v1's prefix answer: approximate tree plus threshold logic (§7.5 tier 1)
vLLM production-stack / vllm-project/router	vLLM's own routers	Session affinity (measured 96% vs 93% cache hit), consistent hashing
LiteLLM	The most widely deployed multi-provider proxy	Virtual key → team/project → budget hierarchy, cooldowns, keeping spend writes off the request path, budget-triggered fallback
Envoy AI Gateway	An Envoy-based AI gateway (v1.0, June 2026)	Token-based (not merely request-based) rate limiting

What we deliberately did not take: a Kubernetes dependency (bare metal is the target), the Envoy ext_proc architecture (single binary is the goal — we write the router directly in Go), semantic routing / LLM-based model selection (out of scope), and external provider management (not in v1).

---

3. Architecture

                    ┌───────────────────────────────────────────┐
  Clients ────────► │  llmgw server                             │
  (OpenAI API)      │  ┌────────────┐  ┌──────────────────────┐ │
                    │  │  Router    │  │  Control Plane       │ │
                    │  │ (data)     │◄─┤  - Inventory         │ │
                    │  │            │  │  - Catalog           │ │
                    │  └─────┬──────┘  │  - Scheduler         │ │
                    │        │         │  - Store (phase 5)   │ │
                    │        │         └─────────▲────────────┘ │
                    └────────┼───────────────────┼──────────────┘
                 HTTP (eth)  │                   │ gRPC stream (mTLS, agent dials out)
        ┌────────────────────┼───────────────────┼─────────────────────┐
        ▼                    ▼                   │                     ▼
  ┌───────────┐        ┌───────────┐       ┌───────────┐         ┌───────────┐
  │ llmgw     │  IB-A  │ llmgw     │       │ llmgw     │  IB-B   │ llmgw     │
  │ agent     │◄──────►│ agent     │       │ agent     │◄───────►│ agent     │  ...
  │ vLLM rank0│  NCCL  │ vLLM rank1│       │ SGLang    │         │ (no IB)   │
  └───────────┘        └───────────┘       └───────────┘         └───────────┘
      Fabric A island                        Fabric B island       none island

3.1 One binary, several roles

llmgw server   --config server.yaml     # router + control plane (same process in v1)
llmgw agent    --config agent.yaml      # on every GPU node
llmgw ctl ...                           # CLI: list nodes, deploy models, drain, ...
llmgw sim-engine                        # fake OpenAI-compatible engine for testing

router and control must eventually be able to run as separate processes (llmgw server --roles=router), so the dependency between them goes through an interface (see §6.4).

3.2 Build note

go-nvml requires cgo. Put the agent's NVML code behind a //go:build nvml tag and use a fake / nvidia-smi-based discovery path in untagged builds. That keeps the server side cgo-free and lets development machines without GPUs run the tests.

---

4. Project Layout

llmgw/
├── cmd/llmgw/main.go              # cobra root; subcommands
├── api/
│   └── proto/llmgw/v1/*.proto     # agent <-> control plane
├── internal/
│   ├── config/                    # YAML + env via koanf
│   ├── agent/
│   │   ├── agent.go               # main loop, control plane connection
│   │   ├── discovery/
│   │   │   ├── gpu_nvml.go        # //go:build nvml
│   │   │   ├── gpu_fallback.go
│   │   │   ├── ib.go              # reads /sys/class/infiniband
│   │   │   └── topology.go        # NVLink, PCIe, NUMA, GPU<->HCA affinity
│   │   ├── supervisor/            # engine process management (exec, restart, logs)
│   │   ├── engines/
│   │   │   ├── engine.go          # Engine interface
│   │   │   ├── vllm.go
│   │   │   └── sglang.go
│   │   └── health/                # polls engine /health and /metrics
│   ├── control/
│   │   ├── inventory/             # node and GPU state
│   │   ├── catalog/               # model and profile definitions
│   │   ├── scheduler/             # placement algorithm
│   │   ├── reconciler/            # desired vs actual loop
│   │   └── agentsvc/              # gRPC server
│   ├── router/                    # HTTP handlers, dispatch, metrics
│   │   ├── proxy/                 # streaming proxy to upstream
│   │   ├── admission/             # saturation detector, band queues (§7.2)
│   │   ├── scheduler/             # Filter/Scorer/Picker pipeline (§7.3)
│   │   ├── prefix/                # tier 0 session hash, tier 1 approx tree, tier 2 kv index
│   │   ├── endpoint/              # live endpoint state, inflight, staleness
│   │   ├── breaker/               # circuit breaker + cooldown
│   │   └── scrape/                # reads engine /metrics (the §7.6 contract)
│   ├── auth/                      # API keys, org/team/project, budgets
│   ├── ratelimit/                 # token bucket (requests + tokens)
│   ├── usage/                     # token counting and recording
│   ├── store/                     # SQLite (modernc.org/sqlite), migrations
│   ├── obs/                       # slog, prometheus, otel
│   └── sim/                       # fake engine
├── deploy/
│   ├── systemd/
│   └── examples/
├── test/e2e/
└── docs/

The OpenAI handlers live in internal/router itself rather than an openai/ subpackage: they need the dispatch loop, and splitting them apart would have meant an interface whose only implementation is the router.

4.1 Libraries

Need	Choice
CLI	github.com/spf13/cobra
Config	github.com/knadh/koanf/v2
RPC	connectrpc.com/connect (gRPC compatible, HTTP/2, easy to debug)
GPU	github.com/NVIDIA/go-nvml
DB	modernc.org/sqlite (cgo-free) + pressly/goose migrations
Metrics	github.com/prometheus/client_golang, parsing via github.com/prometheus/common/expfmt
Tracing	go.opentelemetry.io/otel
Logging	log/slog (stdlib)
Rate limiting	golang.org/x/time/rate (plus our own bucket for token limits)
Hashing	github.com/cespare/xxhash/v2
HA (v2)	github.com/hashicorp/raft or etcd
Testing	stdlib testing, github.com/stretchr/testify (optional)

---

5. Agent

5.1 GPU discovery

For every GPU:

• index, UUID, name, VRAM, compute capability (for FP8 and similar support), driver/CUDA version
• NUMA node, PCIe bus id
• NVLink peers (NVML GetNvLinkRemotePciInfo) → which GPUs pair efficiently for TP within a node
• health: XID errors, ECC, temperature, memory in use (are there non-engine processes?)

GPUClass is derived automatically: normalize(name) + "-" + round(vram_gib) + "g" → h100-80g. It can be overridden in configuration.

The size comes from what the driver reports, in GiB, which does not always match the marketing name: an L40S is sold as 48GB and reports 44.99 GiB, so its class is l40s-45g. Consistency matters more than matching the box here, because the number is what decides whether a model fits. The model token is kept in full for the same reason — a 4090 and a 4080 are not interchangeable, so the class is 4090-24g, not rtx-24g.

5.2 IB discovery and fabric identity

From /sys/class/infiniband/<hca>/ports/<n>/:

• state (ACTIVE?), phys_state, rate, link_layer (InfiniBand / Ethernet → RoCE)
• gids/0 → subnet prefix
• sm_lid, lid
• the HCA's PCIe address and NUMA node → GPU affinity

How is fabric identity determined? Automatic detection is not reliable, because OpenSM's default subnet prefix (fe80::) is the same on every fabric. In order of preference:

1. An explicit label in the agent config: fabrics: [{name: ib-a, hcas: [mlx5_0, mlx5_1]}] (recommended)
2. If every OpenSM has a unique subnet_prefix, automatic mapping from the GID prefix (a prefix → fabric table in the server config)
3. Otherwise the node is marked fabric=unknown and excluded from multi-node placement.

Bonus verification (v1.1): for two nodes claiming the same fabric, the agents verify reachability with a short ibv_rc_pingpong-style test or ibping. A mismatch raises an alert.

A node may have HCAs attached to more than one fabric, and the model must allow it (Node.Fabrics []FabricAttachment). A replica, however, picks exactly one fabric and uses only that fabric's HCAs.

5.3 Engine interface

type Engine interface {
    Name() string                                   // "vllm", "sglang"
    Validate(p Profile, gpus []GPU) error           // compute capability, quantization fit, ...
    BuildCommand(spec ShardSpec) (*exec.Cmd, error) // builds arguments and environment
    HealthURL(spec ShardSpec) string
    MetricsURL(spec ShardSpec) string
    ParseMetrics(r io.Reader) (EngineStats, error)  // engine-specific metric names
}

type ShardSpec struct {
    ReplicaID   string
    Rank        int      // 0 = leader (the one serving HTTP)
    NNodes      int
    LeaderAddr  string   // dist init address (ethernet IP)
    LeaderPort  int
    GPUIndices  []int
    HCAs        []string // HCAs of the chosen fabric
    HTTPPort    int
    Profile     Profile
    ModelPath   string
}

type EngineStats struct {
    // The mandatory contract from §7.6 — without these the router cannot route.
    Running, Waiting int
    KVCacheUsage     float64 // 0..1   (vllm:kv_cache_usage_perc)
    // Optional.
    BlockSize, NumGPUBlocks int
    PrefixHitRate          float64
    TTFTp50, TPOTp50       time.Duration // from histograms
    DrainRate              float64       // observed completions/sec; for the `load` scorer
    LoRA                   LoRAState     // v2
    ScrapedAt              time.Time     // for staleness detection — the router uses this
}

Supporting a new engine is defined as: being able to fill in the mandatory three from §7.6.

Multi-node launch:

• SGLang supports it natively: --nnodes N --node-rank R --dist-init-addr host:port --tp T.
• vLLM: depending on the version, either Ray (ray start --head on the leader, ray start --address on the workers) or the newer multi-node multiprocessing flags. Check the documentation for the vLLM version you deploy; vllm.go should offer both paths behind a launchMode.

Environment generation (critical):

CUDA_VISIBLE_DEVICES=<GPUIndices>
NCCL_IB_HCA=<HCAs, comma separated>    # the chosen fabric only
NCCL_SOCKET_IFNAME=<mgmt interface>     # bootstrap
NCCL_IB_DISABLE=1                       # single node, or no IB
NCCL_CROSS_NIC / NCCL_NET_GDR_LEVEL     # optional, from the profile
GLOO_SOCKET_IFNAME=<mgmt interface>

5.4 Supervisor

• Start processes in their own process group (SysProcAttr.Setpgid) and kill the whole group when stopping (SIGTERM → timeout → SIGKILL).
• stdout/stderr → line-oriented slog plus a ring buffer (the last N lines, for llmgw ctl logs).
• State machine: Pending → Starting → Loading → Ready → Draining → Stopped | Failed.
• Ready only once /health returns 200 and /v1/models lists the model.
• Model loading takes a long time; startupTimeout comes from the profile (e.g. 20 min).
• On a crash: report to the control plane and let the control plane decide about restarting. Restarting a single shard of a multi-node replica accomplishes nothing; the whole replica must be rebuilt.
• If the agent restarts, try to adopt still-running engine processes (pid file plus cmdline check) and clean up whatever cannot be adopted.

5.5 Model weights

• v1: a shared filesystem (NFS/Lustre/Weka) or a path pre-populated on the node. The agent only checks existence and size.
• v1.1: the agent downloads from HF/S3 into a local cache, reports progress, and enforces a disk quota.

5.6 Agent ↔ control plane protocol

The agent dials out (NAT- and firewall-friendly) over a bidirectional stream:

service AgentService {
  rpc Connect(stream AgentMessage) returns (stream ControlMessage);
}

message AgentMessage {
  oneof msg {
    Register     register  = 1; // node info, GPUs, fabrics, versions
    Heartbeat    heartbeat = 2; // every 5s: GPU health, shard states
    ShardStatus  shard     = 3; // sent on every state change
    EngineStats  stats     = 4; // every 2s (the router may also scrape directly)
  }
}

message ControlMessage {
  oneof msg {
    StartShard   start = 1;
    StopShard    stop  = 2;
    DrainShard   drain = 3;
    Ack          ack   = 4;
  }
}

• Commands are idempotent: a repeated StartShard with the same shard_id is a no-op.
• mTLS, with node identity taken from the certificate CN. The control plane compares the CN against the node id the agent claims and refuses the stream when they disagree, which is what makes a node's identity something it proves rather than something it picks.
• The CA is internal, and has to be: public authorities stopped issuing the clientAuth usage, so no Let's Encrypt certificate can authenticate an agent however convenient it would be. llmgw ctl cert init-ca | issue-server | issue-client runs a small authority whose key lives on one machine and whose root is trusted everywhere.
• The server certificate must carry every address agents dial as a subject alternative name, bare IP addresses included. Hostname verification ignores the common name entirely, so a certificate that names the host only in its subject is rejected by every client.
• This covers the fleet-facing port only. The client-facing API is a public surface and is fronted by a publicly trusted certificate instead.
• Three missed heartbeats mark the node Unreachable; after 60s it becomes Lost and its replicas are rescheduled.

---

6. Control Plane

6.1 Data model (abridged)

type Node struct {
    ID, Hostname, MgmtAddr string
    Labels     map[string]string
    GPUs       []GPU
    Fabrics    []FabricAttachment // empty means no IB
    State      NodeState          // Ready, Cordoned, Draining, Unreachable, Lost
    LastSeen   time.Time
}

type FabricAttachment struct {
    Fabric string   // "ib-a"
    HCAs   []string
    Rate   int      // Gbps
}

type Model struct {
    Name      string            // what users see
    Aliases   []string
    Source    string            // path or HF id
    Profiles  []Profile
    Scaling   Scaling           // min/max replicas, target queue, ...
    Fallbacks []string          // other model names
}

type Profile struct {
    Name        string
    GPUClass    string          // "h100-80g"
    Engine      string          // "vllm"
    TP, PP      int
    GPUsPerNode int             // usually TP
    Nodes       int             // PP*... ; a fabric is mandatory when > 1
    Args        []string        // --quantization fp8 --max-model-len 32768 ...
    Env         map[string]string
    Weight      int             // capacity weight for the router (from benchmarks)
    Priority    int             // placement preference
    StartupTimeout time.Duration
}

type Replica struct {
    ID, Model, Profile string
    Fabric   string             // set for multi-node replicas
    Shards   []Shard
    State    ReplicaState
    Endpoint string             // rank0's HTTP address
}

6.2 Scheduler (placement)

Input: the desired replica count, the profile list, and the inventory.

Hard constraints:

1. All shards share one GPUClass.
2. When Nodes > 1, every node must attach to the same Fabric and have ACTIVE HCAs on it.
3. GPUsPerNode free GPUs, preferably in the same NVLink domain and on the same NUMA node as the HCA.
4. The node is Ready and not Cordoned.
5. The engine's Validate must pass (e.g. an FP8 profile never lands on a card with compute capability < 8.9).

Soft preferences (scored):

• the profile's Priority
• bin packing: fill partially used nodes first, keeping empty nodes free for large models
• spread a model's replicas across fabrics and nodes (failure domain distribution)
• GPU-to-HCA NUMA affinity

Algorithm (v1, simple and deterministic):

for each profile in model.Profiles sorted by Priority desc:
    pools := groupNodes(by GPUClass == profile.GPUClass, by Fabric if profile.Nodes > 1)
    for each pool sorted by score:
        nodes := pickNodes(pool, profile.Nodes, profile.GPUsPerNode)  // greedy + NVLink group selection
        if nodes != nil:
            return Placement{profile, pool.Fabric, nodes, gpuAssignments}
return ErrUnschedulable (with reasons — essential for debugging)

groupNodes: nodes without IB join the pool for Nodes == 1 profiles, and never for Nodes > 1.

llmgw ctl explain-placement <model> must show why each pool was rejected.

6.2.1 Heterogeneous GPUs: the SLO/capacity model

Comparing GPU classes through a single scalar "weight" is too crude; the model taken from AIBrix's heterogeneous GPU optimizer is more accurate. For each (model, profile) pair, ctl bench (§10) produces a capacity profile:

type CapacityProfile struct {
    Model, Profile string
    GPUClass       string
    // Maximum concurrency that still meets the SLO; the real capacity measure.
    MaxConcurrencyAtSLO int
    ThroughputTokPerSec float64
    TTFTp99, TPOTp99    time.Duration
    RelativeCost        float64   // relative GPU-hour cost, from configuration
    MeasuredAt          time.Time
    EngineVersion       string    // the profile expires when the version changes
}

An example SLO: TPOT p99 ≤ 50ms && TTFT p99 ≤ 2s.

The profile is used in three places:

1. Router weight: Profile.Weight is no longer hand-written; it derives from MaxConcurrencyAtSLO.
2. Admission thresholds: TargetWaitingPerReplica (§7.2.1) is suggested from the measurement.
3. Placement and autoscaling objective: "minimize total cost while meeting the SLO". For a predicted load, the replica count needed per GPU class is ceil(load / MaxConcurrencyAtSLO), and the cheapest combination wins. In v1 this only produces advice via ctl plan <model> --target-rps N; automatic application arrives with the v1.1 autoscaler.

When EngineVersion or the profile arguments change the measurement goes stale. llmgw must surface that (llmgw_capacity_profile_stale{model,profile}) rather than silently reusing an obsolete number.

6.3 Reconciler

• Every 5s, or on an event: compare desired (model scaling) with actual (replica states).
• Missing replica → schedule → send StartShard for each shard (workers first, then rank0, or whatever order the engine requires).
• Any shard in a replica Failed → stop the whole replica and reschedule with backoff.
• Excess replicas → Drain first (the router stops sending new requests, existing ones finish), then Stop.
• Node Lost → treat its replicas as Failed.

Autoscaling (v1.1): a per-model targetWaitingPerReplica, averaged from router metrics, with cooldown-guarded scaling. Shrinking lower-priority models when no GPUs are free (preemption) is v2.

6.4 Interface to the router

type EndpointSource interface {
    // model name -> Ready replicas (+ pool, weight, fabric, gpuclass, capacity profile)
    Endpoints(model string) []Endpoint
    Subscribe() <-chan struct{} // change notification
}

In-memory in v1; once the router and control plane split apart, the same interface is backed by a watch stream.

The router always caches this list: even if the control plane goes down it keeps routing to the replicas it knows about, which mitigates the single-point-of-failure risk in §14. When the cache ages past endpoint_cache_max_age (say 5 min) it only raises a warning metric; it never stops traffic.

6.5 What is stored, and what is not

v0.1 of this plan listed eight tables. Most of them were a mistake, and the reason is worth stating because it applies to every piece of state here:

A database that describes the cluster can be wrong. The cluster cannot.

An engine that crashes during a restart leaves a row saying "running". A node whose GPU developed an ECC fault leaves a row saying the card is fine. Every such row is a trap: the scheduler believes it, places work accordingly, and the failure surfaces somewhere else entirely. Rebuilding the same state from the fleet on every reconnect cannot go stale, because the fleet is the thing being described.

So:

State	Persisted	Where it actually comes from
Nodes, GPUs, fabrics	no	Every agent sends its full inventory on registration (§5.6)
Replicas, shards	no	Reported by agents; see the restart hole below
Models, profiles	no	The YAML catalog is the source of truth
API keys and the tenancy tree	yes	Created at runtime, secret, rotated (§7.9)
Usage records and rollups	yes	Accounting must survive a restart (§7.9)
Capacity profiles from ctl bench	yes	Measured, not declared (§6.2.1)
Audit events	yes	An audit trail that forgets is not one

Everything in the "yes" column belongs to tenancy and benchmarking, which is phase 5. The store lands there, not in phase 3, so its schema is shaped by real requirements instead of guesses.

The control plane restart hole

Not persisting replica state has one consequence that must be closed before it matters. If the control plane restarts while engines are running:

1. The engines keep serving — correct, and the reason the router talks to them directly.
2. Agents reconnect and re-register their hardware.
3. The control plane has no memory of any replica, because Register does not carry one.
4. The reconciler sees zero replicas and places new ones.
5. The old engines keep holding GPUs that the inventory now believes are free, so the new engines fail to allocate — with an error pointing at the new process rather than the old one.

The fix is on the agent side, not in a database: Register carries the shards the agent is currently supervising, and the control plane rebuilds replica state from what the fleet reports. The supervisor already knows its live shards and already finds, through pid files, the ones that survived an agent restart (FindOrphans). A shard nothing claims after one reconciliation pass is an orphan and is killed.

This is strictly better than a database would be: it cannot disagree with reality, and it covers agent restarts and control plane restarts with one mechanism.

Store choice

SQLite in WAL mode, via modernc.org/sqlite so the server stays cgo-free.

The deciding argument is not throughput. One row per request at 1000 req/s is 86 million rows a day, which sounds like a reason to reach for Postgres, but the write path is batched anyway (§7.9.3) and a transaction of 500 rows every 200ms is nothing for either engine. Batching is required for a different reason, and once it exists the volume argument favours neither.

What decides it is the failure mode. Postgres is a second service, and the question it raises is whether the gateway still serves traffic when that service is unreachable. An inference gateway that stops serving because its accounting database is down is worse than one that buffers some accounting. SQLite is embedded, so the store shares the gateway's fate rather than adding an independent way to be down — and the single-binary goal falls out of the same choice.

It stops being enough at the point where there is more than one gateway instance, because budgets and rate-limit counters then have to be counted somewhere shared; per-instance counters mean N instances allow N times the configured limit (§7.9.6). Postgres arrives with that, not before.

Three rules keep the swap cheap, and they are worth following from the first commit:

1. Nothing on the request path touches the database. Keys are cached in memory and read through on a miss, usage goes to a batch writer, budgets are in-memory counters reconciled against the store. This matters more than the engine choice and is what makes either engine survivable.
2. Portable SQL. No AUTOINCREMENT and no rowid tricks, ON CONFLICT … DO UPDATE rather than INSERT OR REPLACE, timestamps as integer unix millis, money and tokens as integers.
3. Migrations per dialect, sharing version numbers. A single migration set for both engines would be a promise that cannot be kept — SQLite's ALTER TABLE barely exists. Same numbering, same meaning, different files.

---

7. Router

7.1 Request lifecycle

Request
 ├─ 1. Auth       : key → project/team, permitted models, priority band, routing profile
 ├─ 2. Quota      : RPM/TPM and budget (in-memory counters; no DB on the request path)
 ├─ 3. Parse      : model, stream, messages/prompt, max_tokens, user.
 │                  The rest of the body passes through as untouched JSON.
 ├─ 4. Resolve    : model/alias → candidate pools (fabric × gpu_class)
 ├─ 5. ADMISSION  : "now?"      §7.2   ← waits here while the pool is saturated
 ├─ 6. SCHEDULING : "where?"    §7.3   filter → score → pick
 ├─ 7. Proxy      : stream to upstream, propagate cancellation           §7.8
 └─ 8. After the request (async): usage, spend, metrics, logs

The critical separation (taken from GIE / llm-d). When a request proceeds and where it goes are two distinct decisions and must stay distinct. Kept apart, the router can hold requests in its own central queue and apply priority and tenant fairness before they enter an engine queue.

Why this matters: an engine's own queue is FIFO and has no notion of tenants or priority. Once a request is in there, no prioritization is possible. When a batch job fills the vLLM queue with 500 requests, the TTFT of an interactive chat request arriving behind them collapses and there is nothing the router can do. v0.1 had no such layer; for a shared cluster this was the most serious omission.

7.2 Admission control (flow control)

7.2.1 Saturation detector

One saturation signal is produced per pool, combining two metrics (llm-d's approach — queue depth alone is not enough, KV pressure is a genuine exhaustion signal too):

// per replica, then aggregated across the pool as p90 or mean
func saturation(e EngineStats, t Thresholds) float64 {
    q  := float64(e.Waiting) / t.TargetWaitingPerReplica   // e.g. target 4
    kv := e.KVCacheUsage / t.KVCeiling                      // e.g. ceiling 0.90
    return math.Max(q, kv)
}

The goal is to keep a small but healthy buffer on the engine: enough requests for continuous batching to stay efficient, but not enough for the queue to balloon. Set TargetWaitingPerReplica too low and the GPU starves; too high and queueing delay returns. Both thresholds must be tunable per model/profile, and ctl bench (§10) should propose them.

Two lags have to be closed, and they need different answers. The engine's queue is scraped, so it is up to one interval old; the router's own inflight count is not. But inflight includes running requests, and treating those as queue makes a perfectly healthy engine look overloaded every interval. What is actually queued is inflight − batch capacity, and batch capacity is not in the metric contract — so llmgw learns it as the high-water mark of observed running. The second lag is shorter and subtler: between admission granting a request and the router dispatching it, the request is invisible everywhere. A simultaneous burst can therefore pass the check en masse, which is exactly the batch-job arrival pattern this layer exists to survive. The controller closes it by counting its own outstanding leases against dispatched inflight.

7.2.2 Priority bands

Every API key maps to a band, and the band sets the saturation ceiling:

bands:
  - { name: platinum, priority: 400, saturation_ceiling: 1.00 }  # flows to the healthy limit
  - { name: gold,     priority: 300, saturation_ceiling: 0.85 }
  - { name: silver,   priority: 200, saturation_ceiling: 0.60 }
  - { name: bronze,   priority: 100, saturation_ceiling: 0.30 }  # batch jobs

A ceiling of 1.00 means "hold only once the pool is past its healthy limit", not "never hold". Holding even the top band is deliberate: there is nobody above it to protect, so the benefit is a bounded engine queue. A flood dumped straight into a FIFO engine queue makes every request in that band slow, while queueing it at the gateway preserves ordering, TTL and tenant fairness. A band that must never wait — a health probe, say — gets a ceiling above 1 (1000).

The dispatch loop scans bands from highest to lowest. Once pool saturation exceeds a band's ceiling, that band's requests wait in the queue. The result: as the cluster fills, batch traffic slows down first and interactive traffic is protected. (In llm-d's measurements platinum, gold and silver stayed under a second while bronze batch fell far behind — exactly the desired behaviour.)

7.2.3 Fairness and queue management

• Fairness within a band: when several tenants share a band, dispatch turns are distributed round-robin. One tenant's traffic spike cannot consume the whole band.
• Capacity: each band's queue is bounded. When it is full, new requests get an immediate 503 — far better than accumulating silently.
• TTL: a request that waits longer than queue_ttl is dropped (504). There is no point sending the GPU work the client has already given up on.
• Client disconnect: if the connection drops while queued, the request is discarded quietly (context cancellation).
• Displacement (v1.1): to make room for a higher band, drop the newest requests from the lowest band — the oldest ones are already near their TTL and have invested the most waiting.

type Admission interface {
    // Blocks until the request may proceed; errors on cancellation or TTL.
    Acquire(ctx context.Context, req *RequestInfo) (Lease, error)
}

Lease.Release() is called once the response completes; it also maintains the router's inflight counter.

7.3 Scheduler: filter → score → pick

The list of "strategies" from v0.1 is replaced by a plugin pipeline (the GIE EPP model). No strategy is a separate monolith; scorers are combined by weight.

type Filter interface { // drops candidates (hard constraints)
    Name() string
    Filter(ctx *SchedCtx, eps []Endpoint) []Endpoint
}

type Scorer interface { // 0..1 score; 1 is best
    Name() string
    Score(ctx *SchedCtx, eps []Endpoint) map[EndpointID]float64
}

type Picker interface { // chooses from the scores
    Pick(ctx *SchedCtx, scores map[EndpointID]float64) Endpoint
}

type RoutingProfile struct {
    Name    string
    Filters []Filter
    Scorers []WeightedScorer   // {Scorer, Weight}
    Picker  Picker
}

Total score: score(ep) = Σ wᵢ · scoreᵢ(ep), with each scorer normalizing to 0..1 internally.

A two-stage decision. Because the topology is heterogeneous, the pool is chosen first and the replica second:

1. Pool selection: the pool with the highest Priority; if every replica there is saturated (§7.2.1), spill over to the next pool. This prevents falling back to the A100 pool before the H100 pool is actually full.
2. Replica selection: filter/score/pick within the chosen pool.

Routing profiles, from configuration (a per-model default, overridable per request via the x-llmgw-profile header):

Profile	Weights	Use
chat	prefix 3.0, queue 1.0, kv 1.0	Multi-turn chat and agents — high cache reuse
batch	queue 3.0, kv 1.0, prefix 0.5	Bulk work — throughput matters, TTFT does not
long-context	kv 3.0, prefix 2.0, queue 1.0	Long prompts — KV pressure is the real bottleneck
debug	round-robin picker	Troubleshooting

7.4 Scorers and their formulas

Scorer	Formula	Notes
queue	1 − min(1, waiting / target_waiting)	Queue depth
kv	1 − kv_cache_usage	KV occupancy; a hard penalty above the ceiling
prefix	matched_blocks / prompt_blocks	§7.5
load	1 − norm(running / drain_rate)	AIBrix: drain_rate is the observed completion rate. running alone cannot spot a slow replica.
inflight	1 − norm(router_inflight)	The router's own count; compensates for scrape lag. At a 2s scrape interval, herding is unavoidable without it.
pool-priority	constant, from the profile's Priority	Pool preference
session	session hash match	§7.5 tier 0
lora (v2)	is the adapter loaded	vllm:lora_requests_info

An alternative cost formulation (Dynamo). Modelling prefill and decode cost directly in blocks may be more accurate than normalized scores:

cost(ep) = w_overlap · (prompt_blocks − cached_blocks(ep)) + decode_blocks(ep)
choice   = argmin cost

A high w_overlap favours cache reuse (better TTFT); a low one evens out load (better ITL). Both can be expressed as a Scorer, so the pipeline survives either way: weighted scores in v1, with this cost form added as a scorer in v1.1 and compared using ctl bench.

Pickers:

• max-score: the highest score. Deterministic but prone to herding.
• p2c (power of two choices): two random candidates, the better one. The default once there are many replicas.
• softmax(temperature): Dynamo's router_temperature. Treats scores as logits and samples. temperature=0 degenerates to argmax. A small temperature breaks the oscillation that synchronized decisions cause among equally scored replicas.

7.5 Prefix / KV cache awareness — three tiers

This is the single largest win the router can deliver, but the cost gap between the tiers is enormous. They stay separate so they can be enabled per model.

Tier 0 — session affinity (v1, cheap, high return)

The client-supplied x-session-id header (falling back to OpenAI's user field) maps to a replica via bounded-load consistent hashing. If that replica is overloaded — beyond load_factor times the average — the next one on the ring is used. Roughly 100 lines of code; production-stack measured cache hits going from about 93% to 96% versus round robin. For multi-turn chat and agent workloads this is the best cost/benefit ratio available.

Tier 1 — approximate radix tree (v1.1, moderate cost)

The router maintains a tree that estimates each worker's cache from the request prefixes it has seen (the SGLang router's approach). It does not know the true cache state; it updates lazily and costs little.

The thresholds — the part that matters in real implementations:

• cache_threshold: below this match ratio, treat it as "no cache" and let load balancing decide.
• balance_abs_threshold and balance_rel_threshold: if max_load − min_load > abs and max_load / min_load > rel, ignore the prefix entirely and send to the least loaded replica. (AIBrix adds the same guard.) Without it, prefix affinity drowns one replica: the cache hit rate rises while p99 latency collapses.
• eviction_interval plus max_tree_size: keep the tree from growing without bound.

Which branch a decision took must be recorded as a metric: cache_hit | cache_miss_min_load | load_balance_override | stale | no_tree | no_healthy_worker. Without that, tuning prefix routing is guesswork.

Tier 2 — precise KV index (v2, expensive, optional)

vLLM publishes block events over ZMQ with --kv-events-config (BlockStored, BlockRemoved). The router subscribes and maintains a block_hash → {replica} map, giving it a true global view.

The trap — a cost the plan must acknowledge: block hashes must be computed identically to vLLM's token-id-based scheme. That forces the router to tokenize requests with the same tokenizer as the engine. On the Go side that means loading a tokenizer per model (HF tokenizers via cgo, or a pure-Go implementation), tracking version compatibility, and paying the tokenization CPU cost per request. This is why AIBrix requires a "remote tokenizer". Tiers 0 and 1 need no tokenizer — hence they are v1 and this is v2.

Event loss: if a gap appears in the event sequence numbers, request the missing range from the replay endpoint; if that fails, drop the index for that replica and fall back to tier 1 for it. Returning to approximate information beats silently routing on wrong data. If a replica restarts, its index is reset.

7.6 The engine metric contract

This is the formal definition of "adding a new engine". If an engine exposes these metrics in Prometheus format, llmgw can route to it (GIE's model server protocol):

Contract field	vLLM metric	llama.cpp metric	Mandatory
Waiting	vllm:num_requests_waiting	llamacpp:requests_deferred	yes
Running	vllm:num_requests_running	llamacpp:requests_processing	yes
KVCacheUsage	vllm:kv_cache_usage_perc	—	where the engine has it
PromptTokens	vllm:prompt_tokens_total	llamacpp:prompt_tokens_total	no
GenerationTokens	vllm:generation_tokens_total	llamacpp:tokens_predicted_total	no
BlockSize, NumGPUBlocks	vllm:cache_config_info	—	no
LoRA state	vllm:lora_requests_info (max_lora, running_lora_adapters, waiting_lora_adapters)	—	no

Mappings are per engine, not per cluster. A cluster running llama.cpp on its CPU nodes and vLLM on its GPU nodes has two vocabularies for the same numbers, and one global mapping would leave half the fleet unmeasurable. An endpoint that names no engine is read with vLLM's names.

Running and Waiting are the two that cannot be omitted: an engine that reports neither how much it is doing nor how much is queued cannot be scheduled against at all. KVCacheUsage is required of engines that publish it — a mapping that names it and does not get it is a contract violation, while a mapping that leaves it unnamed is a statement that this engine has no such metric. Current llama.cpp is the second case, and saturation there falls back to queue depth alone. That is a real loss of signal, mitigated by llama.cpp serving a fixed number of slots, so its queue depth already says when it is full.

Correction: an earlier version of this plan gave llama.cpp's KV metric as llamacpp:kv_cache_usage_ratio. Build b10991 does not publish it — it serves fifteen metric families and none is a cache ratio. Naming it failed the contract on every scrape and left the router with no numbers at all rather than two out of three.

Correction: v0.1 said vllm:gpu_cache_usage_perc; the correct name is vllm:kv_cache_usage_perc. Metric names drift between versions, so the ParseMetrics mapping must be overridable from configuration, and the agent must report the engine version when it registers.

Scrape rules:

• Every Ready endpoint every 1–2s, one goroutine per endpoint, with jitter.
• A failed scrape marks the endpoint stale; its score drops but it is not removed.
• Three consecutive failures trip the breaker (§7.7).
• A contract violation is reported at warn level, once, and again when it clears. It is a compatibility problem rather than a blip: every scrape fails, the router schedules to that endpoint blind, and per-scrape debug logging meant a cluster could look entirely healthy while none of its routing decisions were informed by anything.
• So that decisions are never made purely on stale data, the inflight scorer (§7.4) stays enabled at all times.

7.7 Reliability

• Retry: only before the first byte has been written upstream, only on connection errors, 503 or 429, always against a different endpoint, at most N times. No retries once the stream has started.
• Cooldown (the LiteLLM lesson): an endpoint that errors leaves the candidate pool for cooldown_seconds. A long fallback list plus retries without cooldown equals a retry storm: one endpoint's rate limit spreads across the whole cluster.
• Circuit breaker: a closed/open/half-open state machine per endpoint, with exponential backoff.
• Fallback: when a model has no usable endpoints, fall back to a model in its Fallbacks list (if the key permits it). The response carries an x-llmgw-served-model header. Budget fallback: when a key exceeds its budget, optionally downgrade to a cheaper model rather than rejecting the request (§7.9).
• Timeouts: separate limits for connection, first token (TTFT) and total, plus an inter-token idle timeout while streaming.
• Cancellation: when the client disconnects, cancel the upstream request too (context propagation) so the GPU stops working. This applies to queued requests as well.
• Draining: a Draining replica receives no new requests; active ones run to completion or time out.
• Control plane loss: the router keeps serving from its last known endpoint list (§14).

7.8 Streaming proxy details

• http.Transport with a high MaxIdleConnsPerHost and HTTP/1.1 keep-alive (engines rarely speak h2c).
• Read lines with bufio.Reader, pass data: lines through, and call http.Flusher.Flush() after every event.
• If the client did not send stream_options.include_usage, the router may add it (configurable) to capture usage from the final chunk, and then withhold that chunk from a client that did not ask for it.
• Normalize upstream error JSON into OpenAI's format.
• Response headers: x-llmgw-replica, x-llmgw-pool, x-llmgw-served-model, x-llmgw-queue-ms (time spent in admission) and x-llmgw-cache-decision (§7.5).

7.9 Identity, quota and usage (tenancy)

LiteLLM's production-proven model:

Organization ─► Team ─► Project ─► API Key

Each level carries a budget (in tokens or notional cost units), RPM/TPM limits and a permitted model list. A key additionally carries its priority band (§7.2.2) and its default routing profile (§7.3).

• API keys look like sk-llmgw-...; only the SHA-256 hash is stored. A leaked key is rotated without touching upstream credentials.
• Rate limiting: RPM via x/time/rate plus TPM. The token limit is reserved at request start as max_tokens + estimated prompt and reconciled against the real figure when the response completes. (The prompt estimate is bytes/4 in v1; it becomes exact once the tier 2 tokenizer arrives.)
• Budgets: periodic (daily/monthly) spend. On exhaustion: 429, or a budget fallback. Zero-cost models are exempt from budget checks, so a key whose budget is spent can still be given a "cheap model" fallback.
• Spend writes stay off the request path: usage records go to a channel and are batched into SQLite, with hourly rollups. Nothing writes to the database on the request path.
• Replica pinning (debug): the x-llmgw-replica header, scoped to the key. If the pinned replica is in cooldown or gone, the request falls through to normal routing.
• v1 is a single instance with in-memory limiters. With multiple instances this becomes Redis or consistent-hash stickiness (v2).

7.9.1 How keys are stored

A key is a 256-bit random value, shown once at creation and never again:

sk-llmgw-<base62(32 random bytes)>

Only sha256(key) is stored, indexed. Authentication hashes what the client presented and looks it up. Nothing can reconstruct the key from the database, so a database leak does not hand over working credentials.

SHA-256 rather than bcrypt or argon2, deliberately. Slow password hashing exists because humans pick guessable passwords, and the cost buys time against an offline dictionary attack. A 256-bit random value has no dictionary to attack: at any hash rate, guessing it is infeasible. What a slow hash would buy here is nothing, and what it would cost is tens of milliseconds on every single request. Using the fast hash is the correct choice precisely because the input is high entropy — and that is the assumption to re-examine if the key format ever changes.

Alongside the hash, a key_prefix (sk-llmgw-4f2a…) is stored in clear so the UI and audit log can identify a key without revealing it.

CREATE TABLE api_keys (
  id           TEXT PRIMARY KEY,
  key_hash     BLOB NOT NULL UNIQUE,   -- sha256 of the key; the only copy
  key_prefix   TEXT NOT NULL,          -- for display, never for auth
  name         TEXT NOT NULL,
  project_id   TEXT NOT NULL REFERENCES projects(id),
  band         TEXT,                   -- §7.2.2; NULL inherits the project's
  routing_profile TEXT,                -- §7.3
  created_at   INTEGER NOT NULL,
  expires_at   INTEGER,                -- NULL never expires
  revoked_at   INTEGER,                -- soft delete, so usage rows keep a valid FK
  last_used_at INTEGER                 -- written lazily; see below
);
CREATE INDEX api_keys_by_project ON api_keys(project_id);

Revocation is a soft delete. Usage rows reference the key, and an audit trail that loses its subject when the key is deleted is not an audit trail.

last_used_at is written at most once a minute per key, from the same batch writer as usage. Writing it per request would put a database write back on the request path for a field nobody reads in real time.

7.9.2 The tenancy tree and how limits resolve

CREATE TABLE organizations (id TEXT PRIMARY KEY, name TEXT NOT NULL);
CREATE TABLE teams    (id TEXT PRIMARY KEY, org_id  TEXT NOT NULL REFERENCES organizations(id), name TEXT NOT NULL);
CREATE TABLE projects (id TEXT PRIMARY KEY, team_id TEXT NOT NULL REFERENCES teams(id),         name TEXT NOT NULL);

-- Limits hang off any level rather than being duplicated onto each one.
CREATE TABLE limits (
  scope_type TEXT NOT NULL,            -- 'org' | 'team' | 'project' | 'key'
  scope_id   TEXT NOT NULL,
  rpm        INTEGER,                  -- NULL means no limit at this level
  tpm        INTEGER,
  budget_micros  INTEGER,             -- per period, in micro-units; NULL means no budget
  budget_period  TEXT,                 -- 'day' | 'month'
  allowed_models TEXT,                 -- JSON array; NULL means all
  PRIMARY KEY (scope_type, scope_id)
);

A request's effective limit is the most restrictive value along key → project → team → org. A team cannot raise a limit its organization set, which is the only composition rule that makes a budget mean anything. Model allowlists compose the same way, by intersection: a key cannot add a model its team withheld. An empty allowlist is not an absent one — absent means every model, empty means none.

Budgets are counted in integer micro-units rather than a float. A budget is accumulated by repeated addition, and repeated addition of floats drifts.

Resolving four levels per request would be four lookups on the hot path, so the resolved view is computed once and cached per key hash, invalidated whenever anything on that key's chain changes. Authentication is then a single map lookup, with the database consulted only on a cache miss.

7.9.3 How accounting is written

The volume decides the design. At 1000 requests/sec, one row per request is 86 million rows a day, and SQLite has a single writer. Inserting per request would serialize every request behind that writer.

So the path is:

request completes
   → usage record onto a buffered channel   (nanoseconds, never blocks)
   → batch writer: every 200ms or 500 records, one transaction
   → hourly rollup job collapses raw rows
   → raw rows older than the retention window are deleted

CREATE TABLE usage (
  ts            INTEGER NOT NULL,      -- unix millis
  key_id        TEXT NOT NULL,
  project_id    TEXT NOT NULL,
  model         TEXT NOT NULL,
  served_model  TEXT NOT NULL,         -- differs from model when a fallback served it
  pool          TEXT,
  status        INTEGER NOT NULL,
  prompt_tokens     INTEGER NOT NULL,
  completion_tokens INTEGER NOT NULL,
  cached_tokens     INTEGER NOT NULL,  -- prefix cache hit, from the engine
  ttft_ms       INTEGER,
  duration_ms   INTEGER NOT NULL,
  queue_ms      INTEGER                -- time spent in admission (§7.2)
);
CREATE INDEX usage_by_key_ts ON usage(key_id, ts);

-- What survives retention, and what budgets are enforced against.
CREATE TABLE usage_rollup (
  hour_ts    INTEGER NOT NULL,
  key_id     TEXT NOT NULL,
  project_id TEXT NOT NULL,
  model      TEXT NOT NULL,
  requests   INTEGER NOT NULL,
  prompt_tokens     INTEGER NOT NULL,
  completion_tokens INTEGER NOT NULL,
  cached_tokens     INTEGER NOT NULL,
  units      REAL NOT NULL,            -- billable cost, per §7.9.4
  PRIMARY KEY (hour_ts, key_id, model)
);

The honest tradeoff: records sitting in the channel when the process is killed are lost — at most a few hundred milliseconds of accounting. Making that impossible means a durable write before each response, which puts a disk fsync on every request. For usage that informs budgets and invoices, losing a fraction of a second of records to a hard kill is the better trade. If this ever has to feed external billing, that trade has to be revisited, and the answer then is an append-only journal the batch writer drains, not a synchronous insert.

7.9.4 How budgets are enforced without reading the database

Budgets are enforced per request, so they cannot involve a query. Instead:

• On startup, the current period's spend is loaded per key from usage_rollup plus the un-rolled tail of usage. A restart must not reset a budget, which it would if the counter lived only in memory.
• From then on an in-memory counter is incremented as responses complete. The database catches up through the batch writer.
• At the period boundary the counter resets and the new period starts empty.

"Units" are deliberately not dollars. Self-hosted models have no per-token price, so a unit is whatever the operator configures — GPU-seconds, a notional rate per million tokens, or 1.0 for everything, which reduces budgets to token quotas. This is also what makes the plan's budget fallback work: a model priced at zero stays available to a key whose budget is spent (§7.7).

7.9.5 Token counting

Two numbers exist and they are not the same:

• The estimate, computed before dispatch as bytes/4, used to reserve TPM quota and to score prefix affinity. It is a guess and is labelled as one.
• The engine's report, taken from the response's usage block. This is authoritative and is what gets recorded.

The router asks engines for usage even when the client did not, then drops that chunk on the way back (§7.8), so accounting does not depend on client behaviour. TPM is reserved at request start as max_tokens + estimate and reconciled against the real figure on completion — reserving optimistically would let a key exceed its limit by a whole response.

The estimate stops being a guess only when a real tokenizer arrives, which the plan defers to v2 along with tier 2 prefix routing (§7.5, §14).

7.9.6 Multiple router instances

Everything above assumes one instance. Two would each hold their own rate limit counters and each allow the configured limit, so the effective limit doubles.

v1 is single-instance and this is a documented constraint rather than a bug. The answer when it arrives is either shared counters in Redis, or consistent hashing on the key so every request for one key reaches the same instance. The second is cheaper and degrades better, but rebalances counters whenever the instance set changes.

---

8. Observability

Prometheus metrics (examples):

llmgw_requests_total{model,pool,status}
llmgw_request_duration_seconds{model,pool}
llmgw_ttft_seconds{model,pool}
llmgw_tokens_total{model,direction="prompt|completion"}
llmgw_endpoint_inflight{replica}
llmgw_endpoint_score{replica}
llmgw_replicas{model,state}
llmgw_gpu_free{node,gpu_class,fabric}
llmgw_schedule_failures_total{model,reason}
llmgw_agent_connected{node}

# admission / flow control (§7.2)
llmgw_admission_queue_depth{band,model}
llmgw_admission_wait_seconds{band,model}          # histogram
llmgw_admission_rejected_total{band,reason="full|ttl|client_gone"}
llmgw_pool_saturation{pool}                       # 0..1, the detector's output

# routing decisions (§7.5) — prefix routing cannot be tuned without these
llmgw_cache_decision_total{model,decision="hit|miss_min_load|balance_override|stale|no_tree"}
llmgw_prefix_match_ratio{model}                   # histogram
llmgw_scorer_value{scorer,replica}
llmgw_endpoint_metrics_stale{replica}
llmgw_capacity_profile_stale{model,profile}

• OpenTelemetry traces: router span → upstream span (the traceparent header is forwarded to the engine).
• Structured logs (slog JSON) with request_id everywhere.
• llmgw ctl top: live queue depth and occupancy per pool.

---

9. Configuration Examples

9.1 server.yaml

listen: ":8080"
admin_listen: ":9090"          # metrics, admin API, pprof
agent_listen: ":7443"
tls:
  ca: /etc/llmgw/ca.pem
  cert: /etc/llmgw/server.pem
  key: /etc/llmgw/server-key.pem
store:
  path: /var/lib/llmgw/llmgw.db

fabrics:
  - name: ib-a
    gid_prefix: "fec0:0000:0000:000a"   # optional automatic mapping
  - name: ib-b
    gid_prefix: "fec0:0000:0000:000b"

router:
  scrape_interval: 2s
  inject_usage: true
  retries: 2
  cooldown_seconds: 15            # how long a failing endpoint leaves the candidate pool
  endpoint_cache_max_age: 5m      # keep serving this long if the control plane goes down

  # §7.2 admission — the saturation detector
  admission:
    target_waiting_per_replica: 4   # proposed by ctl bench
    kv_ceiling: 0.90
    queue_ttl: 30s
    queue_capacity_per_band: 2000
    bands:
      - { name: platinum, priority: 400, saturation_ceiling: 1.00 }
      - { name: gold,     priority: 300, saturation_ceiling: 0.85 }
      - { name: silver,   priority: 200, saturation_ceiling: 0.60 }
      - { name: bronze,   priority: 100, saturation_ceiling: 0.30 }

  # §7.3 routing profiles
  default_profile: chat
  profiles:
    chat:
      scorers: { prefix: 3.0, queue: 1.0, kv: 1.0, inflight: 1.0 }
      picker:  { type: p2c }
    batch:
      scorers: { queue: 3.0, kv: 1.0, load: 1.0, prefix: 0.5 }
      picker:  { type: max-score }
    long-context:
      scorers: { kv: 3.0, prefix: 2.0, queue: 1.0 }
      picker:  { type: softmax, temperature: 0.2 }

  # §7.5 prefix awareness
  prefix:
    session_affinity: { enabled: true, header: x-session-id, load_factor: 1.25 }  # tier 0
    approx_tree:                                                                  # tier 1
      enabled: false          # v1.1
      cache_threshold: 0.30
      balance_abs_threshold: 16
      balance_rel_threshold: 1.5
      max_tree_size: 2000000
      eviction_interval: 60s
    kv_events: { enabled: false }   # tier 2 — needs a tokenizer, v2

models:
  - name: llama-3.3-70b
    source: /models/Llama-3.3-70B-Instruct
    scaling: { min: 2, max: 6 }
    fallbacks: [qwen2.5-32b]
    routing_profile: chat
    slo: { ttft_p99: 2s, tpot_p99: 50ms }   # ctl bench measures capacity against this SLO
    profiles:
      - name: h100-tp4-fp8
        gpu_class: h100-80g
        engine: vllm
        tp: 4
        nodes: 1
        priority: 100
        weight: 100
        args: ["--quantization", "fp8", "--max-model-len", "32768"]
      - name: a100-tp8
        gpu_class: a100-80g
        engine: vllm
        tp: 8
        nodes: 1
        priority: 50
        weight: 45

  - name: deepseek-large
    source: /models/DeepSeek-V3
    scaling: { min: 1, max: 2 }
    profiles:
      - name: h100-2node
        gpu_class: h100-80g
        engine: sglang
        tp: 16
        nodes: 2           # → must live on a single fabric
        gpus_per_node: 8
        priority: 100
        weight: 100
        startup_timeout: 40m

9.2 agent.yaml

node_id: gpu-a-01
server: "llmgw-server.internal:7443"
mgmt_interface: eth0
tls: { ca: ..., cert: ..., key: ... }
labels: { rack: r12 }
fabrics:
  - name: ib-a
    hcas: [mlx5_0, mlx5_1, mlx5_2, mlx5_3]
port_range: [20000, 20999]        # engine HTTP and dist-init ports
engines:
  vllm:   { bin: /opt/venvs/vllm/bin/vllm, launch_mode: ray }
  sglang: { python: /opt/venvs/sglang/bin/python }
work_dir: /var/lib/llmgw-agent

---

10. Benchmarking and Weighting

llmgw ctl bench <model> --profile <p>:

• Drives a target replica at increasing concurrency with fixed prompt and output lengths.
• Measures TTFT p50/p95, TPOT, throughput (tok/s) and the maximum concurrency that still meets the SLO.
• The output is the CapacityProfile from §6.2.1: MaxConcurrencyAtSLO, throughput, TTFT/TPOT p99.
• --apply updates three things at once: the profile's weight, target_waiting_per_replica, and the autoscaling target.
• This is the only sound way to compare GPU classes; never use an estimated weight.
• --sweep-prefix mode measures behaviour at 0/50/90% prefix cache hit rates, so routing profile weights (§7.3) can be tuned from measurements instead of guesses.
• A profile goes stale when the engine version changes, so every ctl bench result is tagged with the engine version.

---

11. Roadmap

Phase 0 — Skeleton (1 week)

• go mod init, cobra commands, koanf config, slog, Makefile, golangci-lint
• sim-engine: OpenAI-compatible fake engine (tunable latency, token rate, queue, vLLM-format /metrics)
• CI: go test ./..., race detector

Phase 1 — Static router (2 weeks)

• OpenAI handlers, streaming proxy, cancellation propagation
• Static endpoint list from configuration
• The Filter/Scorer/Picker pipeline (§7.3) — a plugin pipeline from the start, not one "strategy"
• queue, kv, inflight scorers; max-score and p2c pickers
• Session affinity (§7.5 tier 0) — cheap, an early win
• Metrics scraping (the §7.6 contract), staleness detection, retry, cooldown, breaker
• Prometheus metrics plus decision metrics (§8) — llmgw_requests_total, llmgw_ttft_seconds, llmgw_scorer_value, llmgw_candidates_considered, llmgw_candidates_filtered_total, llmgw_tokens_total
• Exit criterion: 1000 concurrent streams behind 3 sim-engines; upstream cancellation on client disconnect verified; router overhead p99 < 2 ms.

Phase 1.5 — Admission control (1 week) ⭐

• Saturation detector (§7.2.1)
• Priority bands, round-robin tenant fairness within a band, queue capacity and TTL
• Admission interface and lease, cancellation while queued
• llmgw_admission_* metrics
• Exit criterion: while 500 concurrent batch requests flow through the bronze band, the TTFT p99 of interactive requests in the platinum band stays under 1.5× its idle value. Do not move on before writing this test — it is the entire reason phase 1.5 exists.

Phase 2 — Agent: discovery and single-node engines (2 weeks)

• GPU discovery (nvidia-smi path), IB discovery from sysfs
• Engine interface, vLLM and SGLang command and environment construction
• llama.cpp adapter, and with it CPU profiles: a profile may omit its GPU class, and only an engine that can serve without a card may do so. This is what makes the whole path testable on machines that have no GPUs — verified end to end on four VMs serving Qwen2.5-0.5B on CPU at ~50 tok/s, twelve concurrent requests balancing 4/4/4 across three replicas.
• Supervisor (process groups, log buffer, state machine, orphan adoption)
• NVML backend behind the nvml build tag, with NVLink topology. It compiles in CI but has not run against a driver: verify on a GPU node before relying on the NVLink domains it reports.
• connect-go stream, register/heartbeat, shard status
• mTLS: a private CA (llmgw ctl cert), certificates on both ends, and the control plane refusing any agent whose claimed node id does not match its certificate. Plaintext HTTP/2 remains the default and still must stay on a trusted network; agent_tls/tls in the configuration turns it on.
• Exit criterion (met, via configuration rather than ctl): a model starts and stops on a node. Demonstrated with real binaries: llmgw server and llmgw agent as separate processes, the agent registering, a replica being placed, an engine launched through the vLLM adapter, traffic served, and the node's loss detected when the agent is killed. ctl itself is phase 3's remaining item.

Phase 3 — Control plane (2–3 weeks)

• Register carries the agent's running shards, so a control plane restart rebuilds replica state from the fleet instead of orphaning live engines (§6.5)
• Orphan adoption: an agent takes back engines a previous agent left running, and kills whatever it cannot identify or the catalog will not claim
• Inventory: nodes, hardware, fabric attachments, allocations
• Placement: profile priority, bin packing, NVLink-aware GPU selection, explain-placement rejection reasons, property-based invariant tests
• Catalog: models, profiles, alias and fallback resolution, validation
• Reconciler: desired vs actual, replica lifecycle, node expiry, backoff
• EndpointSource wired from the reconciler to the router
• ctl: nodes, models, replicas, cordon, uncordon, drain, explain-placement
• ctl logs: needs a request/response pair on the agent stream, which the protocol does not have. The failure path already carries an engine log tail with every status report, so the common case is covered; what is missing is reading a healthy shard's output on demand.
• The router's EndpointSource becomes dynamic
• ctl: nodes, models, replicas, cordon, uncordon, drain, explain-placement
• ctl logs: needs a request/response pair on the agent stream, which the protocol does not have. The failure path already carries an engine log tail with every status report, so the common case is covered; what is missing is reading a healthy shard's output on demand.
• Exit criterion (met against a simulated agent fleet): shutting down a node moves its replicas elsewhere while the router keeps serving. Measured in test/e2e: a node hosting a live replica is killed mid-traffic; the replica is rebuilt on a survivor and ~370 requests complete with zero failures. The real agent path still needs the stream, so this is proof of the control logic, not of the transport.

Phase 4 — Multi-node and fabrics (2 weeks)

• Fabric identity from the agent's explicit labels
• Multi-node placement (the fabric constraint), port allocation, rank ordering
• NCCL environment generation, HCA selection
• Replica-level failure handling (one dead shard kills the replica)

The launch path is now built too:

• Only the leader is health-checked. Engine.ServesHTTP(spec) says which ranks answer the API; a worker is judged by staying alive, and the replica's real readiness is the leader answering, which only happens once every worker has joined. Health-checking a worker's port used to time out and fail a replica that was coming up perfectly well.

• A shard can be several processes. Engine.Commands(spec) returns them in start order: every command but the last is a sidecar kept alive for the shard's lifetime and torn down with it, and the last one's exit is the shard's. vLLM under Ray is the reason — the leader runs a Ray head and then the server, a worker runs only the join and no vLLM process at all.

• The leader starts first. Both engines put the rendezvous point on rank 0, so a worker started first has nothing to connect to. An earlier version had this backwards; vLLM's server blocks until the Ray cluster has the GPUs it needs, so the leader can start immediately and waits on its own.

• Rendezvous is waited for, not assumed. A command can require that LeaderAddr:LeaderPort accepts a connection before the next one starts, bounded by rendezvous_timeout so a head that never appears fails the shard instead of hanging it.

• Not verified against real engines. Everything above is tested with stub processes. The Ray sequencing, and whether the deployed vLLM version wants ray or mp, can only be confirmed on real hardware.

• Exit criterion: a 2-node replica on each fabric; killing a node rebuilds the replica on the same fabric and never spreads it across two.

Phase 5 — Production features (3 weeks)

• SQLite store and migrations. It arrives here rather than in phase 3 because everything that genuinely needs persisting — keys, usage, capacity profiles — is a phase 5 feature. Writing it earlier would have meant designing a schema against guesses (§6.5).
• Tenancy: org/team/project/key hierarchy, key storage and authentication, limit resolution (most restrictive along the chain), llmgw keys
• Budgets, RPM/TPM enforcement and the usage batch writer (§7.9.3, §7.9.4). Rate limits are a sliding window, not a fixed one: a fixed window lets a caller send its whole allowance at the end of one window and again at the start of the next, so the observed rate doubles exactly when the system is least able to take it.
• Model allowlists and key→band mapping. With auth on, the band comes from the key and the header is ignored — it was trusted only while there was no identity to consult, which let any client promote itself.
• Budget fallback
• Routing profiles from configuration plus the x-llmgw-profile header
• Approximate radix tree (§7.5 tier 1), with its thresholds and decision metrics
• Fallback models and pool spillover
• ctl bench → CapacityProfile (§6.2.1), ctl plan
• OTel tracing
• systemd units, documentation

Phase 6 — v2 ideas

• Precise KV index (§7.5 tier 2) — vLLM KV events plus tokenizer integration
• Dynamo-style block cost function, tuning the softmax picker
• Admission displacement (dropping requests from lower bands)
• Autoscaling and priority-based preemption
• Control plane HA (Raft)
• Weight downloading and cache management
• Prefill/decode disaggregation (§12)
• LoRA adapter routing (affinity to a replica that already has the adapter)
• External provider support (as a fallback)
• Admin web UI

---

12. Prefill/Decode Disaggregation (v2 notes)

• Prefill and decode replicas must live on the same fabric (KV transfer goes over RDMA).
• A two-stage flow in the router: send to a prefill replica → KV handle → decode replica.
• Engine support varies sharply by version (vLLM KV connectors, SGLang PD mode, NVIDIA Dynamo/NIXL). Wrapping the engine's own built-in PD router as a single "endpoint" is less risky than implementing this ourselves.
• What llm-d and Dynamo show: PD's benefit lies mainly in long context. For short-prompt chat traffic the KV transfer cost eats the gain. Measure your workload's ISL/OSL distribution with the benchmark in §10 first, and enable PD only for long-context models.
• The scheduler gains a Role: prefill|decode field and a "paired placement on the same fabric" constraint.

---

13. Test Strategy

• Unit: the scheduler (table-driven: heterogeneous inventory plus expected placement or failure reason), balancer scores, SSE parsing, metric parsing (fixtures taken from real vLLM/SGLang /metrics output).
• Property-based: the invariants "a multi-node replica lives on one fabric" and "one GPU class per replica" always hold in scheduler output (testing/quick or rapid).
• Integration: server plus N fake agents (fake GPU/IB inventory) plus sim-engines, in one process.
• Chaos: kill agents, break the network (toxiproxy), crash engines, slow engines, and kill the control plane to verify the router keeps serving from its cache.
• Priority isolation: phase 1.5's exit criterion as a permanent regression test — high-priority TTFT is preserved under low-priority load. This test must stay in CI.
• Routing quality: give sim-engine a realistic prefix cache model (block-level LRU) and measure the hit rate against synthetic multi-turn traffic. The comparison between round robin, session affinity and the approximate tree must be numeric.
• Load: k6 or our own Go load generator; measure the router's own overhead (target: p99 under 2 ms of added latency).
• Real hardware: at the end of every phase, a smoke test on at least one node with IB and one without.

---

14. Risks and Open Questions

Topic	Note
Engine version differences	CLI flags and metric names change often → make engine adapters version-aware and overridable from configuration.
vLLM multi-node launch	Ray or built-in multiprocessing? Must be tested on the version actually deployed.
Fabric detection	Can a unique subnet prefix be set on each OpenSM? If not, manual labelling is mandatory.
Multiple fabrics per node	Does this occur? If so, HCA-to-GPU NUMA matching matters.
Model storage	Is there a shared filesystem, and what are the load times? Startup timeouts follow from this.
Mixed driver/CUDA versions	Should engine virtualenvs differ per GPU class? (The engines config must be overridable per profile.)
Single control plane	A SPOF in v1. The router caches the endpoint list (§6.4) and keeps working through a brief control plane outage.
Security	Engine ports must be reachable only by the router (firewall or bind address).
Tokenizer consistency	The precise KV index (§7.5 tier 2) forces the router to use the same tokenizer as the engine. In Go that means cgo or a separate tokenizer service, which is why it is deferred to v2; v1's prefix approach needs no tokenizer. The same limitation makes the TPM prompt estimate approximate in v1 (bytes/4).
Prefix affinity backfiring	Without the load-imbalance guard (the §7.5 tier 1 thresholds), prefix routing raises the cache hit rate while drowning a single replica and wrecking p99. Prefix affinity must not be enabled before those thresholds are.
Tuning the admission thresholds	Too low a target_waiting_per_replica starves the GPU; too high brings queueing delay back. This single number governs the system's behaviour, so measure it with ctl bench rather than guessing.
KV event loss	ZMQ is fire-and-forget; a lost event makes the router silently misroute. Sequence-gap detection and falling back to tier 1 are mandatory (§7.5).
Metric staleness	At 1000 requests/sec with a 2s scrape interval, decisions rest entirely on stale data. That is why the inflight scorer is not optional.
Usage lost on a hard kill	Records buffered for the batch writer are lost if the process is killed — at most a few hundred milliseconds of accounting. Acceptable for budgets; not acceptable if this ever feeds external billing, and the answer then is an append-only journal rather than a synchronous insert (§7.9.3).
Rate limits with several routers	Counters are per instance, so N instances allow N times the configured limit. v1 is single-instance by design; Redis or consistent-hash stickiness is the fix (§7.9.6).
Control plane restart	Until Register carries running shards, restarting the control plane orphans live engines: they keep holding GPUs the inventory believes are free, and the next placement fails to allocate with an error pointing at the wrong process (§6.5).

---

15. First Steps

1. mkdir llmgw && cd llmgw && go mod init <module-path>/llmgw
2. cmd/llmgw/main.go: an empty cobra skeleton with the server, agent, ctl and sim-engine subcommands.
3. internal/sim: the fake engine — make this the first real code, because everything else is tested against it.
4. internal/router/proxy: streaming proxy plus a cancellation test.
5. internal/router/scheduler: the Filter/Scorer/Picker interfaces plus the queue, kv and inflight scorers. Do not write a single "least-load" function — build the pipeline from the start; coming back to it later is expensive.
6. internal/router/admission: saturation detector and band queues. Phase 1.5.
7. Write the end-to-end tests for the phase 1 and phase 1.5 exit criteria, then move to phase 2.

Use internal/router/scheduler/ rather than internal/router/balancer/ (the tree in §4 reflects this): "balancer" understates the job — this component does more than balance load.