ilkerandClaude Opus 5 40427b2705 plan: correct the store design and specify how tenancy is actually stored
Two problems with what v0.1 said about persistence.

First, it listed eight tables, most of which were a mistake. A database that
describes the cluster can be wrong; the cluster cannot. An engine that dies
during a restart leaves a row saying "running", and the scheduler believes it.
Nodes, replicas, shards and the catalog are all rebuilt from the fleet or from
configuration on every reconnect, so they need no rows at all. What genuinely
needs persisting — API keys, usage, capacity profiles — is entirely phase 5
work, so the store moves there and gets a schema shaped by real requirements
instead of guesses.

Not persisting replica state has one consequence that must be closed first:
restarting the control plane currently orphans live engines. They keep holding
GPUs the inventory believes are free, and the next placement fails to allocate
with an error pointing at the new process rather than the old one. The fix
belongs on the agent, not in a database — Register carries the shards the agent
is supervising, and the supervisor's orphan scan (written, tested, never called)
kills whatever nothing claims. That cannot disagree with reality, and it covers
agent restarts and control plane restarts with one mechanism. Added to phase 3.

Second, §7.9 described tenancy behaviour but never said how any of it is stored,
which left the most security-sensitive part of the system unspecified. It now
covers the schema for keys and the tenancy tree, why a fast hash is the correct
choice for a 256-bit random key and what assumption that rests on, how limits
resolve to the most restrictive value along the chain, why authentication is a
map lookup rather than four queries, the batching pipeline that keeps 86 million
rows a day off the request path, what "units" mean when self-hosted models have
no per-token price, and how budgets survive a restart.

Two tradeoffs are now stated rather than left implicit: usage buffered for the
batch writer is lost on a hard kill, which is fine for budgets and not fine for
external billing; and rate limit counters are per instance, so N routers allow N
times the limit. Both are in the risk table.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 06:08:16 +02:00
2026-09-16 01:00:03 +02:00
2026-09-16 01:53:38 +02:00

llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.

It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.

Status

Releases track the roadmap phases; see CHANGELOG.md.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router (filter/score/pick, streaming proxy) released
v0.3.0 1.5 Admission control (priority bands) released
v0.4.0 2 Agent: GPU/IB discovery, single-node engines planned
v0.5.0 3 Control plane: catalog, scheduler, reconciler planned
v0.6.0 4 Multi-node placement and fabric awareness planned
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

As of v0.3.0 the router serves traffic and protects it: a batch workload no longer starves interactive requests. Still missing before v1.0.0: authentication, rate limits and budgets, and the control plane that replaces the static endpoint list.

Building

make build      # bin/llmgw
make test       # unit tests
make race       # with the race detector

Running the router

# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8001=llama \
  --endpoint e2=http://127.0.0.1:8002=llama

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'

For anything real use a config file; deploy/examples/server.yaml is commented throughout. Metrics, health and pprof are on the admin port (:9090 by default), deliberately separate from the client-facing one.

Requests carry a priority band (x-llmgw-band). Under load, low bands wait at the gateway while high bands keep flowing, so the cluster degrades from the bottom up instead of everywhere at once. Measured effect: interactive p99 stays within 1.24x of idle while 500 concurrent batch requests run, against 58x with admission off.

Routing profiles select how endpoints are scored: chat favours session affinity, batch favours the shortest queue, long-context favours free KV cache. Pick one per model in the config, or per request with the x-llmgw-profile header.

sim-engine

Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:

  • continuous batching (bounded number of concurrent sequences)
  • KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
  • automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
  • per-token latency growing with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics using vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'

Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks, --prefill-tps, --decode-tps, --batch-contention and --prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for deterministic runs.

The engine metric contract

For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc

Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.

Module path

gitea.mynodes.xyz/ilker/llmgw. To change it:

make rename NEW=git.example.com/team/llmgw

Language

Code, comments, tests, commit messages and documentation are in English.

S
Description
No description provided
Readme
30 MiB
0 Stars 1 Watchers 0 Forks
Languages
Go 94.8%
HTML 3.2%
CSS 1%
JavaScript 0.4%
Shell 0.3%
Other 0.2%