Ilker ManapandClaude Opus 5 ad37bee081
CI / test (push) Successful in 1m30s
Drive on-demand loading end to end, not just per component
On-demand loading is four components agreeing: the router notices nothing is
serving, the control plane decides the model may be started, the reconciler
places it, and the endpoint list carries it back. Each half was tested
alone, and the wiring between them is where this session's real bugs have
been — a metrics collector registered after the registry was built, an
adoption path that never health-checked — neither of which failed a unit
test.

Three tests through the real request path: one request loads a model that
was not running and is answered by it, an idle one is stopped with the node
still heartbeating so it is eviction rather than node loss, and an unknown
model still gets the unknown-model error, because on-demand must not turn
every typo into "loading".

Writing them was itself the check. The first version heartbeated once and
then slept through an UnreachableAfter of 300 milliseconds, so the node went
quiet and nothing was placed — which looks exactly like the feature failing.
A wrong assertion too: with an on-demand-only cluster and nothing loaded,
naming what is served means naming nothing, and "no model is currently being
served" is the honest answer rather than a lost error message.

This matters more now that there is no cluster to deploy to. Live
verification found several things today that no unit test would have, and
these tests are what has to replace it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-17 17:52:49 +02:00
2026-09-16 01:00:03 +02:00
2026-09-17 07:33:49 +02:00

llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.

It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.

Getting started

Starting from nothing — no packages installed, no drivers, no nodes — follow docs/install.md: it builds a control plane, adds a node and serves a model, and says plainly which parts llmgw does not install for you.

Guide For
docs/install.md Standing up a cluster from scratch
docs/node-onboarding.md Adding nodes to one that exists
docs/onboarding.md Giving people accounts and keys

Status

Releases track the roadmap phases; see CHANGELOG.md.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router (filter/score/pick, streaming proxy) released
v0.3.0 1.5 Admission control (priority bands) released
v0.4.0 2 Agent: GPU/IB discovery, engine supervision released
v0.5.0 3 Control plane: catalog, placement, reconciler, ctl released
v0.6.0 4 Multi-node placement and fabric awareness planned
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

As of v0.3.0 the router serves traffic and protects it: a batch workload no longer starves interactive requests. Still missing before v1.0.0: authentication, rate limits and budgets, and the control plane that replaces the static endpoint list.

Building

make build      # bin/llmgw
make test       # unit tests
make race       # with the race detector

Operating a cluster

llmgw ctl nodes                      # capacity, health, what is running where
llmgw ctl models                     # catalog and replica counts
llmgw ctl replicas --model llama     # replicas and their shards
llmgw ctl explain-placement llama    # why a model has not been placed
llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs

ctl talks to the admin port (:9090 by default), not the client-facing one. Add --json to any command for machine-readable output.

explain-placement is the one worth knowing about: "unschedulable" on its own says nothing about which of six constraints bit, so it prints every candidate considered and the reason it was rejected.

Running the router

# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8001=llama \
  --endpoint e2=http://127.0.0.1:8002=llama

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'

For anything real use a config file; deploy/examples/server.yaml is commented throughout. Metrics, health and pprof are on the admin port (:9090 by default), deliberately separate from the client-facing one.

Requests carry a priority band (x-llmgw-band). Under load, low bands wait at the gateway while high bands keep flowing, so the cluster degrades from the bottom up instead of everywhere at once. Measured effect: interactive p99 stays within 1.24x of idle while 500 concurrent batch requests run, against 58x with admission off.

Routing profiles select how endpoints are scored: chat favours session affinity, batch favours the shortest queue, long-context favours free KV cache. Pick one per model in the config, or per request with the x-llmgw-profile header.

sim-engine

Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:

  • continuous batching (bounded number of concurrent sequences)
  • KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
  • automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
  • per-token latency growing with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics using vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'

Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks, --prefill-tps, --decode-tps, --batch-contention and --prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for deterministic runs.

The engine metric contract

For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc

Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.

Module path

gitea.mynodes.xyz/ilker/llmgw. To change it:

make rename NEW=git.example.com/team/llmgw

Language

Code, comments, tests, commit messages and documentation are in English.

S
Description
No description provided
Readme
26 MiB
0 Stars 2 Watchers 1 Forks
v0.16.3
Latest
2026-11-03 09:56:29 +01:00
Languages
Go 81.1%
JavaScript 15.2%
HTML 2.6%
CSS 0.6%
Python 0.2%
Other 0.2%