ilkerandClaude Opus 5 664c8e9126
CI / test (push) Successful in 1m16s
ci: add workflow_dispatch and take the Go version from go.mod
The runner this targets is a containerized act_runner registered with the
ubuntu-latest label, separate from maas-builder: that one runs in host mode as
root on the image build machine, which is not somewhere a test suite belongs.

workflow_dispatch lets the queued runs be retried from the UI once a runner
exists, without needing an empty commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 01:19:12 +02:00
2026-09-16 01:00:03 +02:00

llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.

It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.

Status

Phase Scope State
0 Skeleton and sim-engine done
1 Static router (filter/score/pick, streaming proxy) in progress
1.5 Admission control (priority bands) todo
2 Agent: GPU/IB discovery, single-node engines todo
3 Control plane: catalog, scheduler, reconciler todo
4 Multi-node placement and fabric awareness todo
5 Tenancy, prefix affinity, benchmarking todo

Building

make build      # bin/llmgw
make test       # unit tests
make race       # with the race detector

sim-engine

Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:

  • continuous batching (bounded number of concurrent sequences)
  • KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
  • automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
  • per-token latency growing with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics using vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'

Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks, --prefill-tps, --decode-tps, --batch-contention and --prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for deterministic runs.

The engine metric contract

For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc

Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.

Module path

gitea.mynodes.xyz/ilker/llmgw. To change it:

make rename NEW=git.example.com/team/llmgw

Language

Code, comments, tests, commit messages and documentation are in English.

S
Description
No description provided
Readme
26 MiB
0 Stars 2 Watchers 1 Forks
v0.16.3
Latest
2026-11-03 09:56:29 +01:00
Languages
Go 81.1%
JavaScript 15.2%
HTML 2.6%
CSS 0.6%
Python 0.2%
Other 0.2%