Ilker ManapandClaude Opus 5
dfbb7557dd
The mirror would happily hold a four-shard model that nothing could serve. A catalog entry names one source, the agent fetched one URL, and so a version of several files had no way to be described — which is every safetensors model, and every GGUF too large to ship whole. A source ending in a slash names a version rather than a file. The agent fetches every file in it into one directory and hands the engine the directory: vLLM reads the shards and the config out of the folder, and llama.cpp is given the first part and finds the rest beside it. The trailing slash is the whole signal. A flag beside the source would be a second place for the two to disagree, and a URL already distinguishes a directory from a file everywhere else. The mirror answers a version with a manifest instead of a file listing. A plain file server gives HTML there, and something that has to know what a version consists of would be left parsing it. It is on the byte listener, not the management API: fetching weights is the one thing every node does, and making it depend on the administrative port would undo the split that keeps that port closeable. Built from the directory rather than a file written beside it. A manifest on disk is a second record of what is there, and the two disagree the first time somebody copies a file in by hand — which the mirror explicitly allows. The whole set or nothing. A directory with three of four shards loads as a broken model rather than failing to load, so a completion marker is written last and an interrupted fetch starts the missing files again rather than being found and served. Files already there at the right size are not fetched twice. Progress is reported under the version's address, not each file's: the shards waiting on it asked for the version, and progress under a file's own URL reaches nobody. A filename from the manifest is refused if it contains a separator. The mirror is a place an operator copies files into by hand, so what it reports is not automatically safe to join onto a path. An HTML listing where a manifest was expected says so. "invalid character '<'" sends somebody looking at the wrong thing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
llmgw
A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.
It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.
Getting started
Starting from nothing — no packages installed, no drivers, no nodes — follow docs/install.md: it builds a control plane, adds a node and serves a model, and says plainly which parts llmgw does not install for you.
| Guide | For |
|---|---|
| docs/install.md | Standing up a cluster from scratch |
| docs/node-onboarding.md | Adding nodes to one that exists |
| docs/onboarding.md | Giving people accounts and keys |
Status
Releases track the roadmap phases; see CHANGELOG.md.
| Version | Phase | Scope | State |
|---|---|---|---|
| v0.1.0 | 0 | Skeleton and sim-engine | released |
| v0.2.0 | 1 | Static router (filter/score/pick, streaming proxy) | released |
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
| v0.7.0 | — | Conversations, agent memory, disk management | released |
| v0.8.0 | — | PostgreSQL, leader election, agent failover | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.6.0 a cluster runs unattended: agents report their hardware, the control plane places replicas — including ones larger than a single machine — models load when someone asks for them and stop when nobody does, and keys, budgets and rate limits decide who may use any of it.
Before v1.0.0: tracing, reference documentation, and filtering on ctl nodes
and ctl replicas for fleets too large to read a whole table of. Two items
that were on this list are not gaps: a model falls through to its fallbacks
when there is no capacity as well as when a budget is spent, and routing
profiles are chosen by the key rather than by a request header on purpose —
a header would let any client promote itself into the top priority band.
One thing is worth saying plainly: multi-node replicas are proven on a simulated cluster and on real hardware without a fabric. The InfiniBand, NVML and NCCL paths are written against the formats and failure modes those tools actually have, but they have not yet run on a machine that has them.
Documentation
- docs/install.md — installing from nothing: control plane, nodes, engines, weights, first request
- docs/quickstart.md — the short path
- docs/choosing-models.md — which models to run, how many of each, and where the weights should live
- docs/data.md — what llmgw stores, what it does not, and what leaves the machine; written for a security review
- docs/reference.md — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does
- deploy/mirror/ — a model mirror, which is the cheapest and most useful thing to build around llmgw
- deploy/ha/ — running two control planes, and the one health check that has to be right
- docs/node-onboarding.md, docs/onboarding.md — adding machines, and adding people
Building
make build # bin/llmgw
make test # unit tests
make race # with the race detector
Operating a cluster
llmgw ctl nodes # capacity, health, what is running where
llmgw ctl models # catalog and replica counts
llmgw ctl replicas --model llama # replicas and their shards
llmgw ctl explain-placement llama # why a model has not been placed
llmgw ctl drain gpu-a-01 # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01 # stop new placement, leave what runs
ctl talks to the admin port (:9090 by default), not the client-facing one.
Add --json to any command for machine-readable output.
explain-placement is the one worth knowing about: "unschedulable" on its own
says nothing about which of six constraints bit, so it prints every candidate
considered and the reason it was rejected.
Running the router
# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
--endpoint e1=http://127.0.0.1:8001=llama \
--endpoint e2=http://127.0.0.1:8002=llama
curl -N localhost:8080/v1/chat/completions \
-H 'content-type: application/json' \
-H 'x-session-id: demo' \
-d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'
For anything real use a config file; deploy/examples/server.yaml
is commented throughout. Metrics, health and pprof are on the admin port (:9090
by default), deliberately separate from the client-facing one.
Requests carry a priority band (x-llmgw-band). Under load, low bands wait at
the gateway while high bands keep flowing, so the cluster degrades from the
bottom up instead of everywhere at once. Measured effect: interactive p99 stays
within 1.24x of idle while 500 concurrent batch requests run, against 58x with
admission off.
Routing profiles select how endpoints are scored: chat favours session
affinity, batch favours the shortest queue, long-context favours free KV
cache. Pick one per model in the config, or per request with the
x-llmgw-profile header.
sim-engine
Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:
- continuous batching (bounded number of concurrent sequences)
- KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
- automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
- per-token latency growing with batch size
- dropping a request and freeing its blocks when the client disconnects
/metricsusing vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b
curl -N localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'
Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks,
--prefill-tps, --decode-tps, --batch-contention and
--prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for
deterministic runs.
The engine metric contract
For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:
vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc
Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.
Module path
gitea.mynodes.xyz/ilker/llmgw. To change it:
make rename NEW=git.example.com/team/llmgw
Language
Code, comments, tests, commit messages and documentation are in English.