Ilker ManapandClaude Opus 5 55262418c9
CI / test (push) Successful in 1m35s
Put quotas somewhere they can be seen and changed
A quota could be enforced and reported, and set nowhere but a shell. Nothing in
the browser showed one at all — not what a team may spend, not what it has
spent, not that it had no limit. In practice that means quotas nobody sets.

/admin/quotas lists the whole tree, whether or not a group has a limit: the one
without a quota is the one most likely to need one, so it cannot be the row the
page leaves out. Spend over the last thirty days sits beside the budget, with
the fraction used.

Setting a quota here is a change of position. It was kept to the shell because
raising a budget is a financial act and a stolen admin session should not
perform one. That argument holds for exactly one row on this page: resolution
takes the most restrictive value along the chain, so raising a team's or a
project's limit cannot exceed what the organization allows. Only an
organization's own limit is bounded by nothing. Every change is logged with who
made it and what it was before.

Per-key limits stay on the CLI. A key belongs to a person and is already
bounded by its project; letting an admin session reach individual keys would
put every caller's throughput one click away, and the page refuses a key scope
rather than rendering it and failing later.

A limit an ancestor already caps lower is marked as written-but-unreachable.
Resolution means a project given more than its team is not a bigger budget, it
is a number nobody will reach — and without saying so the page would show it as
though it applied, with the first sign being a refusal that disagrees with the
screen.

The group is chosen, not typed. Ids became random in the previous commit so two
organizations could both have a "platform" team, which also made them
impossible to remember and a typo impossible to notice: it would write a quota
onto nothing and report success.

A blank field is no limit of that kind, not zero — zero is a limit of nothing,
which refuses every request. Clearing removes the row rather than writing nulls
across it: both resolve the same, only one tells a reader nobody has set a limit
here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-27 14:44:13 +02:00
2026-09-16 01:00:03 +02:00
2026-09-27 07:26:54 +02:00

llmgw

A single-binary LLM gateway for clusters with multiple InfiniBand fabrics and heterogeneous GPUs.

It serves LLMs running on your own GPU servers (vLLM / SGLang) behind one OpenAI-compatible endpoint. Design document: llmgw-plan.md.

Getting started

Starting from nothing — no packages installed, no drivers, no nodes — follow docs/install.md: it builds a control plane, adds a node and serves a model, and says plainly which parts llmgw does not install for you.

Guide For
docs/install.md Standing up a cluster from scratch
docs/node-onboarding.md Adding nodes to one that exists
docs/onboarding.md Giving people accounts and keys

Status

Releases track the roadmap phases; see CHANGELOG.md.

Version Phase Scope State
v0.1.0 0 Skeleton and sim-engine released
v0.2.0 1 Static router (filter/score/pick, streaming proxy) released
v0.3.0 1.5 Admission control (priority bands) released
v0.4.0 2 Agent: GPU/IB discovery, engine supervision released
v0.5.0 3 Control plane: catalog, placement, reconciler, ctl released
v0.6.0 4 Multi-node placement and fabric awareness released
v0.7.0 — Conversations, agent memory, disk management released
v0.8.0 — PostgreSQL, leader election, agent failover released
v1.0.0 5 Tenancy, prefix affinity, benchmarking planned

As of v0.6.0 a cluster runs unattended: agents report their hardware, the control plane places replicas — including ones larger than a single machine — models load when someone asks for them and stop when nobody does, and keys, budgets and rate limits decide who may use any of it.

Before v1.0.0: tracing, reference documentation, and filtering on ctl nodes and ctl replicas for fleets too large to read a whole table of. Two items that were on this list are not gaps: a model falls through to its fallbacks when there is no capacity as well as when a budget is spent, and routing profiles are chosen by the key rather than by a request header on purpose — a header would let any client promote itself into the top priority band.

One thing is worth saying plainly: multi-node replicas are proven on a simulated cluster and on real hardware without a fabric. The InfiniBand, NVML and NCCL paths are written against the formats and failure modes those tools actually have, but they have not yet run on a machine that has them.

Documentation

  • docs/install.md — installing from nothing: control plane, nodes, engines, weights, first request
  • docs/quickstart.md — the short path
  • docs/choosing-models.md — which models to run, how many of each, and where the weights should live
  • docs/data.md — what llmgw stores, what it does not, and what leaves the machine; written for a security review
  • docs/reference.md — every configuration key, generated from the structs and checked by a test, so it cannot describe a setting that does not exist or miss one that does
  • deploy/mirror/ — a model mirror, which is the cheapest and most useful thing to build around llmgw
  • deploy/ha/ — running two control planes, and the one health check that has to be right
  • docs/node-onboarding.md, docs/onboarding.md — adding machines, and adding people

Building

make build      # bin/llmgw
make test       # unit tests
make race       # with the race detector

Operating a cluster

llmgw ctl nodes                      # capacity, health, what is running where
llmgw ctl models                     # catalog and replica counts
llmgw ctl replicas --model llama     # replicas and their shards
llmgw ctl explain-placement llama    # why a model has not been placed
llmgw ctl drain gpu-a-01             # move work off a node, one replica at a time
llmgw ctl cordon gpu-a-01            # stop new placement, leave what runs

ctl talks to the admin port (:9090 by default), not the client-facing one. Add --json to any command for machine-readable output.

explain-placement is the one worth knowing about: "unschedulable" on its own says nothing about which of six constraints bit, so it prints every candidate considered and the reason it was rejected.

Running the router

# Two fake engines and a router in front of them.
./bin/llmgw sim-engine --listen :8001 --model llama &
./bin/llmgw sim-engine --listen :8002 --model llama &
./bin/llmgw server \
  --endpoint e1=http://127.0.0.1:8001=llama \
  --endpoint e2=http://127.0.0.1:8002=llama

curl -N localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -H 'x-session-id: demo' \
  -d '{"model":"llama","messages":[{"role":"user","content":"hello"}],"stream":true}'

For anything real use a config file; deploy/examples/server.yaml is commented throughout. Metrics, health and pprof are on the admin port (:9090 by default), deliberately separate from the client-facing one.

Requests carry a priority band (x-llmgw-band). Under load, low bands wait at the gateway while high bands keep flowing, so the cluster degrades from the bottom up instead of everywhere at once. Measured effect: interactive p99 stays within 1.24x of idle while 500 concurrent batch requests run, against 58x with admission off.

Routing profiles select how endpoints are scored: chat favours session affinity, batch favours the shortest queue, long-context favours free KV cache. Pick one per model in the config, or per request with the x-llmgw-profile header.

sim-engine

Simulates the inference-engine behaviours the router cares about, so the router can be developed and tested without GPUs. What it models:

  • continuous batching (bounded number of concurrent sequences)
  • KV cache block accounting, queueing once blocks run out (head-of-line blocking included)
  • automatic prefix caching (chained block hashes plus an LRU) and its effect on time to first token
  • per-token latency growing with batch size
  • dropping a request and freeing its blocks when the client disconnects
  • /metrics using vLLM's metric names
./bin/llmgw sim-engine --listen :8000 --model llama-3.3-70b

curl -N localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"hello"}],"max_tokens":8,"stream":true}'

Tune load and latency behaviour with --max-num-seqs, --num-gpu-blocks, --prefill-tps, --decode-tps, --batch-contention and --prefix-cache-blocks. For chaos testing use --fail-rate, and --seed for deterministic runs.

The engine metric contract

For the router to make routing decisions an engine must expose these three metrics (plan §7.6). sim-engine publishes them under vLLM's real names:

vllm:num_requests_running
vllm:num_requests_waiting
vllm:kv_cache_usage_perc

Adding support for a new engine means satisfying that contract. Metric names drift between versions, so the name mapping is configurable.

Module path

gitea.mynodes.xyz/ilker/llmgw. To change it:

make rename NEW=git.example.com/team/llmgw

Language

Code, comments, tests, commit messages and documentation are in English.

S
Description
No description provided
Readme
26 MiB
0 Stars 2 Watchers 1 Forks
v0.16.3
Latest
2026-11-03 09:56:29 +01:00
Languages
Go 81.1%
JavaScript 15.2%
HTML 2.6%
CSS 0.6%
Python 0.2%
Other 0.2%