Say what v0.6.0 is, and let the release actually publish it
CI / test (push) Successful in 1m56s
Release / build (push) Failing after 2m31s

The changelog had two "Unreleased" headings. The second sat below v0.5.0 and
was the detail behind it — phases 2 and 3, written while they were in progress
and never folded in when the release was cut — so it claimed as unreleased
work that had been shipped for two days. It is now a section of v0.5.0. The
status table said v0.5.0 was planned, which its own release notes disprove.

The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the
headline and the rest is what a cluster needs before anyone but its operator
is let near it.

Two things would have stopped the release workflow publishing anything:

Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on
the instance's default mode, and under the restricted default it is read-only
on releases. The job now asks for `releases: write`, which is correct under
either default. An upload that is refused now fails the job — without `-f`,
curl treats a 403 body as a successful transfer, so the release would end up
with some of its files, or none, and a green tick either way.

`workflow_dispatch` is offered but the upload assumed a tag. Fired from a
branch it would have created a release named "main".

README's status paragraph still said authentication, budgets and the control
plane were missing. It now says what is actually left, including the part
worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on
a machine that has any of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 00:29:15 +02:00
1 parent 0cac9d6986
commit a2b3f63a0d
3 files changed
+176 -16

No files matched your search

+22 -5
View File
@@ -14,6 +14,12 @@ on:
jobs:
build:
runs-on: ubuntu-latest
# The built-in token every job is given carries whatever the instance's
# default mode allows, and under the restricted default that is read-only
# on releases — so the build would succeed and the upload would 403. Asking
# for the one scope this needs works under either default.
permissions:
releases: write
steps:
- uses: actions/checkout@v4
with:
@@ -83,12 +89,22 @@ jobs:
TOKEN: ${{ secrets.GITEA_TOKEN }}
run: |
set -euo pipefail
# Gitea gives every job a GITEA_TOKEN, so this is not the usual
# "you forgot to add a secret" — an empty one means the job was denied
# a token, which is an instance setting rather than a repository one.
if [ -z "${TOKEN:-}" ]; then
echo "No GITEA_TOKEN secret, so nothing can be uploaded."
echo "Add one in the repository's Actions settings with package or"
echo "repository write scope; the binaries are in dist/ meanwhile."
echo "The job was given no GITEA_TOKEN, so nothing can be uploaded."
echo "Check Settings -> Actions -> General; the binaries are in dist/."
exit 1
fi
# workflow_dispatch can be fired from a branch, where GITHUB_REF_NAME
# is the branch name. Uploading then would create a release called
# "main". The binaries are still built and kept as a check that the
# release build works; only the publishing is a tag's business.
if [ "${GITHUB_REF_TYPE:-}" != "tag" ]; then
echo "not a tag (ref=${GITHUB_REF_NAME}); built, not published"
exit 0
fi
tag="${GITHUB_REF_NAME}"
api="${GITHUB_SERVER_URL}/api/v1/repos/${GITHUB_REPOSITORY}"
@@ -106,8 +122,9 @@ jobs:
for f in dist/*; do
echo "uploading $(basename "$f")"
curl -sS -X POST -H "Authorization: token $TOKEN" \
curl -sS -f -X POST -H "Authorization: token $TOKEN" \
-F "attachment=@$f" \
"$api/releases/$id/assets?name=$(basename "$f")" >/dev/null
"$api/releases/$id/assets?name=$(basename "$f")" >/dev/null \
|| { echo "upload of $(basename "$f") was refused" >&2; exit 1; }
done
echo "attached $(ls dist | wc -l) files to $tag"
+142 -6
View File
@@ -9,14 +9,133 @@ is only released once its **exit criterion** is met, not when its code lands.
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
APIs change between minors without a deprecation period.
## Unreleased
## v0.6.0 — 2026-09-18
Phase 4: a replica can be bigger than one machine, and the fabric it spans
decides where it is allowed to go. Around that, everything a cluster needs
before anyone other than its operator is let near it — keys, budgets, a way in
through a browser, and enough visibility to answer "why is nothing scheduling".
### Multi-node replicas
A replica is now a set of shards with ranks, a leader and a bootstrap address,
rather than one process. Two invariants are enforced rather than documented: a
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
several times slower, and nothing says which.
A replica missing a shard is not ready. It was previously derived from the
shards that had reported, so a two-node replica with one node up looked
servable and took traffic into a model that could not answer.
Adoption after a restart happens one agent at a time, because that is how
agents reconnect. Merging them as they arrive is what keeps a multi-node
replica whole instead of being torn into fragments that are then killed as
strays — and an adopted engine is health-checked before it is called ready,
rather than being assumed alive because its process exists.
### Fabrics are distribution domains too
An isolated fabric is not only a compute boundary: weights present in one are
not present in the other. Node weight inventories are reported and summarised
per fabric, so "the model is on the cluster" is never mistaken for "the model
is reachable from the nodes that would run it". A fabric name that matches
nothing a node actually has is refused at registration rather than becoming an
unschedulable replica hours later.
### Who may use it
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
still no cgo). Rate limits and budgets are enforced on the request path; usage
is written off it, in batches, so accounting never adds latency to inference.
Budgets survive a restart — counters are rebuilt from the store, because a
limit anyone can clear by bouncing the process is not a limit.
A spent budget is now served a cheaper model where the catalog offers one,
rather than refused. For a team trying models out, being dropped to something
smaller is a far better answer than a wall, and the response says which model
answered.
People can sign up, be approved, and issue and revoke their own keys. A key is
shown once; only its hash is kept.
### The browser surface
A landing page that says how to connect, a read-only cluster dashboard, and an
in-browser chat that never puts an API key in the browser. The dashboard shows
who is using the cluster and what they are running — per person and per model,
with requests, tokens, errors, spend and latency.
All of it is served from the client-facing port behind a session, deliberately
not from the admin port, which also serves pprof — and pprof can dump the heap
that holds every API key in plaintext.
### Models without a restart
The catalog is live. Models can be added and removed from the command line or
the API, take effect immediately, and outlive a restart; configuration still
wins over the store, and the whole catalog is validated before anything is
written.
A model is only listed once it can actually serve. It was listed as soon as it
was known, so the first thing a new user did was ask for a model that returned
503.
### Loading on demand, and letting go
A model whose scaling allows it is started when someone asks for it, and
stopped when nobody has for a while. An idle replica is evicted to make room
for one that was asked for, with guards that exist because each is a way to
lose work quietly: only on-demand models, only single-node replicas, never one
that has never been used, and never one younger than its minimum lifetime.
Unknown is not idle. A replica with no recorded use is left alone rather than
treated as idle since the epoch.
### Knowing whether it will run before it runs
- `ctl plan` works out whether a model fits the cluster and how it would be
split, in the precision the chosen engine can actually serve — recommending
a quantisation the engine cannot load would cause the failure the command
exists to prevent.
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
installed. Flags drift between releases, and the alternative is finding out
during a model load.
- Placement now knows a model's context length and memory footprint, and node
RAM, so a CPU profile is not placed on a machine that cannot hold it.
- Placement waits for the fleet to settle after a restart rather than deciding
against a half-reported cluster.
### Weights
Nodes report the model files they can see, without llmgw owning the storage:
a shared filer, an image exported over the fabric, rsync or a copy made by hand
all look the same to placement, which only cares whether the files are there.
Reporting nothing means "not asked to look", not "has nothing".
Where the simple arrangement is not enough, a node can fetch a model by URL —
single-flight, resumable, written to a temporary name and renamed atomically,
so a killed download never leaves something that looks like weights.
### Seeing the fleet
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
scrape time rather than from a background goroutine, and served on their own
endpoint rather than through the one that also exposes pprof. The same view is
in the dashboard, so nothing else needs installing to see it.
### Onboarding a node
Adding a node is one command. There are systemd units, a private CA and a
from-nothing install guide that starts with a machine that has no CUDA and no
binary.
### llama.cpp, and CPU profiles
@@ -63,6 +182,23 @@ agent turn it on.
- **A broken metric contract was invisible.** It was logged per scrape at debug
level, so a cluster could look healthy while every routing decision was made
blind. It is now a warning, once, with a matching line when it clears.
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
over about 2 KB was undercounted, so tokens-per-minute limits let through far
more than they were set to.
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
bfloat16 does not fit two 80 GB cards. It does.
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
already written the response header, which is the one point at which headers
stop being changeable. A client had no way to tell a fallback from the model
it asked for.
- **Fleet metrics published nothing.** They were registered after the registry
had been built, so the collector existed and was never scraped.
- **GPU discovery broke on older drivers.** A field added for utilisation was
required rather than optional, so a driver that does not report it produced no
inventory at all instead of an inventory without that number.
- **Replica ids and ports were reused** while the processes holding them were
still alive.
## v0.5.0 — 2026-09-16
@@ -98,10 +234,10 @@ not have. The failure path already ships an engine log tail with every status
report, so the case that matters most is covered; reading a healthy shard's
output on demand is not.
## Unreleased — phase 4 in progress
### Phases 2 and 3, in detail
Agent groundwork. Everything here works without GPUs and is covered by tests;
what is missing is the control plane connection, so there is no release yet.
Written as the work landed, which is why it reads as a log rather than as
notes. Everything here shipped in v0.5.0.
### Added
+12 -5
View File
@@ -29,13 +29,20 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
As of v0.3.0 the router serves traffic and protects it: a batch workload no
longer starves interactive requests. Still missing before v1.0.0: authentication,
rate limits and budgets, and the control plane that replaces the static endpoint
list.
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
control plane places replicas — including ones larger than a single machine —
models load when someone asks for them and stop when nobody does, and keys,
budgets and rate limits decide who may use any of it.
Before v1.0.0: routing profiles selectable per request, spillover to a fallback
pool, tracing, and reference documentation. One thing is worth saying plainly:
multi-node replicas are proven on a simulated cluster and on real hardware
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
formats and failure modes those tools actually have, but they have not yet run
on a machine that has them.
## Building