Say what v0.6.0 is, and let the release actually publish it
The changelog had two "Unreleased" headings. The second sat below v0.5.0 and was the detail behind it — phases 2 and 3, written while they were in progress and never folded in when the release was cut — so it claimed as unreleased work that had been shipped for two days. It is now a section of v0.5.0. The status table said v0.5.0 was planned, which its own release notes disprove. The first "Unreleased" becomes v0.6.0: eighty commits, of which phase 4 is the headline and the rest is what a cluster needs before anyone but its operator is let near it. Two things would have stopped the release workflow publishing anything: Gitea gives every job a built-in GITEA_TOKEN, but what it may do depends on the instance's default mode, and under the restricted default it is read-only on releases. The job now asks for `releases: write`, which is correct under either default. An upload that is refused now fails the job — without `-f`, curl treats a 403 body as a successful transfer, so the release would end up with some of its files, or none, and a green tick either way. `workflow_dispatch` is offered but the upload assumed a tag. Fired from a branch it would have created a release named "main". README's status paragraph still said authentication, budgets and the control plane were missing. It now says what is actually left, including the part worth saying out loud: the InfiniBand, NVML and NCCL paths have never run on a machine that has any of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
0cac9d6986
commit
a2b3f63a0d
3 files changed
+176
-16
No files matched your search
@@ -14,6 +14,12 @@ on:
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
# The built-in token every job is given carries whatever the instance's
|
||||
# default mode allows, and under the restricted default that is read-only
|
||||
# on releases — so the build would succeed and the upload would 403. Asking
|
||||
# for the one scope this needs works under either default.
|
||||
permissions:
|
||||
releases: write
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
@@ -83,12 +89,22 @@ jobs:
|
||||
TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# Gitea gives every job a GITEA_TOKEN, so this is not the usual
|
||||
# "you forgot to add a secret" — an empty one means the job was denied
|
||||
# a token, which is an instance setting rather than a repository one.
|
||||
if [ -z "${TOKEN:-}" ]; then
|
||||
echo "No GITEA_TOKEN secret, so nothing can be uploaded."
|
||||
echo "Add one in the repository's Actions settings with package or"
|
||||
echo "repository write scope; the binaries are in dist/ meanwhile."
|
||||
echo "The job was given no GITEA_TOKEN, so nothing can be uploaded."
|
||||
echo "Check Settings -> Actions -> General; the binaries are in dist/."
|
||||
exit 1
|
||||
fi
|
||||
# workflow_dispatch can be fired from a branch, where GITHUB_REF_NAME
|
||||
# is the branch name. Uploading then would create a release called
|
||||
# "main". The binaries are still built and kept as a check that the
|
||||
# release build works; only the publishing is a tag's business.
|
||||
if [ "${GITHUB_REF_TYPE:-}" != "tag" ]; then
|
||||
echo "not a tag (ref=${GITHUB_REF_NAME}); built, not published"
|
||||
exit 0
|
||||
fi
|
||||
tag="${GITHUB_REF_NAME}"
|
||||
api="${GITHUB_SERVER_URL}/api/v1/repos/${GITHUB_REPOSITORY}"
|
||||
|
||||
@@ -106,8 +122,9 @@ jobs:
|
||||
|
||||
for f in dist/*; do
|
||||
echo "uploading $(basename "$f")"
|
||||
curl -sS -X POST -H "Authorization: token $TOKEN" \
|
||||
curl -sS -f -X POST -H "Authorization: token $TOKEN" \
|
||||
-F "attachment=@$f" \
|
||||
"$api/releases/$id/assets?name=$(basename "$f")" >/dev/null
|
||||
"$api/releases/$id/assets?name=$(basename "$f")" >/dev/null \
|
||||
|| { echo "upload of $(basename "$f") was refused" >&2; exit 1; }
|
||||
done
|
||||
echo "attached $(ls dist | wc -l) files to $tag"
|
||||
+142
-6
@@ -9,14 +9,133 @@ is only released once its **exit criterion** is met, not when its code lands.
|
||||
| v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released |
|
||||
| v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released |
|
||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 |
|
||||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned |
|
||||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned |
|
||||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released |
|
||||
| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
||||
APIs change between minors without a deprecation period.
|
||||
|
||||
## Unreleased
|
||||
## v0.6.0 — 2026-09-18
|
||||
|
||||
Phase 4: a replica can be bigger than one machine, and the fabric it spans
|
||||
decides where it is allowed to go. Around that, everything a cluster needs
|
||||
before anyone other than its operator is let near it — keys, budgets, a way in
|
||||
through a browser, and enough visibility to answer "why is nothing scheduling".
|
||||
|
||||
### Multi-node replicas
|
||||
|
||||
A replica is now a set of shards with ranks, a leader and a bootstrap address,
|
||||
rather than one process. Two invariants are enforced rather than documented: a
|
||||
replica never spans two fabrics, and a tensor-parallel group never mixes GPU
|
||||
classes. Both are the kind of mistake that does not fail — it hangs, or it runs
|
||||
several times slower, and nothing says which.
|
||||
|
||||
A replica missing a shard is not ready. It was previously derived from the
|
||||
shards that had reported, so a two-node replica with one node up looked
|
||||
servable and took traffic into a model that could not answer.
|
||||
|
||||
Adoption after a restart happens one agent at a time, because that is how
|
||||
agents reconnect. Merging them as they arrive is what keeps a multi-node
|
||||
replica whole instead of being torn into fragments that are then killed as
|
||||
strays — and an adopted engine is health-checked before it is called ready,
|
||||
rather than being assumed alive because its process exists.
|
||||
|
||||
### Fabrics are distribution domains too
|
||||
|
||||
An isolated fabric is not only a compute boundary: weights present in one are
|
||||
not present in the other. Node weight inventories are reported and summarised
|
||||
per fabric, so "the model is on the cluster" is never mistaken for "the model
|
||||
is reachable from the nodes that would run it". A fabric name that matches
|
||||
nothing a node actually has is refused at registration rather than becoming an
|
||||
unschedulable replica hours later.
|
||||
|
||||
### Who may use it
|
||||
|
||||
API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so
|
||||
still no cgo). Rate limits and budgets are enforced on the request path; usage
|
||||
is written off it, in batches, so accounting never adds latency to inference.
|
||||
Budgets survive a restart — counters are rebuilt from the store, because a
|
||||
limit anyone can clear by bouncing the process is not a limit.
|
||||
|
||||
A spent budget is now served a cheaper model where the catalog offers one,
|
||||
rather than refused. For a team trying models out, being dropped to something
|
||||
smaller is a far better answer than a wall, and the response says which model
|
||||
answered.
|
||||
|
||||
People can sign up, be approved, and issue and revoke their own keys. A key is
|
||||
shown once; only its hash is kept.
|
||||
|
||||
### The browser surface
|
||||
|
||||
A landing page that says how to connect, a read-only cluster dashboard, and an
|
||||
in-browser chat that never puts an API key in the browser. The dashboard shows
|
||||
who is using the cluster and what they are running — per person and per model,
|
||||
with requests, tokens, errors, spend and latency.
|
||||
|
||||
All of it is served from the client-facing port behind a session, deliberately
|
||||
not from the admin port, which also serves pprof — and pprof can dump the heap
|
||||
that holds every API key in plaintext.
|
||||
|
||||
### Models without a restart
|
||||
|
||||
The catalog is live. Models can be added and removed from the command line or
|
||||
the API, take effect immediately, and outlive a restart; configuration still
|
||||
wins over the store, and the whole catalog is validated before anything is
|
||||
written.
|
||||
|
||||
A model is only listed once it can actually serve. It was listed as soon as it
|
||||
was known, so the first thing a new user did was ask for a model that returned
|
||||
503.
|
||||
|
||||
### Loading on demand, and letting go
|
||||
|
||||
A model whose scaling allows it is started when someone asks for it, and
|
||||
stopped when nobody has for a while. An idle replica is evicted to make room
|
||||
for one that was asked for, with guards that exist because each is a way to
|
||||
lose work quietly: only on-demand models, only single-node replicas, never one
|
||||
that has never been used, and never one younger than its minimum lifetime.
|
||||
|
||||
Unknown is not idle. A replica with no recorded use is left alone rather than
|
||||
treated as idle since the epoch.
|
||||
|
||||
### Knowing whether it will run before it runs
|
||||
|
||||
- `ctl plan` works out whether a model fits the cluster and how it would be
|
||||
split, in the precision the chosen engine can actually serve — recommending
|
||||
a quantisation the engine cannot load would cause the failure the command
|
||||
exists to prevent.
|
||||
- `ctl tool-check` verifies llmgw's engine flags against the engines actually
|
||||
installed. Flags drift between releases, and the alternative is finding out
|
||||
during a model load.
|
||||
- Placement now knows a model's context length and memory footprint, and node
|
||||
RAM, so a CPU profile is not placed on a machine that cannot hold it.
|
||||
- Placement waits for the fleet to settle after a restart rather than deciding
|
||||
against a half-reported cluster.
|
||||
|
||||
### Weights
|
||||
|
||||
Nodes report the model files they can see, without llmgw owning the storage:
|
||||
a shared filer, an image exported over the fabric, rsync or a copy made by hand
|
||||
all look the same to placement, which only cares whether the files are there.
|
||||
Reporting nothing means "not asked to look", not "has nothing".
|
||||
|
||||
Where the simple arrangement is not enough, a node can fetch a model by URL —
|
||||
single-flight, resumable, written to a temporary name and renamed atomically,
|
||||
so a killed download never leaves something that looks like weights.
|
||||
|
||||
### Seeing the fleet
|
||||
|
||||
Prometheus metrics for nodes, GPUs and replicas, collected from live state at
|
||||
scrape time rather than from a background goroutine, and served on their own
|
||||
endpoint rather than through the one that also exposes pprof. The same view is
|
||||
in the dashboard, so nothing else needs installing to see it.
|
||||
|
||||
### Onboarding a node
|
||||
|
||||
Adding a node is one command. There are systemd units, a private CA and a
|
||||
from-nothing install guide that starts with a machine that has no CUDA and no
|
||||
binary.
|
||||
|
||||
### llama.cpp, and CPU profiles
|
||||
|
||||
@@ -63,6 +182,23 @@ agent turn it on.
|
||||
- **A broken metric contract was invisible.** It was logged per scrape at debug
|
||||
level, so a cluster could look healthy while every routing decision was made
|
||||
blind. It is now a warning, once, with a matching line when it clears.
|
||||
- **Token estimates came from the routing prefix, not the prompt.** Every prompt
|
||||
over about 2 KB was undercounted, so tokens-per-minute limits let through far
|
||||
more than they were set to.
|
||||
- **Model-fitting double-counted overhead.** Whole-model overhead was charged per
|
||||
rank and a 10% reserve taken twice, so `plan` reported that a 70B model in
|
||||
bfloat16 does not fit two 80 GB cards. It does.
|
||||
- **`x-llmgw-served-model` was never sent.** It was set after the proxy had
|
||||
already written the response header, which is the one point at which headers
|
||||
stop being changeable. A client had no way to tell a fallback from the model
|
||||
it asked for.
|
||||
- **Fleet metrics published nothing.** They were registered after the registry
|
||||
had been built, so the collector existed and was never scraped.
|
||||
- **GPU discovery broke on older drivers.** A field added for utilisation was
|
||||
required rather than optional, so a driver that does not report it produced no
|
||||
inventory at all instead of an inventory without that number.
|
||||
- **Replica ids and ports were reused** while the processes holding them were
|
||||
still alive.
|
||||
|
||||
## v0.5.0 — 2026-09-16
|
||||
|
||||
@@ -98,10 +234,10 @@ not have. The failure path already ships an engine log tail with every status
|
||||
report, so the case that matters most is covered; reading a healthy shard's
|
||||
output on demand is not.
|
||||
|
||||
## Unreleased — phase 4 in progress
|
||||
### Phases 2 and 3, in detail
|
||||
|
||||
Agent groundwork. Everything here works without GPUs and is covered by tests;
|
||||
what is missing is the control plane connection, so there is no release yet.
|
||||
Written as the work landed, which is why it reads as a log rather than as
|
||||
notes. Everything here shipped in v0.5.0.
|
||||
|
||||
### Added
|
||||
|
||||
|
||||
@@ -29,13 +29,20 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md).
|
||||
| v0.3.0 | 1.5 | Admission control (priority bands) | released |
|
||||
| v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released |
|
||||
| v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released |
|
||||
| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned |
|
||||
| v0.6.0 | 4 | Multi-node placement and fabric awareness | released |
|
||||
| v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned |
|
||||
|
||||
As of v0.3.0 the router serves traffic and protects it: a batch workload no
|
||||
longer starves interactive requests. Still missing before v1.0.0: authentication,
|
||||
rate limits and budgets, and the control plane that replaces the static endpoint
|
||||
list.
|
||||
As of v0.6.0 a cluster runs unattended: agents report their hardware, the
|
||||
control plane places replicas — including ones larger than a single machine —
|
||||
models load when someone asks for them and stop when nobody does, and keys,
|
||||
budgets and rate limits decide who may use any of it.
|
||||
|
||||
Before v1.0.0: routing profiles selectable per request, spillover to a fallback
|
||||
pool, tracing, and reference documentation. One thing is worth saying plainly:
|
||||
multi-node replicas are proven on a simulated cluster and on real hardware
|
||||
without a fabric. The InfiniBand, NVML and NCCL paths are written against the
|
||||
formats and failure modes those tools actually have, but they have not yet run
|
||||
on a machine that has them.
|
||||
|
||||
## Building
|
||||
|
||||
|
||||
Reference in new issue
Block a user