diff --git a/.gitea/workflows/release.yml b/.gitea/workflows/release.yml index ced26b7..de8ddf9 100644 --- a/.gitea/workflows/release.yml +++ b/.gitea/workflows/release.yml @@ -14,6 +14,12 @@ on: jobs: build: runs-on: ubuntu-latest + # The built-in token every job is given carries whatever the instance's + # default mode allows, and under the restricted default that is read-only + # on releases — so the build would succeed and the upload would 403. Asking + # for the one scope this needs works under either default. + permissions: + releases: write steps: - uses: actions/checkout@v4 with: @@ -83,12 +89,22 @@ jobs: TOKEN: ${{ secrets.GITEA_TOKEN }} run: | set -euo pipefail + # Gitea gives every job a GITEA_TOKEN, so this is not the usual + # "you forgot to add a secret" — an empty one means the job was denied + # a token, which is an instance setting rather than a repository one. if [ -z "${TOKEN:-}" ]; then - echo "No GITEA_TOKEN secret, so nothing can be uploaded." - echo "Add one in the repository's Actions settings with package or" - echo "repository write scope; the binaries are in dist/ meanwhile." + echo "The job was given no GITEA_TOKEN, so nothing can be uploaded." + echo "Check Settings -> Actions -> General; the binaries are in dist/." exit 1 fi + # workflow_dispatch can be fired from a branch, where GITHUB_REF_NAME + # is the branch name. Uploading then would create a release called + # "main". The binaries are still built and kept as a check that the + # release build works; only the publishing is a tag's business. + if [ "${GITHUB_REF_TYPE:-}" != "tag" ]; then + echo "not a tag (ref=${GITHUB_REF_NAME}); built, not published" + exit 0 + fi tag="${GITHUB_REF_NAME}" api="${GITHUB_SERVER_URL}/api/v1/repos/${GITHUB_REPOSITORY}" @@ -106,8 +122,9 @@ jobs: for f in dist/*; do echo "uploading $(basename "$f")" - curl -sS -X POST -H "Authorization: token $TOKEN" \ + curl -sS -f -X POST -H "Authorization: token $TOKEN" \ -F "attachment=@$f" \ - "$api/releases/$id/assets?name=$(basename "$f")" >/dev/null + "$api/releases/$id/assets?name=$(basename "$f")" >/dev/null \ + || { echo "upload of $(basename "$f") was refused" >&2; exit 1; } done echo "attached $(ls dist | wc -l) files to $tag" diff --git a/CHANGELOG.md b/CHANGELOG.md index e57f57a..71463a9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,14 +9,133 @@ is only released once its **exit criterion** is met, not when its code lands. | v0.2.0 | 1 | Static router: streaming proxy, filter/score/pick | released | | v0.3.0 | 1.5 | Admission control: priority bands, saturation detector | released | | v0.4.0 | 2 | Agent: GPU/IB discovery, single-node engines | released as part of v0.5.0 | -| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | planned | -| v0.6.0 | 4 | Multi-node replicas and fabric awareness | planned | +| v0.5.0 | 3 | Control plane: catalog, placement, reconciler | released | +| v0.6.0 | 4 | Multi-node replicas and fabric awareness | released | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | Until v1.0.0 the minor version is the breaking one: configuration formats and Go APIs change between minors without a deprecation period. -## Unreleased +## v0.6.0 — 2026-09-18 + +Phase 4: a replica can be bigger than one machine, and the fabric it spans +decides where it is allowed to go. Around that, everything a cluster needs +before anyone other than its operator is let near it — keys, budgets, a way in +through a browser, and enough visibility to answer "why is nothing scheduling". + +### Multi-node replicas + +A replica is now a set of shards with ranks, a leader and a bootstrap address, +rather than one process. Two invariants are enforced rather than documented: a +replica never spans two fabrics, and a tensor-parallel group never mixes GPU +classes. Both are the kind of mistake that does not fail — it hangs, or it runs +several times slower, and nothing says which. + +A replica missing a shard is not ready. It was previously derived from the +shards that had reported, so a two-node replica with one node up looked +servable and took traffic into a model that could not answer. + +Adoption after a restart happens one agent at a time, because that is how +agents reconnect. Merging them as they arrive is what keeps a multi-node +replica whole instead of being torn into fragments that are then killed as +strays — and an adopted engine is health-checked before it is called ready, +rather than being assumed alive because its process exists. + +### Fabrics are distribution domains too + +An isolated fabric is not only a compute boundary: weights present in one are +not present in the other. Node weight inventories are reported and summarised +per fabric, so "the model is on the cluster" is never mistaken for "the model +is reachable from the nodes that would run it". A fabric name that matches +nothing a node actually has is refused at registration rather than becoming an +unschedulable replica hours later. + +### Who may use it + +API keys, tenants and per-model allowlists, on SQLite (`modernc.org/sqlite`, so +still no cgo). Rate limits and budgets are enforced on the request path; usage +is written off it, in batches, so accounting never adds latency to inference. +Budgets survive a restart — counters are rebuilt from the store, because a +limit anyone can clear by bouncing the process is not a limit. + +A spent budget is now served a cheaper model where the catalog offers one, +rather than refused. For a team trying models out, being dropped to something +smaller is a far better answer than a wall, and the response says which model +answered. + +People can sign up, be approved, and issue and revoke their own keys. A key is +shown once; only its hash is kept. + +### The browser surface + +A landing page that says how to connect, a read-only cluster dashboard, and an +in-browser chat that never puts an API key in the browser. The dashboard shows +who is using the cluster and what they are running — per person and per model, +with requests, tokens, errors, spend and latency. + +All of it is served from the client-facing port behind a session, deliberately +not from the admin port, which also serves pprof — and pprof can dump the heap +that holds every API key in plaintext. + +### Models without a restart + +The catalog is live. Models can be added and removed from the command line or +the API, take effect immediately, and outlive a restart; configuration still +wins over the store, and the whole catalog is validated before anything is +written. + +A model is only listed once it can actually serve. It was listed as soon as it +was known, so the first thing a new user did was ask for a model that returned +503. + +### Loading on demand, and letting go + +A model whose scaling allows it is started when someone asks for it, and +stopped when nobody has for a while. An idle replica is evicted to make room +for one that was asked for, with guards that exist because each is a way to +lose work quietly: only on-demand models, only single-node replicas, never one +that has never been used, and never one younger than its minimum lifetime. + +Unknown is not idle. A replica with no recorded use is left alone rather than +treated as idle since the epoch. + +### Knowing whether it will run before it runs + +- `ctl plan` works out whether a model fits the cluster and how it would be + split, in the precision the chosen engine can actually serve — recommending + a quantisation the engine cannot load would cause the failure the command + exists to prevent. +- `ctl tool-check` verifies llmgw's engine flags against the engines actually + installed. Flags drift between releases, and the alternative is finding out + during a model load. +- Placement now knows a model's context length and memory footprint, and node + RAM, so a CPU profile is not placed on a machine that cannot hold it. +- Placement waits for the fleet to settle after a restart rather than deciding + against a half-reported cluster. + +### Weights + +Nodes report the model files they can see, without llmgw owning the storage: +a shared filer, an image exported over the fabric, rsync or a copy made by hand +all look the same to placement, which only cares whether the files are there. +Reporting nothing means "not asked to look", not "has nothing". + +Where the simple arrangement is not enough, a node can fetch a model by URL — +single-flight, resumable, written to a temporary name and renamed atomically, +so a killed download never leaves something that looks like weights. + +### Seeing the fleet + +Prometheus metrics for nodes, GPUs and replicas, collected from live state at +scrape time rather than from a background goroutine, and served on their own +endpoint rather than through the one that also exposes pprof. The same view is +in the dashboard, so nothing else needs installing to see it. + +### Onboarding a node + +Adding a node is one command. There are systemd units, a private CA and a +from-nothing install guide that starts with a machine that has no CUDA and no +binary. ### llama.cpp, and CPU profiles @@ -63,6 +182,23 @@ agent turn it on. - **A broken metric contract was invisible.** It was logged per scrape at debug level, so a cluster could look healthy while every routing decision was made blind. It is now a warning, once, with a matching line when it clears. +- **Token estimates came from the routing prefix, not the prompt.** Every prompt + over about 2 KB was undercounted, so tokens-per-minute limits let through far + more than they were set to. +- **Model-fitting double-counted overhead.** Whole-model overhead was charged per + rank and a 10% reserve taken twice, so `plan` reported that a 70B model in + bfloat16 does not fit two 80 GB cards. It does. +- **`x-llmgw-served-model` was never sent.** It was set after the proxy had + already written the response header, which is the one point at which headers + stop being changeable. A client had no way to tell a fallback from the model + it asked for. +- **Fleet metrics published nothing.** They were registered after the registry + had been built, so the collector existed and was never scraped. +- **GPU discovery broke on older drivers.** A field added for utilisation was + required rather than optional, so a driver that does not report it produced no + inventory at all instead of an inventory without that number. +- **Replica ids and ports were reused** while the processes holding them were + still alive. ## v0.5.0 — 2026-09-16 @@ -98,10 +234,10 @@ not have. The failure path already ships an engine log tail with every status report, so the case that matters most is covered; reading a healthy shard's output on demand is not. -## Unreleased — phase 4 in progress +### Phases 2 and 3, in detail -Agent groundwork. Everything here works without GPUs and is covered by tests; -what is missing is the control plane connection, so there is no release yet. +Written as the work landed, which is why it reads as a log rather than as +notes. Everything here shipped in v0.5.0. ### Added diff --git a/README.md b/README.md index 5eae3d3..8f34af2 100644 --- a/README.md +++ b/README.md @@ -29,13 +29,20 @@ Releases track the roadmap phases; see [CHANGELOG.md](CHANGELOG.md). | v0.3.0 | 1.5 | Admission control (priority bands) | released | | v0.4.0 | 2 | Agent: GPU/IB discovery, engine supervision | released | | v0.5.0 | 3 | Control plane: catalog, placement, reconciler, ctl | released | -| v0.6.0 | 4 | Multi-node placement and fabric awareness | planned | +| v0.6.0 | 4 | Multi-node placement and fabric awareness | released | | v1.0.0 | 5 | Tenancy, prefix affinity, benchmarking | planned | -As of v0.3.0 the router serves traffic and protects it: a batch workload no -longer starves interactive requests. Still missing before v1.0.0: authentication, -rate limits and budgets, and the control plane that replaces the static endpoint -list. +As of v0.6.0 a cluster runs unattended: agents report their hardware, the +control plane places replicas — including ones larger than a single machine — +models load when someone asks for them and stop when nobody does, and keys, +budgets and rate limits decide who may use any of it. + +Before v1.0.0: routing profiles selectable per request, spillover to a fallback +pool, tracing, and reference documentation. One thing is worth saying plainly: +multi-node replicas are proven on a simulated cluster and on real hardware +without a fabric. The InfiniBand, NVML and NCCL paths are written against the +formats and failure modes those tools actually have, but they have not yet run +on a machine that has them. ## Building