Stop the install guide contradicting itself about weights
CI / test (push) Successful in 1m41s

It said "there is no distribution mechanism yet" and then, two paragraphs
later, documented the distribution mechanism. The known-gaps list — the part
someone reads at 2am — still said "no cache, no fetch, no progress". Fetching
landed in this cycle; both sentences were written before it and left standing.

node-onboarding.md carried the same claim.

What is actually still missing is narrower and worth naming precisely: a fetch
reports no progress, so a first start on a large model looks like a hang, and
every node fetches its own copy, so eight nodes on a new model are eight
downloads from one origin.

Also: the dashboard now shows who is using the cluster and what they are
running, which the sentence describing it did not mention; and one sentence
about idle timeouts had lost the words that made it a sentence.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ilker ManapandClaude Opus 5 committed 2026-09-18 06:04:00 +02:00
1 parent 7de8bacfc0
commit b58d67602d
2 files changed
+19 -10

No files matched your search

+16 -9
View File
@@ -305,9 +305,10 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
### Weights
Put them where the node can read them. There is no distribution mechanism yet;
`scp`, a shared filesystem, or the engine's own downloader are all valid, and
which you want depends on how many nodes you have.
Put them where the node can read them. `scp`, a shared filesystem, or the
engine's own downloader are all valid, and which you want depends on how many
nodes you have. If you would rather not choose, skip to the next section and
let llmgw fetch them.
```
sudo install -d -o root -g root /var/lib/llmgw-models
@@ -501,8 +502,8 @@ scaling: { min: 0, max: 1, idle_timeout: 15m }
model starts it and waits briefly; if it is not serving by then the client gets
a 503 with `Retry-After` and an honest "the model is loading", rather than being
told nothing serves it — a client told to retry succeeds where one told the
model does not exist gives up. Once nobody has asked for `idle_timeout`, the
replicas are stopped and the cards go back to the pool.
model does not exist gives up. Once nobody has asked for it in `idle_timeout`,
the replicas are stopped and the cards go back to the pool.
`min: 0, max: 0` is different and equally useful: configured, and deliberately
not running. A model with no `scaling` stanza at all gets that, because making
@@ -566,9 +567,9 @@ temperature, ECC faults and utilisation.
**The dashboard is the answer for most clusters.** `/admin` shows capacity per
GPU class, free whole nodes per fabric, which replicas are holding hardware
nothing is asking for, and the same node and model tables `ctl` prints — with
no monitoring stack to stand up beside it. One binary, no dependencies, which
is the point.
nothing is asking for, who is using the cluster and what they are running, and
the same node and model tables `ctl` prints — with no monitoring stack to stand
up beside it. One binary, no dependencies, which is the point.
Prometheus is worth adding when you want history, alerts, or one view across
several clusters. The dashboard shows what is true now; it keeps nothing.
@@ -625,7 +626,13 @@ requests. On a slow model, set `timeouts.generation` higher.
## Known gaps, so you find them here rather than at 2am
- **Weights are yours to distribute.** No cache, no fetch, no progress.
- **A fetch is silent while it runs.** llmgw will fetch weights for you, but
it reports no progress, so a first start on a large model looks like a hang.
`ls -l` in the model directory is the only progress bar there is.
- **Every node fetches its own copy.** There is no seeding between nodes, so
bringing up eight nodes on a new model is eight downloads from the same
origin. On a fabric where the nodes can see each other this is wasteful, and
it is the obvious next thing to build.
- **Engines are yours to install.** Onboarding verifies one runs; it installs
nothing.
- **One control plane.** No HA yet; it is a single process with a SQLite file.
+3 -1
View File
@@ -139,7 +139,9 @@ So the node is not finished until:
Both are deliberately outside this script. Engines are a platform decision with
versions and drivers attached, and distributing weights is its own problem —
hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight.
a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
copy per node, which covers the simple case; anything larger stays the
operator's arrangement, and llmgw is content to find the files already there.
The `sim` engine needs neither, which is what makes it useful for exercising
everything else.