Stop the install guide contradicting itself about weights
It said "there is no distribution mechanism yet" and then, two paragraphs later, documented the distribution mechanism. The known-gaps list — the part someone reads at 2am — still said "no cache, no fetch, no progress". Fetching landed in this cycle; both sentences were written before it and left standing. node-onboarding.md carried the same claim. What is actually still missing is narrower and worth naming precisely: a fetch reports no progress, so a first start on a large model looks like a hang, and every node fetches its own copy, so eight nodes on a new model are eight downloads from one origin. Also: the dashboard now shows who is using the cluster and what they are running, which the sentence describing it did not mention; and one sentence about idle timeouts had lost the words that made it a sentence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
7de8bacfc0
commit
b58d67602d
2 files changed
+19
-10
No files matched your search
+16
-9
@@ -305,9 +305,10 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
|
||||
|
||||
### Weights
|
||||
|
||||
Put them where the node can read them. There is no distribution mechanism yet;
|
||||
`scp`, a shared filesystem, or the engine's own downloader are all valid, and
|
||||
which you want depends on how many nodes you have.
|
||||
Put them where the node can read them. `scp`, a shared filesystem, or the
|
||||
engine's own downloader are all valid, and which you want depends on how many
|
||||
nodes you have. If you would rather not choose, skip to the next section and
|
||||
let llmgw fetch them.
|
||||
|
||||
```
|
||||
sudo install -d -o root -g root /var/lib/llmgw-models
|
||||
@@ -501,8 +502,8 @@ scaling: { min: 0, max: 1, idle_timeout: 15m }
|
||||
model starts it and waits briefly; if it is not serving by then the client gets
|
||||
a 503 with `Retry-After` and an honest "the model is loading", rather than being
|
||||
told nothing serves it — a client told to retry succeeds where one told the
|
||||
model does not exist gives up. Once nobody has asked for `idle_timeout`, the
|
||||
replicas are stopped and the cards go back to the pool.
|
||||
model does not exist gives up. Once nobody has asked for it in `idle_timeout`,
|
||||
the replicas are stopped and the cards go back to the pool.
|
||||
|
||||
`min: 0, max: 0` is different and equally useful: configured, and deliberately
|
||||
not running. A model with no `scaling` stanza at all gets that, because making
|
||||
@@ -566,9 +567,9 @@ temperature, ECC faults and utilisation.
|
||||
|
||||
**The dashboard is the answer for most clusters.** `/admin` shows capacity per
|
||||
GPU class, free whole nodes per fabric, which replicas are holding hardware
|
||||
nothing is asking for, and the same node and model tables `ctl` prints — with
|
||||
no monitoring stack to stand up beside it. One binary, no dependencies, which
|
||||
is the point.
|
||||
nothing is asking for, who is using the cluster and what they are running, and
|
||||
the same node and model tables `ctl` prints — with no monitoring stack to stand
|
||||
up beside it. One binary, no dependencies, which is the point.
|
||||
|
||||
Prometheus is worth adding when you want history, alerts, or one view across
|
||||
several clusters. The dashboard shows what is true now; it keeps nothing.
|
||||
@@ -625,7 +626,13 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
|
||||
## Known gaps, so you find them here rather than at 2am
|
||||
|
||||
- **Weights are yours to distribute.** No cache, no fetch, no progress.
|
||||
- **A fetch is silent while it runs.** llmgw will fetch weights for you, but
|
||||
it reports no progress, so a first start on a large model looks like a hang.
|
||||
`ls -l` in the model directory is the only progress bar there is.
|
||||
- **Every node fetches its own copy.** There is no seeding between nodes, so
|
||||
bringing up eight nodes on a new model is eight downloads from the same
|
||||
origin. On a fabric where the nodes can see each other this is wasteful, and
|
||||
it is the obvious next thing to build.
|
||||
- **Engines are yours to install.** Onboarding verifies one runs; it installs
|
||||
nothing.
|
||||
- **One control plane.** No HA yet; it is a single process with a SQLite file.
|
||||
|
||||
@@ -139,7 +139,9 @@ So the node is not finished until:
|
||||
Both are deliberately outside this script. Engines are a platform decision with
|
||||
versions and drivers attached, and distributing weights is its own problem —
|
||||
hundreds of gigabytes that want a shared filesystem or a cache, not an scp from
|
||||
a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight.
|
||||
a laptop. llmgw will fetch weights for a model whose `source` is a URL, one
|
||||
copy per node, which covers the simple case; anything larger stays the
|
||||
operator's arrangement, and llmgw is content to find the files already there.
|
||||
|
||||
The `sim` engine needs neither, which is what makes it useful for exercising
|
||||
everything else.
|
||||
|
||||
Reference in new issue
Block a user