From b58d67602d8a3d6383f16343831e1f01e6a0f411 Mon Sep 17 00:00:00 2001 From: Ilker Manap Date: Fri, 18 Sep 2026 06:04:00 +0200 Subject: [PATCH] Stop the install guide contradicting itself about weights MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It said "there is no distribution mechanism yet" and then, two paragraphs later, documented the distribution mechanism. The known-gaps list — the part someone reads at 2am — still said "no cache, no fetch, no progress". Fetching landed in this cycle; both sentences were written before it and left standing. node-onboarding.md carried the same claim. What is actually still missing is narrower and worth naming precisely: a fetch reports no progress, so a first start on a large model looks like a hang, and every node fetches its own copy, so eight nodes on a new model are eight downloads from one origin. Also: the dashboard now shows who is using the cluster and what they are running, which the sentence describing it did not mention; and one sentence about idle timeouts had lost the words that made it a sentence. Co-Authored-By: Claude Opus 5 (1M context) --- docs/install.md | 25 ++++++++++++++++--------- docs/node-onboarding.md | 4 +++- 2 files changed, 19 insertions(+), 10 deletions(-) diff --git a/docs/install.md b/docs/install.md index c095a165..4819b788 100644 --- a/docs/install.md +++ b/docs/install.md @@ -305,9 +305,10 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm ### Weights -Put them where the node can read them. There is no distribution mechanism yet; -`scp`, a shared filesystem, or the engine's own downloader are all valid, and -which you want depends on how many nodes you have. +Put them where the node can read them. `scp`, a shared filesystem, or the +engine's own downloader are all valid, and which you want depends on how many +nodes you have. If you would rather not choose, skip to the next section and +let llmgw fetch them. ``` sudo install -d -o root -g root /var/lib/llmgw-models @@ -501,8 +502,8 @@ scaling: { min: 0, max: 1, idle_timeout: 15m } model starts it and waits briefly; if it is not serving by then the client gets a 503 with `Retry-After` and an honest "the model is loading", rather than being told nothing serves it — a client told to retry succeeds where one told the -model does not exist gives up. Once nobody has asked for `idle_timeout`, the -replicas are stopped and the cards go back to the pool. +model does not exist gives up. Once nobody has asked for it in `idle_timeout`, +the replicas are stopped and the cards go back to the pool. `min: 0, max: 0` is different and equally useful: configured, and deliberately not running. A model with no `scaling` stanza at all gets that, because making @@ -566,9 +567,9 @@ temperature, ECC faults and utilisation. **The dashboard is the answer for most clusters.** `/admin` shows capacity per GPU class, free whole nodes per fabric, which replicas are holding hardware -nothing is asking for, and the same node and model tables `ctl` prints — with -no monitoring stack to stand up beside it. One binary, no dependencies, which -is the point. +nothing is asking for, who is using the cluster and what they are running, and +the same node and model tables `ctl` prints — with no monitoring stack to stand +up beside it. One binary, no dependencies, which is the point. Prometheus is worth adding when you want history, alerts, or one view across several clusters. The dashboard shows what is true now; it keeps nothing. @@ -625,7 +626,13 @@ requests. On a slow model, set `timeouts.generation` higher. ## Known gaps, so you find them here rather than at 2am -- **Weights are yours to distribute.** No cache, no fetch, no progress. +- **A fetch is silent while it runs.** llmgw will fetch weights for you, but + it reports no progress, so a first start on a large model looks like a hang. + `ls -l` in the model directory is the only progress bar there is. +- **Every node fetches its own copy.** There is no seeding between nodes, so + bringing up eight nodes on a new model is eight downloads from the same + origin. On a fabric where the nodes can see each other this is wasteful, and + it is the obvious next thing to build. - **Engines are yours to install.** Onboarding verifies one runs; it installs nothing. - **One control plane.** No HA yet; it is a single process with a SQLite file. diff --git a/docs/node-onboarding.md b/docs/node-onboarding.md index e865191b..a0ec78e3 100644 --- a/docs/node-onboarding.md +++ b/docs/node-onboarding.md @@ -139,7 +139,9 @@ So the node is not finished until: Both are deliberately outside this script. Engines are a platform decision with versions and drivers attached, and distributing weights is its own problem — hundreds of gigabytes that want a shared filesystem or a cache, not an scp from -a laptop. llmgw has no answer for it yet; that is a known gap, not an oversight. +a laptop. llmgw will fetch weights for a model whose `source` is a URL, one +copy per node, which covers the simple case; anything larger stays the +operator's arrangement, and llmgw is content to find the files already there. The `sim` engine needs neither, which is what makes it useful for exercising everything else.