Make the install guide describe the system that exists
Read start to finish as somebody standing up a new cluster, it sent them down a path we no longer take. The mirror was absent from all 821 lines, and the page still said that distributing weights was "a known gap, not an oversight" — so the reader would have gone away to solve, by hand, the problem the binary in front of them solves. It is step 4 now, before the nodes, because where a node fetches from is the difference between minutes and a day, and on a customer network the nodes often have no route out at all. Two of the listed gaps had been closed and were still listed: a fetch does report its progress to the control plane, and there is an HA pattern. A gap somebody believes in is worse than one they discover. Added what this week made obvious was missing. Upgrading, including the zero-byte copy that ssh -n silently produces and the size check that is the only thing between that and a fleet on a truncated binary. Backing up, in the order that matters, and the fact that API keys are hashed and do not come back. Pricing in the first configuration, because a budget without a price is displayed, counted against nothing, and never binds. And scale, bench and the source check, none of which were written down anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
03e38e7dda
commit
d8fb198aa5
1 file changed
+291
-31
+291
-31
@@ -13,7 +13,7 @@ this page says otherwise, and where it says otherwise it says so plainly.
|
||||
Read this first, because it is the difference between an afternoon and a week.
|
||||
|
||||
llmgw runs engines and routes requests. It does **not** install GPU drivers,
|
||||
CUDA, inference engines, or model weights. A node missing any of them still
|
||||
CUDA, or the inference engines themselves. A node missing any of them still
|
||||
registers, still reports `ready`, and still fails every replica placed on it —
|
||||
so the useful order is to make a node able to serve *before* onboarding it, not
|
||||
after.
|
||||
@@ -23,12 +23,15 @@ after.
|
||||
| NVIDIA driver and `nvidia-smi` | GPU discovery, classes, health |
|
||||
| InfiniBand stack (`rdma-core`/OFED) | Fabric discovery and the invariant that a replica never spans one |
|
||||
| vLLM, SGLang or llama.cpp | Launching them, the right flags, supervision |
|
||||
| Model weights on disk | Placement, routing, accounting |
|
||||
| Model weights, from somewhere | Fetching them, placement, routing, accounting |
|
||||
|
||||
The reason is boring and unavoidable: drivers and engines are platform decisions
|
||||
with kernel modules and version matrices attached, and distributing weights is
|
||||
its own problem — hundreds of gigabytes that want a shared filesystem, not a
|
||||
copy from somebody's laptop. It is a known gap, not an oversight.
|
||||
with kernel modules and version matrices attached.
|
||||
|
||||
Weights are the one item on that list llmgw does handle. It includes a mirror —
|
||||
step 4 — which holds the files on your own network and hands them to the nodes,
|
||||
so getting weights onto every machine is not a problem you solve before
|
||||
starting. What llmgw will not do is choose the models for you.
|
||||
|
||||
## 0. Before you begin
|
||||
|
||||
@@ -49,7 +52,10 @@ them, so nodes need no inbound access except your own SSH.
|
||||
| --- | --- | --- |
|
||||
| 8080 | clients, the web UI | your users, usually via a reverse proxy |
|
||||
| 7443 | node agents | the node network only |
|
||||
| 9090 | `llmgw ctl`, metrics, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
|
||||
| 9090 | `llmgw ctl`, the management API, pprof | loopback only — it serves pprof, which can dump the process heap, and API keys live in that heap |
|
||||
| 9091 | Prometheus, if you run it | your monitoring host; metrics only, nothing else is served here |
|
||||
| 8081 | node agents fetching weights | the node network only (the mirror machine) |
|
||||
| 8082 | the control plane administering the mirror | the control plane only — this port can delete models and start downloads |
|
||||
|
||||
Decide now which address nodes will use to reach the control plane. It goes into
|
||||
a certificate and changing it later means reissuing one.
|
||||
@@ -148,11 +154,22 @@ auth:
|
||||
# that is the bill or the outage.
|
||||
default_rpm: 60
|
||||
default_tpm: 200000
|
||||
# What a request costs, which is what makes a budget mean anything. Without
|
||||
# a price every request costs zero, so a budget is displayed, counted against
|
||||
# nothing, and never binds — and nobody finds out until they expected it to
|
||||
# stop something. Self-hosted models have no per-token price, so the unit is
|
||||
# yours to choose: GPU-seconds, a notional rate, or 1000 everywhere, which
|
||||
# turns a budget into a plain token quota.
|
||||
default_price:
|
||||
prompt_per_1k: 1000
|
||||
completion_per_1k: 1000
|
||||
# pricing:
|
||||
# qwen2.5-coder-7b-instruct: { prompt_per_1k: 500, completion_per_1k: 1500 }
|
||||
|
||||
landing:
|
||||
contact: "platform@example.com"
|
||||
|
||||
# Start with none. Models are added while the cluster runs — see step 7 —
|
||||
# Start with none. Models are added while the cluster runs — see step 8 —
|
||||
# and anything listed here is declarative: it is in the catalog on every start
|
||||
# and cannot be removed without editing this file.
|
||||
models: []
|
||||
@@ -200,7 +217,103 @@ sudo -u llmgw llmgw ctl status
|
||||
been added yet. It is worth running now so its output is familiar before you
|
||||
need it.
|
||||
|
||||
## 4. Preparing a node
|
||||
## 4. The mirror
|
||||
|
||||
Do this before the nodes, not after. Every node fetches its own copy of a
|
||||
model's weights, and where they fetch from decides whether bringing up a cluster
|
||||
takes minutes or a day. Eight nodes pulling forty gigabytes each from Hugging
|
||||
Face is a day's work, and on a good number of corporate networks it is not
|
||||
possible at all — the nodes have no route out, which is usually the point.
|
||||
|
||||
The mirror is one machine holding the files on your own network. It is the same
|
||||
binary again, and it is the highest-value piece of infrastructure around llmgw
|
||||
for the least work.
|
||||
|
||||
### Running it
|
||||
|
||||
```
|
||||
sudo install -d -o llmgw -g llmgw /srv/models
|
||||
llmgw mirror serve --dir /srv/models --base http://models.internal:8081
|
||||
```
|
||||
|
||||
In production, the unit:
|
||||
|
||||
```
|
||||
sudo cp deploy/systemd/llmgw-mirror.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable --now llmgw-mirror
|
||||
```
|
||||
|
||||
**Two listeners, on purpose.** `:8081` serves bytes and nothing else — that is
|
||||
what every agent talks to, and it is plain HTTP because a weights fetch is a
|
||||
public file transfer that is checksummed anyway. `:8082` is the management API:
|
||||
it can start downloads and delete models, and only the control plane needs it.
|
||||
The split is what lets you lock the second one down without re-certifying every
|
||||
agent that fetches from the first.
|
||||
|
||||
Give the management port the cluster's own CA and it takes client certificates:
|
||||
|
||||
```
|
||||
llmgw mirror serve --dir /srv/models --base http://models.internal:8081 \
|
||||
--tls-ca /etc/llmgw/ca.pem \
|
||||
--tls-cert /etc/llmgw/mirror.pem --tls-key /etc/llmgw/mirror-key.pem
|
||||
```
|
||||
|
||||
Without those three flags the management port is open to anyone who can reach
|
||||
it, and the mirror says so in its own log at every start rather than letting you
|
||||
find out later.
|
||||
|
||||
### The layout
|
||||
|
||||
```
|
||||
/srv/models/
|
||||
qwen2.5-14b-instruct/
|
||||
2026-03/
|
||||
q4_k_m.gguf
|
||||
```
|
||||
|
||||
Model, then version, then files. The version directory is the point of the
|
||||
layout rather than decoration: a repository that changes upstream changes what
|
||||
you are running, and the first sign is a prompt that worked last week answering
|
||||
differently this week with nobody able to say why. Pin it here and that cannot
|
||||
happen.
|
||||
|
||||
A version holding several files is a sharded model, and is served as a set: the
|
||||
agent asks the directory what it consists of and fetches all of it into one
|
||||
place, marking it complete only when the last file lands. A catalogue entry for
|
||||
one names the directory, trailing slash included.
|
||||
|
||||
### Telling the control plane about it
|
||||
|
||||
```yaml
|
||||
# /etc/llmgw/server.yaml
|
||||
mirror:
|
||||
api: https://models.internal:8082
|
||||
tls:
|
||||
ca: /etc/llmgw/ca.pem
|
||||
cert: /etc/llmgw/server.pem
|
||||
key: /etc/llmgw/server-key.pem
|
||||
```
|
||||
|
||||
With this set, `/admin/mirror` appears: what the mirror holds, what it is
|
||||
downloading and how far along, a search of Hugging Face, and a form that puts a
|
||||
mirrored model into service without anyone writing YAML. Without it that page
|
||||
stays unmounted and the mirror is still perfectly usable — a model's `source` is
|
||||
just a URL, and the URL can be yours.
|
||||
|
||||
If the control plane cannot reach the internet, `mirror.registry` points the
|
||||
search at an internal registry mirror instead.
|
||||
|
||||
### Filling it
|
||||
|
||||
```
|
||||
llmgw mirror fetch --model qwen2.5-14b-instruct --version 2026-03 \
|
||||
hf:Qwen/Qwen2.5-14B-Instruct-GGUF/qwen2.5-14b-instruct-q4_k_m.gguf
|
||||
```
|
||||
|
||||
or from the admin page, or by copying files into the directory by hand — the
|
||||
mirror is a directory, and anything that puts files in it works.
|
||||
|
||||
## 5. Preparing a node
|
||||
|
||||
**Do this before onboarding.** A node missing its driver or engine registers
|
||||
healthy and fails on first placement, and the reason is three logs away.
|
||||
@@ -305,10 +418,13 @@ llmgw ctl engine-check vllm --bin /opt/venvs/vllm/bin/vllm
|
||||
|
||||
### Weights
|
||||
|
||||
Nothing to do here if you built the mirror in step 4 — the agent fetches what a
|
||||
model's `source` names, and that source is your mirror. This section is for the
|
||||
arrangements where the files are already on the machine.
|
||||
|
||||
Put them where the node can read them. `scp`, a shared filesystem, or the
|
||||
engine's own downloader are all valid, and which you want depends on how many
|
||||
nodes you have. If you would rather not choose, skip to the next section and
|
||||
let llmgw fetch them.
|
||||
nodes you have.
|
||||
|
||||
```
|
||||
sudo install -d -o root -g root /var/lib/llmgw-models
|
||||
@@ -325,11 +441,18 @@ model's `source` at a file to download and the agent fetches it once, before the
|
||||
engine is launched:
|
||||
|
||||
```yaml
|
||||
# your mirror, which is what step 4 was for
|
||||
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
|
||||
# a whole version, for a model of several files — the trailing slash is the signal
|
||||
source: http://models.internal:8081/llama-3.1-70b-instruct/2026-03/
|
||||
# or straight from Hugging Face, where the nodes can reach it
|
||||
source: hf:Qwen/Qwen2.5-Coder-3B-Instruct-GGUF/qwen2.5-coder-3b-instruct-q4_k_m.gguf
|
||||
# or an ordinary URL
|
||||
source: https://models.internal/qwen2.5-coder-3b-q4_k_m.gguf
|
||||
```
|
||||
|
||||
A fetch reports its progress to the control plane, so `llmgw ctl replicas` and
|
||||
the dashboard both show how far along a node is. A replica pulling a hundred
|
||||
gigabytes and one that is wedged do not look alike.
|
||||
|
||||
Eight shards of one replica starting together produce one download, not eight.
|
||||
An interrupted fetch resumes rather than starting over, and a partial file is
|
||||
never visible under the model's name — so a half-downloaded model cannot be
|
||||
@@ -375,7 +498,7 @@ How much disk a node needs follows from the models rather than a rule of thumb.
|
||||
`llmgw ctl model plan <repo>` prints the sizes; as a starting point, a 70B at
|
||||
bf16 is 141 GB and the same model at q4 is 43 GB.
|
||||
|
||||
## 5. Adding the node
|
||||
## 6. Adding the node
|
||||
|
||||
Two halves, because they need different access: building the machine needs the
|
||||
hypervisor, and issuing its identity needs the certificate authority. Where one
|
||||
@@ -432,7 +555,7 @@ Repeat for each node. Nodes in the same fabric must be given the same `--fabric`
|
||||
name; nodes with no InfiniBand take no `--fabric` at all and are each their own
|
||||
island, which is fine for single-node models.
|
||||
|
||||
## 6. Setting the node's CPU model, if it is a VM
|
||||
## 7. Setting the node's CPU model, if it is a VM
|
||||
|
||||
Proxmox defaults a VM to `kvm64`, which hides AVX, AVX2, FMA and even SSE4.2
|
||||
from the guest. Inference then falls back to scalar code and is not slightly
|
||||
@@ -449,7 +572,7 @@ only change being `--cpu host`:
|
||||
probably did not, and nothing about it looks broken — every number is simply
|
||||
small. Check with `grep -qw avx2 /proc/cpuinfo` on the node.
|
||||
|
||||
## 7. Adding a model
|
||||
## 8. Adding a model
|
||||
|
||||
Write the definition. This is the same shape a model takes in `server.yaml`, so
|
||||
an entry moves between the two by being copied:
|
||||
@@ -458,7 +581,7 @@ an entry moves between the two by being copied:
|
||||
# qwen-coder-3b.yaml
|
||||
name: qwen2.5-coder-3b-instruct
|
||||
aliases: [coder]
|
||||
source: /var/lib/llmgw-models/qwen2.5-coder-3b-instruct-q4_k_m.gguf
|
||||
source: http://models.internal:8081/qwen2.5-coder-3b-instruct/2026-03/q4_k_m.gguf
|
||||
routing_profile: chat
|
||||
# What a client may use in one request. Worth stating: llama.cpp divides
|
||||
# --ctx-size across its parallel slots, so a pool of 65536 over two slots is
|
||||
@@ -481,6 +604,17 @@ llmgw ctl model add -f qwen-coder-3b.yaml
|
||||
llmgw ctl models
|
||||
```
|
||||
|
||||
Adding a model checks that the source is actually there before accepting it, so
|
||||
a typo or a mirror that is down is a sentence now rather than a replica failing
|
||||
on a node in five minutes with an error naming the node. `--no-source-check`
|
||||
skips it, which is right when the nodes reach the mirror by an address this
|
||||
machine does not.
|
||||
|
||||
With a mirror configured you can skip the file entirely: `/admin/mirror` lists
|
||||
what the mirror holds and offers a form that writes the same definition, with
|
||||
the engine, the source and the memory figure already filled in from the file
|
||||
itself.
|
||||
|
||||
```
|
||||
MODEL SERVING REPLICAS STATUS ALIASES
|
||||
qwen2.5-coder-3b-instruct yes 1/1 ready coder
|
||||
@@ -545,7 +679,42 @@ llmgw ctl explain-placement qwen2.5-coder-3b-instruct
|
||||
which names every candidate and why each was rejected, rather than leaving you
|
||||
to infer it.
|
||||
|
||||
## 8. The first request
|
||||
### How many replicas, and how much each one carries
|
||||
|
||||
Replica count is changed while the cluster runs, without editing the
|
||||
definition:
|
||||
|
||||
```
|
||||
llmgw ctl scale qwen2.5-coder-3b-instruct --min 3
|
||||
```
|
||||
|
||||
Models added at runtime only. One declared in `server.yaml` is read again on
|
||||
every start, so a number set here would hold until the next restart and then
|
||||
revert — change it in the file instead.
|
||||
|
||||
How much traffic each replica is given is the profile's `weight`, and guessing
|
||||
it is how a cluster ends up with one saturated replica beside two idle ones.
|
||||
Measure it instead:
|
||||
|
||||
```
|
||||
llmgw ctl bench qwen2.5-coder-3b-instruct --save
|
||||
```
|
||||
|
||||
Concurrency climbs until the latency target stops being met, and the last level
|
||||
that held it is the capacity — peak throughput arrives long after latency has
|
||||
become unusable, so throughput alone would overstate what the hardware serves.
|
||||
It prints the `weight` to put in the profile, and `--save` records the figure
|
||||
against the model so the dashboard can show what the cluster is measured to
|
||||
carry rather than what somebody typed.
|
||||
|
||||
This drives real traffic and competes with real requests. Run it against a
|
||||
cordoned replica, or in a window where that is acceptable.
|
||||
|
||||
An engine that publishes no metrics endpoint is measured from the outside only.
|
||||
The run says so, and queue depth and KV occupancy are then not part of the
|
||||
answer.
|
||||
|
||||
## 9. The first request
|
||||
|
||||
```
|
||||
curl http://<control-plane>:8080/v1/chat/completions \
|
||||
@@ -586,8 +755,10 @@ One scrape target for the whole fleet. The node and GPU figures come from the
|
||||
control plane, which already holds them, so there is nothing to install on the
|
||||
nodes and no Prometheus configuration per machine.
|
||||
|
||||
A starting configuration, with the alerts worth having, is in
|
||||
[deploy/monitoring/prometheus.yml](../deploy/monitoring/prometheus.yml).
|
||||
A starting configuration, with the alerts worth having and a provisioned
|
||||
dashboard, is in [deploy/monitoring/](../deploy/monitoring/) — including where
|
||||
to run it, which is not the control plane: monitoring that dies with the thing
|
||||
it monitors tells you nothing at the moment you need it.
|
||||
|
||||
Two figures repay knowing about. `llmgw_gpu_utilization_percent` is absent
|
||||
rather than zero when the driver does not report it — a flat zero across a
|
||||
@@ -730,6 +901,91 @@ thing to get right is the health check: point it at `/ready`, not `/health`.
|
||||
so a load balancer using it would send half the traffic to something that
|
||||
refuses every request.
|
||||
|
||||
## Upgrading
|
||||
|
||||
One binary on every machine, so an upgrade is a copy and a restart. The part
|
||||
worth care is the copy, not the order.
|
||||
|
||||
**Nothing stops serving.** Engines are started with `KillMode=process`, so they
|
||||
keep running when the agent restarts, and the agent reports them back on its
|
||||
next connection. The control plane holds no replica state across its own
|
||||
restart either — it rebuilds the fleet from what the agents report. A rolling
|
||||
upgrade of the whole cluster interrupts no request that is not in flight at the
|
||||
moment of the restart.
|
||||
|
||||
**A mixed fleet is fine while you roll.** The agent protocol only ever gains
|
||||
fields, and both sides treat an absent one as unknown rather than as a default.
|
||||
Order does not matter; finish the roll the same day anyway, because
|
||||
`ctl nodes` reports each agent's version and a fleet that disagrees with itself
|
||||
is one more thing to hold in your head.
|
||||
|
||||
```
|
||||
VERSION=v0.9.0
|
||||
# copy to every machine first, install nowhere yet
|
||||
for h in 10.0.0.140 10.0.0.141 10.0.0.142 10.0.0.146; do
|
||||
ssh $h 'cat > /tmp/llmgw-new; stat -c %s /tmp/llmgw-new' < llmgw-$VERSION-linux-amd64
|
||||
done
|
||||
```
|
||||
|
||||
**Check every size before installing anything.** This is not defensive writing:
|
||||
on a host whose login shell prints anything, `scp` fails outright, and the
|
||||
obvious workaround — `ssh -n host 'cat > file'` — points stdin at `/dev/null`
|
||||
and writes a *zero-byte file* while reporting success. Installing that on a
|
||||
fleet takes the fleet down. Compare the number on each machine with the local
|
||||
one, then:
|
||||
|
||||
```
|
||||
sudo install -m0755 /tmp/llmgw-new /usr/local/bin/llmgw
|
||||
sudo systemctl restart llmgw-agent # or llmgw-server, or llmgw-mirror
|
||||
llmgw --version
|
||||
```
|
||||
|
||||
Then confirm what you have, rather than assuming:
|
||||
|
||||
```
|
||||
llmgw ctl nodes # every agent back, LAST SEEN in seconds
|
||||
llmgw ctl replicas # same replicas, adopted rather than replaced
|
||||
```
|
||||
|
||||
A replica may come back marked **`ready (stale profile)`**. That means it is
|
||||
running a profile that has since been edited — the reconciler leaves a serving
|
||||
replica alone on purpose, because churning one costs more than the difference.
|
||||
Restart it when you want the new profile to take effect; the flag is there so
|
||||
the choice is yours rather than silent.
|
||||
|
||||
## Backing up
|
||||
|
||||
Three things, in descending order of how badly you want them.
|
||||
|
||||
**The certificate authority.** `ca-key.pem`, wherever you put it in step 2.
|
||||
Losing it does not stop the cluster, and it means every node identity has to be
|
||||
reissued the next time you touch one. Nothing in llmgw can reconstruct it.
|
||||
|
||||
**The store.** Users, API keys, organisations, teams, projects, every limit and
|
||||
budget, all usage history, models added at runtime, and measured capacity.
|
||||
|
||||
```
|
||||
# SQLite
|
||||
sqlite3 /var/lib/llmgw/llmgw.db ".backup '/backup/llmgw-$(date +%F).db'"
|
||||
|
||||
# PostgreSQL
|
||||
pg_dump --format=custom llmgw > /backup/llmgw-$(date +%F).dump
|
||||
```
|
||||
|
||||
`.backup` and `pg_dump` are both safe against a running gateway. Copying the
|
||||
SQLite file with `cp` while it is being written is not.
|
||||
|
||||
**Keys are stored hashed and cannot be recovered from a backup or from
|
||||
anywhere else.** Restoring an old store restores which keys exist and who owns
|
||||
them, not their secrets. A key created after the backup is gone and has to be
|
||||
reissued — which is why the store is worth backing up on a schedule rather than
|
||||
before upgrades.
|
||||
|
||||
**The mirror's directory** is the one thing you can rebuild, by downloading it
|
||||
all again. Worth remembering that a pinned version may no longer be upstream to
|
||||
download: that is exactly what pinning it protected you from, and a backup is
|
||||
what makes the protection outlive the disk.
|
||||
|
||||
## Reclaiming disk
|
||||
|
||||
`llmgw ctl nodes` has a DISK column: free space against the size it is free
|
||||
@@ -772,7 +1028,7 @@ output, which is the actual diagnosis.
|
||||
node and the reason: wrong GPU class, not enough free cards, too little memory,
|
||||
no fabric, a label it does not carry.
|
||||
|
||||
**Everything is slow.** On a VM, check AVX2 (step 6). On CPU, check the
|
||||
**Everything is slow.** On a VM, check AVX2 (step 7). On CPU, check the
|
||||
quantisation — the difference between `q8_0` and `q4_k_m` is roughly a factor of
|
||||
two. `llmgw ctl status` shows how full the cluster is.
|
||||
|
||||
@@ -793,6 +1049,10 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
- `docs/reference.md` — every configuration key, generated from the source, so
|
||||
it describes the version you are running rather than the one somebody last
|
||||
wrote about
|
||||
- `deploy/mirror/README.md` — the mirror in more detail, including the layout
|
||||
and what belongs on which port
|
||||
- `deploy/monitoring/README.md` — Prometheus, Grafana and the alerts, with what
|
||||
each one assumes about this system
|
||||
- `docs/node-onboarding.md` — adding nodes in more detail, and what onboarding
|
||||
deliberately does not do
|
||||
- `docs/onboarding.md` — giving people accounts and keys
|
||||
@@ -800,20 +1060,20 @@ requests. On a slow model, set `timeouts.generation` higher.
|
||||
|
||||
## Known gaps, so you find them here rather than at 2am
|
||||
|
||||
- **A fetch reports its progress where nobody is looking.** The agent logs a
|
||||
percentage as it downloads, so `journalctl -u llmgw-agent` on that node knows
|
||||
exactly how far along it is. Nothing carries it to the control plane, so from
|
||||
`ctl` or the dashboard a replica pulling a hundred gigabytes and one that is
|
||||
wedged still look identical — which is the comparison the log line itself
|
||||
says matters.
|
||||
- **Every node fetches its own copy.** There is no seeding between nodes, so
|
||||
bringing up eight nodes on a new model is eight downloads from the same
|
||||
origin. On a fabric where the nodes can see each other this is wasteful, and
|
||||
it is the obvious next thing to build.
|
||||
bringing up eight nodes on a new model is eight downloads. The mirror makes
|
||||
those eight local, which is most of the problem; on a fabric where the nodes
|
||||
can see each other, seeding between them is the obvious next thing to build.
|
||||
- **A budget without a price never binds.** Set `default_price` or `pricing`
|
||||
(step 3) or every request costs zero, and a budget is something the dashboard
|
||||
displays rather than something that stops anything. The quotas page says so
|
||||
where it can, which is not the same as being told before you rely on it.
|
||||
- **Engines are yours to install.** Onboarding verifies one runs; it installs
|
||||
nothing.
|
||||
- **One control plane.** No HA yet; it is a single process with a SQLite file.
|
||||
Back up `/var/lib/llmgw/llmgw.db`.
|
||||
- **Two control planes have not been run for long.** The leader election and
|
||||
agent failover work and there is a pattern in `deploy/ha/`, but every cluster
|
||||
this has been run on in anger has had one. A single control plane is the
|
||||
well-trodden path.
|
||||
- **Multi-node replicas have not been run on real InfiniBand.** The scheduling,
|
||||
the fabric invariant and the rendezvous are exercised against simulated
|
||||
hardware only. Single-node placement on real GPUs is the well-trodden path;
|
||||
|
||||
Reference in new issue
Block a user