This note summarizes issues observed while deploying and operating llmgw
v0.10.6. Environment-specific identifiers, addresses, credentials, certificate
details, and organization-specific topology have intentionally been omitted.
1. Mirror UI silently creates CPU-only profiles for GGUF models
Observed
Serving a .gguf file from the mirror UI creates a llamacpp profile without gpu_class. The resulting model is consequently placed in the CPU pool. The UI
does not offer a CPU/GPU selector. Its editable "Profile name" field may suggest
that changing the name to a GPU-oriented value affects placement, but profile
names do not determine placement.
The behavior follows from the UI path selecting llamacpp for GGUF and setting
the initial profile to cpu; the submitted form does not populate GPU placement
fields such as gpu_class or tp.
Impact
An administrator can successfully add and run a model while unintentionally
using a large amount of CPU and receiving much lower performance. Nothing in the
serve flow clearly explains why the GPU is unused.
Expected / suggested improvement
Offer a placement selector populated from discovered GPU classes.
Show the actual placement semantics (gpu_class present or absent), rather
than relying on a freely editable profile name.
Before submission, summarize: engine, CPU/GPU placement, GPU count, memory,
context size, and concurrency.
If CPU is intentionally the default, display an explicit warning before
serving a GGUF model on a cluster with available GPUs.
2. GPU placement can silently execute with the CPU backend
Observed
A model was assigned to a GPU profile and the generated llama.cpp command
contained --n-gpu-layers 999, but inference still ran at CPU-like speed and
GPU memory usage remained zero. The installed engine directory contained libggml-cuda.so, while required CUDA runtime/cuBLAS shared libraries were not
available to the dynamic loader. llama.cpp continued by loading only its CPU
backend.
The control plane reported the replica as ready and placed in a GPU pool. The
failure became visible only through ldd, process mappings, and GPU telemetry.
Impact
The scheduler reserves a GPU and reports a healthy GPU replica even though the
process performs inference on the CPU. This can be mistaken for poor GPU
performance or a model problem.
Expected / suggested improvement
Validate the CUDA backend and all dynamic dependencies during engine install.
Make engine-check launch a small GPU operation, not only validate accepted
flags.
When a GPU is assigned, fail readiness if llama.cpp reports no CUDA device or
does not load its CUDA backend.
Surface backend selection and offloaded layer count in replica status.
Do not allow silent CPU fallback for a profile that explicitly requires a
GPU, unless the profile has an explicit fallback policy.
3. Rolling restart can remain pending when every matching GPU is occupied
Observed
After changing a model profile/resource value, ctl restart --model attempted
to place the replacement before stopping the existing replica. With every
matching GPU occupied, the replica remained indefinitely in a state similar to:
ready (stale profile) (restart replacing)
The replacement could not be placed because the old replica itself held the
only suitable GPU. Completing the change required scaling the model to zero and
then back to one.
Impact
Configuration changes appear accepted but never take effect. The operator must
understand the replacement algorithm and manually cause downtime.
Expected / suggested improvement
Detect when the old replica is the only placement blocker.
Clearly report that a surge restart is impossible and offer an explicit
stop-then-start option.
Provide a one-command forced restart for single-replica, single-GPU cases.
4. Node admission is blocked by an unused incompatible engine
Observed
llmgw node add rejected otherwise usable GPU nodes because the control plane's
configured vLLM release required a newer compute capability than those nodes
provided. The same nodes supported llama.cpp and were intended for llama.cpp
profiles only. Admission required --ignore-checks, after which the nodes joined
and advertised llama.cpp successfully.
Impact
A heterogeneous cluster cannot cleanly add older GPU generations for a supported
engine when another globally configured engine is incompatible. Using --ignore-checks also suppresses unrelated checks, making it broader than the
operator's intent.
Expected / suggested improvement
Treat engine incompatibility as per-node engine availability, not necessarily
a node-admission failure.
Allow engines to be enabled/disabled per node or GPU class.
Add a targeted option such as --disable-engine vllm instead of requiring --ignore-checks.
Show that the node will join with a reduced engine set and require confirmation
only when appropriate.
5. Fabric validation can block nodes that will not use that fabric
Observed
Doctor treated a detected but unusable RoCE interface as a fatal issue during
node admission, even though no fabric was supplied and the node was being added
for single-node inference. Other detected interfaces also produced warnings
about missing fabric names and RDMA tooling.
Impact
An unrelated or administratively unused adapter can prevent a node from joining
for local GPU workloads.
Expected / suggested improvement
Validate only explicitly configured fabrics as fatal admission checks.
Report unconfigured adapters as informational or warnings.
Separate base node readiness from optional multi-node/RDMA readiness.
6. Signup failures are too generic
Observed
Attempting to sign up with an email address that already existed produced a
generic failure in the UI. Server-side diagnostics showed a unique-constraint
violation, but the UI did not explain that the account already existed or direct
the user toward sign-in/password recovery.
Impact
Users cannot distinguish duplicate accounts from password-policy, database, or
server errors.
Expected / suggested improvement
Map duplicate identity violations to a clear, safe message.
Offer sign-in and password-recovery actions.
Preserve a request/error identifier for administrator correlation without
exposing database details.
7. Heterogeneous GPU groups need node-specific engine builds
Observed
The cluster contains GPU generations with different CUDA compute capabilities.
The configured vLLM release is valid for newer GPUs but not for older Volta
GPUs. llmgw's engines.vllm setting is global, so node admission and engine
installation attempt to apply the same vLLM version to every node. A valid
llama.cpp-only node can therefore produce a vLLM compatibility failure even
when no vLLM profile will ever run there.
Impact
Operators must choose between disabling vLLM for the whole cluster or using --ignore-checks. The latter hides unrelated admission problems, while the
former prevents newer GPU nodes from using vLLM. Different GPU groups may also
need different vLLM wheels, CUDA runtimes, or locally built binaries.
Expected / suggested improvement
Support an engine allowlist and version map per node or GPU class.
Allow, for example, a CUDA 13 vLLM build on newer GPUs and a CUDA 12, sm_70-compatible build on Volta nodes in the same cluster.
Let node add declare an engine as intentionally disabled for that node,
without requiring the broad --ignore-checks flag.
Preserve the existing gpu_class/engine model profile matching so a vLLM
profile can only select a node with a compatible installed engine.
Expose the installed engine version and supported compute capabilities in
node inventory and placement diagnostics.
Suggested priority
Prevent GPU-profile replicas from becoming ready on a CPU backend.
Improve restart behavior when no surge capacity exists.
Decouple unused fabric checks from base admission.
Improve signup error messages.
Add node/GPU-class-specific engine versions and allowlists.
8. llama.cpp multi-GPU placement is not represented
The agent accepts llama.cpp flags such as --split-mode layer and --tensor-split 1,1,1,1, but the scheduler does not reserve or place the
replica as a four-GPU allocation. A profile using tp: 4 remains unplaced or
can be interpreted as tensor-parallel planning without a matching llama.cpp
GPU reservation.
Suggested fix: add an explicit gpus_per_replica/GPU allocation field for
llama.cpp profiles, reserve that many GPUs on one node, and pass the selected
GPU list to CUDA_VISIBLE_DEVICES and --tensor-split. Reject ambiguous tp/llama.cpp combinations instead of silently ignoring the requested width.
9. vLLM multi-GPU models need a multi-file mirror manifest
vLLM can solve four-way tensor parallelism on T4, but it cannot load GGUF.
AWQ/Safetensors models are suitable, yet large repositories contain multiple
weight shards plus tokenizer and config files. The single source field can
mirror a repository URL as metadata rather than fetching the complete model,
leading to an invalid model path at startup.
Suggested fix: support a mirror manifest containing all files, checksums, and a
local directory path. Publish the directory atomically only after every shard
is present, then allow a vLLM profile to declare tp: 4 and reserve four GPUs.
# llmgw v0.10.6 deployment feedback
This note summarizes issues observed while deploying and operating llmgw
v0.10.6. Environment-specific identifiers, addresses, credentials, certificate
details, and organization-specific topology have intentionally been omitted.
## 1. Mirror UI silently creates CPU-only profiles for GGUF models
### Observed
Serving a `.gguf` file from the mirror UI creates a `llamacpp` profile without
`gpu_class`. The resulting model is consequently placed in the CPU pool. The UI
does not offer a CPU/GPU selector. Its editable "Profile name" field may suggest
that changing the name to a GPU-oriented value affects placement, but profile
names do not determine placement.
The behavior follows from the UI path selecting `llamacpp` for GGUF and setting
the initial profile to `cpu`; the submitted form does not populate GPU placement
fields such as `gpu_class` or `tp`.
### Impact
An administrator can successfully add and run a model while unintentionally
using a large amount of CPU and receiving much lower performance. Nothing in the
serve flow clearly explains why the GPU is unused.
### Expected / suggested improvement
- Offer a placement selector populated from discovered GPU classes.
- Show the actual placement semantics (`gpu_class` present or absent), rather
than relying on a freely editable profile name.
- Before submission, summarize: engine, CPU/GPU placement, GPU count, memory,
context size, and concurrency.
- If CPU is intentionally the default, display an explicit warning before
serving a GGUF model on a cluster with available GPUs.
## 2. GPU placement can silently execute with the CPU backend
### Observed
A model was assigned to a GPU profile and the generated llama.cpp command
contained `--n-gpu-layers 999`, but inference still ran at CPU-like speed and
GPU memory usage remained zero. The installed engine directory contained
`libggml-cuda.so`, while required CUDA runtime/cuBLAS shared libraries were not
available to the dynamic loader. llama.cpp continued by loading only its CPU
backend.
The control plane reported the replica as ready and placed in a GPU pool. The
failure became visible only through `ldd`, process mappings, and GPU telemetry.
### Impact
The scheduler reserves a GPU and reports a healthy GPU replica even though the
process performs inference on the CPU. This can be mistaken for poor GPU
performance or a model problem.
### Expected / suggested improvement
- Validate the CUDA backend and all dynamic dependencies during engine install.
- Make `engine-check` launch a small GPU operation, not only validate accepted
flags.
- When a GPU is assigned, fail readiness if llama.cpp reports no CUDA device or
does not load its CUDA backend.
- Surface backend selection and offloaded layer count in replica status.
- Do not allow silent CPU fallback for a profile that explicitly requires a
GPU, unless the profile has an explicit fallback policy.
## 3. Rolling restart can remain pending when every matching GPU is occupied
### Observed
After changing a model profile/resource value, `ctl restart --model` attempted
to place the replacement before stopping the existing replica. With every
matching GPU occupied, the replica remained indefinitely in a state similar to:
```text
ready (stale profile) (restart replacing)
```
The replacement could not be placed because the old replica itself held the
only suitable GPU. Completing the change required scaling the model to zero and
then back to one.
### Impact
Configuration changes appear accepted but never take effect. The operator must
understand the replacement algorithm and manually cause downtime.
### Expected / suggested improvement
- Detect when the old replica is the only placement blocker.
- Clearly report that a surge restart is impossible and offer an explicit
stop-then-start option.
- Consider `--max-unavailable` / `--max-surge` restart controls.
- Provide a one-command forced restart for single-replica, single-GPU cases.
## 4. Node admission is blocked by an unused incompatible engine
### Observed
`llmgw node add` rejected otherwise usable GPU nodes because the control plane's
configured vLLM release required a newer compute capability than those nodes
provided. The same nodes supported llama.cpp and were intended for llama.cpp
profiles only. Admission required `--ignore-checks`, after which the nodes joined
and advertised llama.cpp successfully.
### Impact
A heterogeneous cluster cannot cleanly add older GPU generations for a supported
engine when another globally configured engine is incompatible. Using
`--ignore-checks` also suppresses unrelated checks, making it broader than the
operator's intent.
### Expected / suggested improvement
- Treat engine incompatibility as per-node engine availability, not necessarily
a node-admission failure.
- Allow engines to be enabled/disabled per node or GPU class.
- Add a targeted option such as `--disable-engine vllm` instead of requiring
`--ignore-checks`.
- Show that the node will join with a reduced engine set and require confirmation
only when appropriate.
## 5. Fabric validation can block nodes that will not use that fabric
### Observed
Doctor treated a detected but unusable RoCE interface as a fatal issue during
node admission, even though no fabric was supplied and the node was being added
for single-node inference. Other detected interfaces also produced warnings
about missing fabric names and RDMA tooling.
### Impact
An unrelated or administratively unused adapter can prevent a node from joining
for local GPU workloads.
### Expected / suggested improvement
- Validate only explicitly configured fabrics as fatal admission checks.
- Report unconfigured adapters as informational or warnings.
- Separate base node readiness from optional multi-node/RDMA readiness.
## 6. Signup failures are too generic
### Observed
Attempting to sign up with an email address that already existed produced a
generic failure in the UI. Server-side diagnostics showed a unique-constraint
violation, but the UI did not explain that the account already existed or direct
the user toward sign-in/password recovery.
### Impact
Users cannot distinguish duplicate accounts from password-policy, database, or
server errors.
### Expected / suggested improvement
- Map duplicate identity violations to a clear, safe message.
- Offer sign-in and password-recovery actions.
- Preserve a request/error identifier for administrator correlation without
exposing database details.
## 7. Heterogeneous GPU groups need node-specific engine builds
### Observed
The cluster contains GPU generations with different CUDA compute capabilities.
The configured vLLM release is valid for newer GPUs but not for older Volta
GPUs. llmgw's `engines.vllm` setting is global, so node admission and engine
installation attempt to apply the same vLLM version to every node. A valid
llama.cpp-only node can therefore produce a vLLM compatibility failure even
when no vLLM profile will ever run there.
### Impact
Operators must choose between disabling vLLM for the whole cluster or using
`--ignore-checks`. The latter hides unrelated admission problems, while the
former prevents newer GPU nodes from using vLLM. Different GPU groups may also
need different vLLM wheels, CUDA runtimes, or locally built binaries.
### Expected / suggested improvement
- Support an engine allowlist and version map per node or GPU class.
- Allow, for example, a CUDA 13 vLLM build on newer GPUs and a CUDA 12,
`sm_70`-compatible build on Volta nodes in the same cluster.
- Let `node add` declare an engine as intentionally disabled for that node,
without requiring the broad `--ignore-checks` flag.
- Preserve the existing `gpu_class`/`engine` model profile matching so a vLLM
profile can only select a node with a compatible installed engine.
- Expose the installed engine version and supported compute capabilities in
node inventory and placement diagnostics.
## Suggested priority
1. Prevent GPU-profile replicas from becoming ready on a CPU backend.
2. Make GGUF placement explicit in the mirror UI.
3. Allow capability-aware, per-node engine availability.
4. Improve restart behavior when no surge capacity exists.
5. Decouple unused fabric checks from base admission.
6. Improve signup error messages.
7. Add node/GPU-class-specific engine versions and allowlists.
## 8. llama.cpp multi-GPU placement is not represented
The agent accepts llama.cpp flags such as `--split-mode layer` and
`--tensor-split 1,1,1,1`, but the scheduler does not reserve or place the
replica as a four-GPU allocation. A profile using `tp: 4` remains unplaced or
can be interpreted as tensor-parallel planning without a matching llama.cpp
GPU reservation.
Suggested fix: add an explicit `gpus_per_replica`/GPU allocation field for
llama.cpp profiles, reserve that many GPUs on one node, and pass the selected
GPU list to `CUDA_VISIBLE_DEVICES` and `--tensor-split`. Reject ambiguous
`tp`/llama.cpp combinations instead of silently ignoring the requested width.
## 9. vLLM multi-GPU models need a multi-file mirror manifest
vLLM can solve four-way tensor parallelism on T4, but it cannot load GGUF.
AWQ/Safetensors models are suitable, yet large repositories contain multiple
weight shards plus tokenizer and config files. The single `source` field can
mirror a repository URL as metadata rather than fetching the complete model,
leading to an invalid model path at startup.
Suggested fix: support a mirror manifest containing all files, checksums, and a
local directory path. Publish the directory atomically only after every shard
is present, then allow a vLLM profile to declare `tp: 4` and reserve four GPUs.
Everything in this report is now implemented. Most of it shipped in v0.11.0 (CHANGELOG has one heading per item). The rest is on main after v0.12.2 and goes out with the next release. Each item was checked against the code; the parts that depend on hardware were checked on the reference cluster (a Quadro RTX 6000 node and CPU nodes).
1. Mirror UI and GGUF placement (v0.11.0)
A placement selector is built from the discovered GPU classes, and the profile is derived from it instead of a free-typed name.
A summary is shown before submitting: engine, CPU/GPU, GPU count, memory, context pool, concurrency.
Serving on the CPU while GPUs are free is refused unless "run on the CPU anyway" is ticked.
2. GPU placement running on the CPU backend
Engine install validates the CUDA backend (--list-devices, ldd on libggml-cuda.so) (v0.11.0).
A GPU replica whose processes hold no memory on their GPUs is refused (v0.11.0). It is also refused when llama.cpp warns that it found no usable GPU, which covers nodes where nvidia-smi cannot be asked (2f595f5, b5d650a).
Replica status shows what each replica computes on: ctl replicas (ON column), the console and the admin API show CUDA with the GPUs the replica holds and how much, or CPU. This works for every engine and survives agent and control-plane restarts (b5d650a, 25b3e12). llama.cpp prints its offloaded-layer count only at -lv 4, so the count appears when it says so.
ctl engine-check llamacpp now loads a small GGUF with every layer on the GPU and asks it for one token. Output on the reference GPU node: computed a token on CUDA: CUDA0: Quadro RTX 6000, 25 of 25 layers offloaded (e4e1520, 2f595f5).
3. Rolling restart stuck when every GPU is occupied
A replica that is the only thing in its replacement's way is stopped first; one that fits nowhere is refused, with the reason (v0.11.0).
ctl restart --in-place stops each replica before starting its replacement (v0.11.0).
ctl restart --max-surge N and --max-unavailable N restart that many of a model's replicas at once (5ae870e).
4. Node admission blocked by an unused engine (v0.11.0)
An incompatible engine is per-node availability; it no longer fails admission.
node add --disable-engine vllm, and engine_overrides per GPU class, node or label.
node add says the node joins with a reduced engine set.
5. Fabric validation blocking nodes
Only fabrics declared with --fabric are fatal; other adapters are warnings (v0.11.0).
Doctor reports two separate readinesses, the node's own and multi-node (fabrics/RDMA), with fabric findings in their own section. node add reports the multi-node one, and ctl nodes names a declared fabric's adapters that are down (f2b2e24).
6. Signup with an existing address
Duplicate detection is fixed for PostgreSQL. The page gives an error reference for operators and links to sign-in (v0.11.0).
The page still does not say outright that an address is taken; that is deliberate, so the form cannot be used to enumerate accounts. Instead it links to the new password reset.
Self-service password reset: /forgot mails a one-time link when the cluster's mail settings are set (SMTP relay, sender, and a public base_url; the link never uses the request's Host). The reply is the same whether or not the address has an account. Without mail, llmgw users reset-link makes the link for an operator to hand over (66f702d). The mail path is tested against a stub SMTP server, not yet a real relay.
7. Heterogeneous GPU engine builds
engine_overrides per GPU class, node or label, CUDA 13 and CUDA 12/Volta vLLM builds in the same cluster, and ctl engine-build per node (v0.11.0).
ctl nodes and the admin API show each node's engine versions and its GPUs' compute capability, for example llamacpp b10991 and 6000-24g (7.5).
A profile's engine_version restricts it to nodes whose engine version starts with that value. explain-placement says which version a node has, or which engines it has when it lacks the one asked for (619dd75).
8. llama.cpp multi-GPU (v0.11.0)
gpus_per_node reserves that many GPUs on one node and sets the visible devices; llama.cpp splits the model by layer across them.
Ambiguous tp/pp combinations with llama.cpp are rejected.
9. vLLM multi-file mirror (v0.11.0)
The mirror keeps a checksummed manifest of every file: shards, config and tokenizer.
The manifest is not served until it is complete, and a node marks its copy complete only after every file is there.
A directory source can be served by vLLM with tp: 4.
Also fixed while verifying this: a standby that took over as leader placed new replicas before the agents had reconnected, then drained the healthy replicas when they reported in. On the reference cluster that replaced the guardrails' judge and a model's replica during a leader restart. A new leader now waits for the fleet, as at start, and three failovers since have adopted every replica unchanged (0add189).
Closing; please reopen if anything here does not match what you see in the field.
Everything in this report is now implemented. Most of it shipped in **v0.11.0** (CHANGELOG has one heading per item). The rest is on `main` after v0.12.2 and goes out with the next release. Each item was checked against the code; the parts that depend on hardware were checked on the reference cluster (a Quadro RTX 6000 node and CPU nodes).
**1. Mirror UI and GGUF placement** (v0.11.0)
- A placement selector is built from the discovered GPU classes, and the profile is derived from it instead of a free-typed name.
- A summary is shown before submitting: engine, CPU/GPU, GPU count, memory, context pool, concurrency.
- Serving on the CPU while GPUs are free is refused unless "run on the CPU anyway" is ticked.
**2. GPU placement running on the CPU backend**
- Engine install validates the CUDA backend (`--list-devices`, `ldd` on `libggml-cuda.so`) (v0.11.0).
- A GPU replica whose processes hold no memory on their GPUs is refused (v0.11.0). It is also refused when llama.cpp warns that it found no usable GPU, which covers nodes where nvidia-smi cannot be asked (2f595f5, b5d650a).
- Replica status shows what each replica computes on: `ctl replicas` (ON column), the console and the admin API show `CUDA` with the GPUs the replica holds and how much, or `CPU`. This works for every engine and survives agent and control-plane restarts (b5d650a, 25b3e12). llama.cpp prints its offloaded-layer count only at `-lv 4`, so the count appears when it says so.
- `ctl engine-check llamacpp` now loads a small GGUF with every layer on the GPU and asks it for one token. Output on the reference GPU node: `computed a token on CUDA: CUDA0: Quadro RTX 6000, 25 of 25 layers offloaded` (e4e1520, 2f595f5).
**3. Rolling restart stuck when every GPU is occupied**
- A replica that is the only thing in its replacement's way is stopped first; one that fits nowhere is refused, with the reason (v0.11.0).
- `ctl restart --in-place` stops each replica before starting its replacement (v0.11.0).
- `ctl restart --max-surge N` and `--max-unavailable N` restart that many of a model's replicas at once (5ae870e).
**4. Node admission blocked by an unused engine** (v0.11.0)
- An incompatible engine is per-node availability; it no longer fails admission.
- `node add --disable-engine vllm`, and `engine_overrides` per GPU class, node or label.
- `node add` says the node joins with a reduced engine set.
**5. Fabric validation blocking nodes**
- Only fabrics declared with `--fabric` are fatal; other adapters are warnings (v0.11.0).
- Doctor reports two separate readinesses, the node's own and multi-node (fabrics/RDMA), with fabric findings in their own section. `node add` reports the multi-node one, and `ctl nodes` names a declared fabric's adapters that are down (f2b2e24).
**6. Signup with an existing address**
- Duplicate detection is fixed for PostgreSQL. The page gives an error reference for operators and links to sign-in (v0.11.0).
- The page still does not say outright that an address is taken; that is deliberate, so the form cannot be used to enumerate accounts. Instead it links to the new password reset.
- Self-service password reset: `/forgot` mails a one-time link when the cluster's `mail` settings are set (SMTP relay, sender, and a public `base_url`; the link never uses the request's Host). The reply is the same whether or not the address has an account. Without mail, `llmgw users reset-link` makes the link for an operator to hand over (66f702d). The mail path is tested against a stub SMTP server, not yet a real relay.
**7. Heterogeneous GPU engine builds**
- `engine_overrides` per GPU class, node or label, CUDA 13 and CUDA 12/Volta vLLM builds in the same cluster, and `ctl engine-build` per node (v0.11.0).
- `ctl nodes` and the admin API show each node's engine versions and its GPUs' compute capability, for example `llamacpp b10991` and `6000-24g (7.5)`.
- A profile's `engine_version` restricts it to nodes whose engine version starts with that value. explain-placement says which version a node has, or which engines it has when it lacks the one asked for (619dd75).
**8. llama.cpp multi-GPU** (v0.11.0)
- `gpus_per_node` reserves that many GPUs on one node and sets the visible devices; llama.cpp splits the model by layer across them.
- Ambiguous `tp`/`pp` combinations with llama.cpp are rejected.
**9. vLLM multi-file mirror** (v0.11.0)
- The mirror keeps a checksummed manifest of every file: shards, config and tokenizer.
- The manifest is not served until it is complete, and a node marks its copy complete only after every file is there.
- A directory source can be served by vLLM with `tp: 4`.
**Also fixed while verifying this:** a standby that took over as leader placed new replicas before the agents had reconnected, then drained the healthy replicas when they reported in. On the reference cluster that replaced the guardrails' judge and a model's replica during a leader restart. A new leader now waits for the fleet, as at start, and three failovers since have adopted every replica unchanged (0add189).
Closing; please reopen if anything here does not match what you see in the field.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
llmgw v0.10.6 deployment feedback
This note summarizes issues observed while deploying and operating llmgw
v0.10.6. Environment-specific identifiers, addresses, credentials, certificate
details, and organization-specific topology have intentionally been omitted.
1. Mirror UI silently creates CPU-only profiles for GGUF models
Observed
Serving a
.gguffile from the mirror UI creates allamacppprofile withoutgpu_class. The resulting model is consequently placed in the CPU pool. The UIdoes not offer a CPU/GPU selector. Its editable "Profile name" field may suggest
that changing the name to a GPU-oriented value affects placement, but profile
names do not determine placement.
The behavior follows from the UI path selecting
llamacppfor GGUF and settingthe initial profile to
cpu; the submitted form does not populate GPU placementfields such as
gpu_classortp.Impact
An administrator can successfully add and run a model while unintentionally
using a large amount of CPU and receiving much lower performance. Nothing in the
serve flow clearly explains why the GPU is unused.
Expected / suggested improvement
gpu_classpresent or absent), ratherthan relying on a freely editable profile name.
context size, and concurrency.
serving a GGUF model on a cluster with available GPUs.
2. GPU placement can silently execute with the CPU backend
Observed
A model was assigned to a GPU profile and the generated llama.cpp command
contained
--n-gpu-layers 999, but inference still ran at CPU-like speed andGPU memory usage remained zero. The installed engine directory contained
libggml-cuda.so, while required CUDA runtime/cuBLAS shared libraries were notavailable to the dynamic loader. llama.cpp continued by loading only its CPU
backend.
The control plane reported the replica as ready and placed in a GPU pool. The
failure became visible only through
ldd, process mappings, and GPU telemetry.Impact
The scheduler reserves a GPU and reports a healthy GPU replica even though the
process performs inference on the CPU. This can be mistaken for poor GPU
performance or a model problem.
Expected / suggested improvement
engine-checklaunch a small GPU operation, not only validate acceptedflags.
does not load its CUDA backend.
GPU, unless the profile has an explicit fallback policy.
3. Rolling restart can remain pending when every matching GPU is occupied
Observed
After changing a model profile/resource value,
ctl restart --modelattemptedto place the replacement before stopping the existing replica. With every
matching GPU occupied, the replica remained indefinitely in a state similar to:
The replacement could not be placed because the old replica itself held the
only suitable GPU. Completing the change required scaling the model to zero and
then back to one.
Impact
Configuration changes appear accepted but never take effect. The operator must
understand the replacement algorithm and manually cause downtime.
Expected / suggested improvement
stop-then-start option.
--max-unavailable/--max-surgerestart controls.4. Node admission is blocked by an unused incompatible engine
Observed
llmgw node addrejected otherwise usable GPU nodes because the control plane'sconfigured vLLM release required a newer compute capability than those nodes
provided. The same nodes supported llama.cpp and were intended for llama.cpp
profiles only. Admission required
--ignore-checks, after which the nodes joinedand advertised llama.cpp successfully.
Impact
A heterogeneous cluster cannot cleanly add older GPU generations for a supported
engine when another globally configured engine is incompatible. Using
--ignore-checksalso suppresses unrelated checks, making it broader than theoperator's intent.
Expected / suggested improvement
a node-admission failure.
--disable-engine vllminstead of requiring--ignore-checks.only when appropriate.
5. Fabric validation can block nodes that will not use that fabric
Observed
Doctor treated a detected but unusable RoCE interface as a fatal issue during
node admission, even though no fabric was supplied and the node was being added
for single-node inference. Other detected interfaces also produced warnings
about missing fabric names and RDMA tooling.
Impact
An unrelated or administratively unused adapter can prevent a node from joining
for local GPU workloads.
Expected / suggested improvement
6. Signup failures are too generic
Observed
Attempting to sign up with an email address that already existed produced a
generic failure in the UI. Server-side diagnostics showed a unique-constraint
violation, but the UI did not explain that the account already existed or direct
the user toward sign-in/password recovery.
Impact
Users cannot distinguish duplicate accounts from password-policy, database, or
server errors.
Expected / suggested improvement
exposing database details.
7. Heterogeneous GPU groups need node-specific engine builds
Observed
The cluster contains GPU generations with different CUDA compute capabilities.
The configured vLLM release is valid for newer GPUs but not for older Volta
GPUs. llmgw's
engines.vllmsetting is global, so node admission and engineinstallation attempt to apply the same vLLM version to every node. A valid
llama.cpp-only node can therefore produce a vLLM compatibility failure even
when no vLLM profile will ever run there.
Impact
Operators must choose between disabling vLLM for the whole cluster or using
--ignore-checks. The latter hides unrelated admission problems, while theformer prevents newer GPU nodes from using vLLM. Different GPU groups may also
need different vLLM wheels, CUDA runtimes, or locally built binaries.
Expected / suggested improvement
sm_70-compatible build on Volta nodes in the same cluster.node adddeclare an engine as intentionally disabled for that node,without requiring the broad
--ignore-checksflag.gpu_class/enginemodel profile matching so a vLLMprofile can only select a node with a compatible installed engine.
node inventory and placement diagnostics.
Suggested priority
8. llama.cpp multi-GPU placement is not represented
The agent accepts llama.cpp flags such as
--split-mode layerand--tensor-split 1,1,1,1, but the scheduler does not reserve or place thereplica as a four-GPU allocation. A profile using
tp: 4remains unplaced orcan be interpreted as tensor-parallel planning without a matching llama.cpp
GPU reservation.
Suggested fix: add an explicit
gpus_per_replica/GPU allocation field forllama.cpp profiles, reserve that many GPUs on one node, and pass the selected
GPU list to
CUDA_VISIBLE_DEVICESand--tensor-split. Reject ambiguoustp/llama.cpp combinations instead of silently ignoring the requested width.9. vLLM multi-GPU models need a multi-file mirror manifest
vLLM can solve four-way tensor parallelism on T4, but it cannot load GGUF.
AWQ/Safetensors models are suitable, yet large repositories contain multiple
weight shards plus tokenizer and config files. The single
sourcefield canmirror a repository URL as metadata rather than fetching the complete model,
leading to an invalid model path at startup.
Suggested fix: support a mirror manifest containing all files, checksums, and a
local directory path. Publish the directory atomically only after every shard
is present, then allow a vLLM profile to declare
tp: 4and reserve four GPUs.Everything in this report is now implemented. Most of it shipped in v0.11.0 (CHANGELOG has one heading per item). The rest is on
mainafter v0.12.2 and goes out with the next release. Each item was checked against the code; the parts that depend on hardware were checked on the reference cluster (a Quadro RTX 6000 node and CPU nodes).1. Mirror UI and GGUF placement (v0.11.0)
2. GPU placement running on the CPU backend
--list-devices,lddonlibggml-cuda.so) (v0.11.0).2f595f5,b5d650a).ctl replicas(ON column), the console and the admin API showCUDAwith the GPUs the replica holds and how much, orCPU. This works for every engine and survives agent and control-plane restarts (b5d650a,25b3e12). llama.cpp prints its offloaded-layer count only at-lv 4, so the count appears when it says so.ctl engine-check llamacppnow loads a small GGUF with every layer on the GPU and asks it for one token. Output on the reference GPU node:computed a token on CUDA: CUDA0: Quadro RTX 6000, 25 of 25 layers offloaded(e4e1520,2f595f5).3. Rolling restart stuck when every GPU is occupied
ctl restart --in-placestops each replica before starting its replacement (v0.11.0).ctl restart --max-surge Nand--max-unavailable Nrestart that many of a model's replicas at once (5ae870e).4. Node admission blocked by an unused engine (v0.11.0)
node add --disable-engine vllm, andengine_overridesper GPU class, node or label.node addsays the node joins with a reduced engine set.5. Fabric validation blocking nodes
--fabricare fatal; other adapters are warnings (v0.11.0).node addreports the multi-node one, andctl nodesnames a declared fabric's adapters that are down (f2b2e24).6. Signup with an existing address
/forgotmails a one-time link when the cluster'smailsettings are set (SMTP relay, sender, and a publicbase_url; the link never uses the request's Host). The reply is the same whether or not the address has an account. Without mail,llmgw users reset-linkmakes the link for an operator to hand over (66f702d). The mail path is tested against a stub SMTP server, not yet a real relay.7. Heterogeneous GPU engine builds
engine_overridesper GPU class, node or label, CUDA 13 and CUDA 12/Volta vLLM builds in the same cluster, andctl engine-buildper node (v0.11.0).ctl nodesand the admin API show each node's engine versions and its GPUs' compute capability, for examplellamacpp b10991and6000-24g (7.5).engine_versionrestricts it to nodes whose engine version starts with that value. explain-placement says which version a node has, or which engines it has when it lacks the one asked for (619dd75).8. llama.cpp multi-GPU (v0.11.0)
gpus_per_nodereserves that many GPUs on one node and sets the visible devices; llama.cpp splits the model by layer across them.tp/ppcombinations with llama.cpp are rejected.9. vLLM multi-file mirror (v0.11.0)
tp: 4.Also fixed while verifying this: a standby that took over as leader placed new replicas before the agents had reconnected, then drained the healthy replicas when they reported in. On the reference cluster that replaced the guardrails' judge and a model's replica during a leader restart. A new leader now waits for the fleet, as at start, and three failovers since have adopted every replica unchanged (
0add189).Closing; please reopen if anything here does not match what you see in the field.