sudo llmgw help
llmgw serves LLMs running on your own GPU servers behind a single
OpenAI-compatible endpoint. One binary, different roles per subcommand.
Usage:
llmgw [flags]
llmgw [command]
Available Commands:
agent Node agent: discovery and engine supervision
bundle Carry releases, engines and models to a site with no internet
completion Generate the autocompletion script for the specified shell
ctl Inspect and operate a running cluster
docs Read the manual: every document, and a page for every command
doctor Check this node's GPUs, NVLink, InfiniBand and engines for what llmgw needs
firewall llmgw's firewall on this machine
help Help about any command
join Let a setup on another machine set this one up, with the code it gave
keys Manage tenants and API keys
mirror Hold model weights for the fleet to fetch from
node Add machines to the cluster
org Organizations, teams and projects: see, add, rename, move, remove
release Sign releases, and make the key they are signed with
server Run the router (phase 1; the control plane arrives in phase 3)
sim-engine OpenAI-compatible fake inference engine (for testing)
upgrade Upgrade this machine's llmgw, and restart the service that runs it
users Approve and manage the people who sign up
wipe Remove llmgw from this machine: its services, configuration and state
Flags:
-h, --help help for llmgw
--log-format string log format: text|json (default "text")
--log-level string log level: debug|info|warn|error (default "info")
-v, --version version for llmgw
Use "llmgw [command] --help" for more information about a command. bu sistemi durdurma yada tekrar başlatmak için komut yok
sudo llmgw help
llmgw serves LLMs running on your own GPU servers behind a single
OpenAI-compatible endpoint. One binary, different roles per subcommand.
Usage:
llmgw [flags]
llmgw [command]
Available Commands:
agent Node agent: discovery and engine supervision
bundle Carry releases, engines and models to a site with no internet
completion Generate the autocompletion script for the specified shell
ctl Inspect and operate a running cluster
docs Read the manual: every document, and a page for every command
doctor Check this node's GPUs, NVLink, InfiniBand and engines for what llmgw needs
firewall llmgw's firewall on this machine
help Help about any command
join Let a setup on another machine set this one up, with the code it gave
keys Manage tenants and API keys
mirror Hold model weights for the fleet to fetch from
node Add machines to the cluster
org Organizations, teams and projects: see, add, rename, move, remove
release Sign releases, and make the key they are signed with
server Run the router (phase 1; the control plane arrives in phase 3)
sim-engine OpenAI-compatible fake inference engine (for testing)
upgrade Upgrade this machine's llmgw, and restart the service that runs it
users Approve and manage the people who sign up
wipe Remove llmgw from this machine: its services, configuration and state
Flags:
-h, --help help for llmgw
--log-format string log format: text|json (default "text")
--log-level string log level: debug|info|warn|error (default "info")
-v, --version version for llmgw
Use "llmgw [command] --help" for more information about a command. bu sistemi durdurma yada tekrar başlatmak için komut yok
Added in 05aa21c, on main for the next release: commands to stop, start and restart llmgw on a machine.
sudo llmgw status # each llmgw service here, and whether it runs
sudo llmgw stop # agent, control plane, mirror: whichever this machine has
sudo llmgw start
sudo llmgw restart agent # or name one: server, agent, mirror
They act on this machine's llmgw services in the order they depend on each other: mirror, server, agent to start, and the reverse to stop.
Stopping a node's agent leaves its engines serving; the agent takes them over again when it starts. llmgw stop --engines stops them too, and the control plane then places their replicas elsewhere.
Stopping the leading control plane hands over to a standby within seconds.
The whole cluster's state is still llmgw ctl status.
Documented in docs/install.md, "Stopping and starting a machine". Tried on the reference cluster: a node's agent stopped and started, with its engines serving throughout.
Added in 05aa21c, on `main` for the next release: commands to stop, start and restart llmgw on a machine.
```
sudo llmgw status # each llmgw service here, and whether it runs
sudo llmgw stop # agent, control plane, mirror: whichever this machine has
sudo llmgw start
sudo llmgw restart agent # or name one: server, agent, mirror
```
- They act on this machine's llmgw services in the order they depend on each other: mirror, server, agent to start, and the reverse to stop.
- Stopping a node's agent leaves its engines serving; the agent takes them over again when it starts. `llmgw stop --engines` stops them too, and the control plane then places their replicas elsewhere.
- Stopping the leading control plane hands over to a standby within seconds.
- The whole cluster's state is still `llmgw ctl status`.
Documented in docs/install.md, "Stopping and starting a machine". Tried on the reference cluster: a node's agent stopped and started, with its engines serving throughout.
Reopened: the commands added in 05aa21c act on one machine only. The request is to stop and start the whole system - every machine of the cluster - which is still missing.
Reopened: the commands added in 05aa21c act on one machine only. The request is to stop and start the whole system - every machine of the cluster - which is still missing.
The whole system can now be stopped, started and restarted with one command each, run on any control plane (leader or standby). This is on main and goes out with the next release.
sudo llmgw stop [--wait] [--note "why"]
Every model's replicas are taken down, which frees every GPU.
Every request for a model is answered 503 system_stopped with Retry-After.
Control planes, agents and the mirror keep running, so the console, the admin pages and llmgw start still work.
No model's definition is changed.
It is a cluster setting (serving.stopped), so it applies on every control plane within seconds and a restart does not undo it.
sudo llmgw start [--wait]
Brings every model back.
--wait returns once every model has the replicas it asks for.
sudo llmgw restart [--ssh-user U]
Restarts every node's agent and every control plane, the leader last. Engines keep serving throughout.
If a control plane can't be reached over ssh, the restart continues with the rest and names that machine at the end.
llmgw status
Shows SERVICE STOPPED while the system is stopped.
llmgw service status|start|stop|restart [server|agent|mirror]
Acts on the llmgw services of the one machine it runs on.
Tested on the reference cluster (7 nodes, 2 control planes, Quadro RTX 6000):
stop --wait finished in 10 s. Afterwards no replica was running, the GPU was down to 1 MiB, and the VIP and both control planes answered 503 system_stopped. All agents, control planes and the mirror stayed active.
start --wait returned after 56 s, with all 7 replicas ready and the GPU model back on CUDA. Running it from the standby works too.
Two bugs found during testing are fixed:
start --wait returned before the control plane had applied the setting (63f7c8c).
After a restart, an agent that registered before its adopted engines passed their health check left their replicas "starting" while they served, until the startup timeout replaced them (2d14918).
Closing; please reopen if anything here does not match what you see in the field.
The whole system can now be stopped, started and restarted with one command each, run on any control plane (leader or standby). This is on `main` and goes out with the next release.
**`sudo llmgw stop [--wait] [--note "why"]`**
- Every model's replicas are taken down, which frees every GPU.
- Every request for a model is answered `503 system_stopped` with `Retry-After`.
- Control planes, agents and the mirror keep running, so the console, the admin pages and `llmgw start` still work.
- No model's definition is changed.
- It is a cluster setting (`serving.stopped`), so it applies on every control plane within seconds and a restart does not undo it.
**`sudo llmgw start [--wait]`**
- Brings every model back.
- `--wait` returns once every model has the replicas it asks for.
**`sudo llmgw restart [--ssh-user U]`**
- Restarts every node's agent and every control plane, the leader last. Engines keep serving throughout.
- If a control plane can't be reached over ssh, the restart continues with the rest and names that machine at the end.
**`llmgw status`**
- Shows `SERVICE STOPPED` while the system is stopped.
**`llmgw service status|start|stop|restart [server|agent|mirror]`**
- Acts on the llmgw services of the one machine it runs on.
Tested on the reference cluster (7 nodes, 2 control planes, Quadro RTX 6000):
- `stop --wait` finished in 10 s. Afterwards no replica was running, the GPU was down to 1 MiB, and the VIP and both control planes answered 503 `system_stopped`. All agents, control planes and the mirror stayed active.
- `start --wait` returned after 56 s, with all 7 replicas ready and the GPU model back on CUDA. Running it from the standby works too.
- Two bugs found during testing are fixed:
- `start --wait` returned before the control plane had applied the setting (63f7c8c).
- After a restart, an agent that registered before its adopted engines passed their health check left their replicas "starting" while they served, until the startup timeout replaced them (2d14918).
Closing; please reopen if anything here does not match what you see in the field.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
sudo llmgw help
llmgw serves LLMs running on your own GPU servers behind a single
OpenAI-compatible endpoint. One binary, different roles per subcommand.
Usage:
llmgw [flags]
llmgw [command]
Available Commands:
agent Node agent: discovery and engine supervision
bundle Carry releases, engines and models to a site with no internet
completion Generate the autocompletion script for the specified shell
ctl Inspect and operate a running cluster
docs Read the manual: every document, and a page for every command
doctor Check this node's GPUs, NVLink, InfiniBand and engines for what llmgw needs
firewall llmgw's firewall on this machine
help Help about any command
join Let a setup on another machine set this one up, with the code it gave
keys Manage tenants and API keys
mirror Hold model weights for the fleet to fetch from
node Add machines to the cluster
org Organizations, teams and projects: see, add, rename, move, remove
release Sign releases, and make the key they are signed with
server Run the router (phase 1; the control plane arrives in phase 3)
sim-engine OpenAI-compatible fake inference engine (for testing)
upgrade Upgrade this machine's llmgw, and restart the service that runs it
users Approve and manage the people who sign up
wipe Remove llmgw from this machine: its services, configuration and state
Flags:
-h, --help help for llmgw
--log-format string log format: text|json (default "text")
--log-level string log level: debug|info|warn|error (default "info")
-v, --version version for llmgw
Use "llmgw [command] --help" for more information about a command. bu sistemi durdurma yada tekrar başlatmak için komut yok
Added in
05aa21c, onmainfor the next release: commands to stop, start and restart llmgw on a machine.llmgw stop --enginesstops them too, and the control plane then places their replicas elsewhere.llmgw ctl status.Documented in docs/install.md, "Stopping and starting a machine". Tried on the reference cluster: a node's agent stopped and started, with its engines serving throughout.
Reopened: the commands added in
05aa21cact on one machine only. The request is to stop and start the whole system - every machine of the cluster - which is still missing.The whole system can now be stopped, started and restarted with one command each, run on any control plane (leader or standby). This is on
mainand goes out with the next release.sudo llmgw stop [--wait] [--note "why"]503 system_stoppedwithRetry-After.llmgw startstill work.serving.stopped), so it applies on every control plane within seconds and a restart does not undo it.sudo llmgw start [--wait]--waitreturns once every model has the replicas it asks for.sudo llmgw restart [--ssh-user U]llmgw statusSERVICE STOPPEDwhile the system is stopped.llmgw service status|start|stop|restart [server|agent|mirror]Tested on the reference cluster (7 nodes, 2 control planes, Quadro RTX 6000):
stop --waitfinished in 10 s. Afterwards no replica was running, the GPU was down to 1 MiB, and the VIP and both control planes answered 503system_stopped. All agents, control planes and the mirror stayed active.start --waitreturned after 56 s, with all 7 replicas ready and the GPU model back on CUDA. Running it from the standby works too.start --waitreturned before the control plane had applied the setting (63f7c8c).2d14918).Closing; please reopen if anything here does not match what you see in the field.