|
|
|
@@ -20,6 +20,10 @@ Until v1.0.0 the minor version is the breaking one: configuration formats and Go
|
|
|
|
|
APIs change between minors without a deprecation period.
|
|
|
|
|
|
|
|
|
|
## Unreleased
|
|
|
|
|
Independent InfiniBand fabrics and NVLink domains, told apart, drawn and
|
|
|
|
|
placed on; a node checked by `llmgw doctor` and added by `llmgw node add`; vLLM
|
|
|
|
|
installed by the nodes themselves; people in many places at once; and a chat
|
|
|
|
|
page, editor setup and a model explorer for the people using it.
|
|
|
|
|
|
|
|
|
|
### Independent InfiniBand fabrics, told apart and drawn
|
|
|
|
|
|
|
|
|
@@ -100,55 +104,36 @@ interface NCCL will use, the unit's memlock limit, and the NCCL environment a
|
|
|
|
|
shard on each fabric would be given. `--json` writes every fact it read, to
|
|
|
|
|
send back; it exits non-zero when something fails.
|
|
|
|
|
|
|
|
|
|
### Each node's CPU and memory, and a page per node
|
|
|
|
|
It also says when the CPU offers no AVX2 — a VM left on QEMU's default CPU
|
|
|
|
|
model, where llama.cpp runs several times slower and nothing else looks wrong —
|
|
|
|
|
and whether vLLM can be installed on the node, and why not.
|
|
|
|
|
|
|
|
|
|
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
|
|
|
|
|
used and available, and each engine's cores and resident memory. The cluster
|
|
|
|
|
page's node table shows CPU and RAM as bars; each node has a page with its
|
|
|
|
|
current load, its engines, its fabric ports, its GPUs, and an hour of charts
|
|
|
|
|
from Prometheus.
|
|
|
|
|
### vLLM installs itself on GPU nodes
|
|
|
|
|
|
|
|
|
|
### Connect an editor, and the Responses API
|
|
|
|
|
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
|
|
|
|
|
that can run it, with nothing done by hand. A Python 3.12 environment is built
|
|
|
|
|
by uv — itself a GitHub release checked against its checksum — from PyPI,
|
|
|
|
|
resolved as PyPI stood a day after that vLLM was published and installed with
|
|
|
|
|
`--require-hashes`, so every node gets the same files whenever it is built. The
|
|
|
|
|
resolved list stays in the environment as `llmgw-requirements.txt`.
|
|
|
|
|
|
|
|
|
|
- The account page makes a key for **Continue**, **Codex** or **Cline** and
|
|
|
|
|
shows what the tool needs, filled in: Continue's assistant file, Codex's
|
|
|
|
|
provider and profile, Cline's four settings. Continue and Codex get a one-line
|
|
|
|
|
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
|
|
|
|
|
no secret — the key comes from the environment — keeps a copy of whatever it
|
|
|
|
|
replaces, and leaves the rest of the file alone.
|
|
|
|
|
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
|
|
|
|
|
chat completions and back — instructions, messages, function and custom
|
|
|
|
|
tools with their calls and outputs, usage, streamed or not — down the same
|
|
|
|
|
path as any request.
|
|
|
|
|
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
|
|
|
|
|
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
|
|
|
|
|
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
|
|
|
|
|
25 GB free while it installs. A node that lacks one says which on the
|
|
|
|
|
cluster page; a node with no GPU skips vLLM without a word.
|
|
|
|
|
- gcc is installed with it, since Triton compiles its kernel launcher on the
|
|
|
|
|
first model load.
|
|
|
|
|
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
|
|
|
|
|
version.
|
|
|
|
|
|
|
|
|
|
### The chat page
|
|
|
|
|
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
|
|
|
|
|
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
|
|
|
|
|
itself was not started there.
|
|
|
|
|
|
|
|
|
|
- A model that is not running shows a progress bar while it loads — waiting
|
|
|
|
|
for a node, downloading, starting — and the message is sent once it is up,
|
|
|
|
|
rather than failing after 45 seconds with "the model is loading".
|
|
|
|
|
- A finished reply is rendered as markdown, by the server under the model
|
|
|
|
|
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
|
|
|
|
|
- Stop, while a reply streams; its tokens, speed and time to first token under
|
|
|
|
|
it; a system prompt and a temperature; and the conversation kept in the
|
|
|
|
|
browser across a reload — the gateway still keeps none.
|
|
|
|
|
|
|
|
|
|
### The mirror page
|
|
|
|
|
|
|
|
|
|
- **Explore models** opens on what is trending on Hugging Face, with views for
|
|
|
|
|
new and most downloaded, filters for task and size, and each repository's
|
|
|
|
|
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
|
|
|
|
|
base model and license. "Only what these nodes can run" is on by default.
|
|
|
|
|
- A repository opens beneath its row with its model card, and a download
|
|
|
|
|
started there starts in place, its progress followed in the table above.
|
|
|
|
|
- What the mirror holds is grouped by model family.
|
|
|
|
|
- An importance matrix published beside a model's quantisations is no longer
|
|
|
|
|
offered as a model.
|
|
|
|
|
|
|
|
|
|
### Your own use
|
|
|
|
|
|
|
|
|
|
The account page (was "Your keys") shows the last thirty days of a person's own
|
|
|
|
|
use and, for each place they spend, a bar for every budget that applies — their
|
|
|
|
|
own, or a project's shared with others, said as such.
|
|
|
|
|
A node is asked only for engines its agent says it can install. An agent older
|
|
|
|
|
than this release says nothing, and is asked for llama.cpp alone, so a cluster
|
|
|
|
|
half way through its upgrade does not show every node as failing vLLM.
|
|
|
|
|
|
|
|
|
|
### `llmgw node add`: a node in one command
|
|
|
|
|
|
|
|
|
@@ -181,114 +166,6 @@ longer lists vLLM as skipped on every CPU node.
|
|
|
|
|
Tried on an empty VM, from the terminal screen and plain: joined with
|
|
|
|
|
llama.cpp ready in under a minute, and 17 seconds to add it again.
|
|
|
|
|
|
|
|
|
|
### vLLM installs itself on GPU nodes
|
|
|
|
|
|
|
|
|
|
The agent installs vLLM as it does llama.cpp: on every node with an NVIDIA GPU
|
|
|
|
|
that can run it, with nothing done by hand. A Python 3.12 environment is built
|
|
|
|
|
by uv — itself a GitHub release checked against its checksum — from PyPI,
|
|
|
|
|
resolved as PyPI stood a day after that vLLM was published and installed with
|
|
|
|
|
`--require-hashes`, so every node gets the same files whenever it is built. The
|
|
|
|
|
resolved list stays in the environment as `llmgw-requirements.txt`.
|
|
|
|
|
|
|
|
|
|
- The default is vLLM 0.30.0, which takes every flag llmgw passes. Set
|
|
|
|
|
`engines.vllm` in `server.yaml` to change it, or to `""` to keep your own.
|
|
|
|
|
- It is built for CUDA 13: it needs driver 580 and a Turing GPU or newer, and
|
|
|
|
|
25 GB free while it installs. A node that lacks one says which on the
|
|
|
|
|
cluster page; a node with no GPU skips vLLM without a word.
|
|
|
|
|
- gcc is installed with it, since Triton compiles its kernel launcher on the
|
|
|
|
|
first model load.
|
|
|
|
|
- SGLang installs the same way, on GPU nodes, when `engines.sglang` names a
|
|
|
|
|
version.
|
|
|
|
|
- `llmgw doctor` says whether vLLM can be installed on the node, and why not.
|
|
|
|
|
|
|
|
|
|
Checked by installing vLLM 0.30.0 this way on a node of the test cluster: 7.7 GB,
|
|
|
|
|
torch 2.13 for CUDA 13.0, importing cleanly. That node has no GPU, so vLLM
|
|
|
|
|
itself was not started there.
|
|
|
|
|
|
|
|
|
|
### The project's website, in every gateway
|
|
|
|
|
|
|
|
|
|
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
|
|
|
|
|
the reverse proxy configurations, and the release's files to download. It is
|
|
|
|
|
served without a key like `/`, whose footer links to it, and says nothing about
|
|
|
|
|
the installation. Its fonts are in the binary, so it makes no request elsewhere.
|
|
|
|
|
The published website is rendered from the same template.
|
|
|
|
|
|
|
|
|
|
### The admin pages
|
|
|
|
|
|
|
|
|
|
- The cluster page leads with capacity, nodes, models and replicas, then usage;
|
|
|
|
|
it refreshes in place instead of reloading, and its usage table names people
|
|
|
|
|
and projects instead of ids.
|
|
|
|
|
- The usage page's legend gives each line's total, not its last hour.
|
|
|
|
|
- The people page keeps an account on one line and its rarer actions in a menu.
|
|
|
|
|
- Every page fits a phone: a wide table scrolls in its own box.
|
|
|
|
|
- Static files are named by their content, so a deploy's new styles and
|
|
|
|
|
scripts reach browsers at once instead of a day later.
|
|
|
|
|
|
|
|
|
|
### Metrics
|
|
|
|
|
|
|
|
|
|
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
|
|
|
|
|
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
|
|
|
|
|
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
|
|
|
|
|
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
|
|
|
|
|
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
|
|
|
|
|
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
|
|
|
|
|
|
|
|
|
|
### Fixes
|
|
|
|
|
|
|
|
|
|
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
|
|
|
|
|
rewriting read it as the start of a string. Comments are now skipped.
|
|
|
|
|
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
|
|
|
|
|
|
|
|
|
|
### Upgrading
|
|
|
|
|
|
|
|
|
|
Upgrade the control plane first, then the agents: an older agent is served as
|
|
|
|
|
before and simply reports none of the new measurements. Existing agent units
|
|
|
|
|
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
|
|
|
|
|
that set `mgmt_interface` keeps it; one that did not now uses the interface
|
|
|
|
|
holding its management address rather than `eth0`. Multi-node vLLM replicas
|
|
|
|
|
start with `mp` unless `launch_mode: ray` is set.
|
|
|
|
|
|
|
|
|
|
### An organization's own page
|
|
|
|
|
|
|
|
|
|
A member of an organization can be made its **manager** (`users join --org …
|
|
|
|
|
--role manager`, or the star on the people page). A manager gets `/org`: the
|
|
|
|
|
organization's usage over time by team, project or person, its teams and
|
|
|
|
|
projects with limits and spend, and its members — read-only, and only what was
|
|
|
|
|
spent in its projects. Administrators can open every organization's page.
|
|
|
|
|
|
|
|
|
|
### How a model runs, from the mirror page
|
|
|
|
|
|
|
|
|
|
Each model a mirrored file is serving now shows how it runs — always on, on
|
|
|
|
|
demand, or stopped — and it can be changed there, without the CLI. The serve
|
|
|
|
|
form asks the same: on demand loads the model when somebody asks for it and
|
|
|
|
|
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
|
|
|
|
|
mode and says to change it there, because the file is read on every start.
|
|
|
|
|
|
|
|
|
|
### Weights the nodes cannot load are not fetched
|
|
|
|
|
|
|
|
|
|
`kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
|
|
|
|
|
was downloaded, catalogued and placed, and failed on every node it was tried on.
|
|
|
|
|
The architecture was in the first kilobytes of the file all along. Now:
|
|
|
|
|
|
|
|
|
|
- The mirror page's registry listing shows each file's architecture and, for
|
|
|
|
|
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
|
|
|
|
|
it. A vision projector (`mmproj`) is marked as not a model on its own.
|
|
|
|
|
- A fetch reads the file's header first — a range request, not a download —
|
|
|
|
|
and refuses one that cannot be served, unless "fetch anyway" is ticked.
|
|
|
|
|
- Serving from the mirror and `llmgw ctl model add` refuse the same.
|
|
|
|
|
|
|
|
|
|
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
|
|
|
|
|
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
|
|
|
|
|
not refused, only not checked.
|
|
|
|
|
|
|
|
|
|
### Fixes
|
|
|
|
|
|
|
|
|
|
- Usage by team, project or organization, and the spend on the quotas page,
|
|
|
|
|
left out every request made on the chat page: they were joined through the
|
|
|
|
|
key, and the chat page has none. They now count where a request was spent.
|
|
|
|
|
- `/org` was answered with "no api key was provided" for a signed-in browser;
|
|
|
|
|
a test now drives every session page through the authentication middleware.
|
|
|
|
|
|
|
|
|
|
### People belong to many places, or to none
|
|
|
|
|
|
|
|
|
|
A person was in exactly one project, chosen at approval, and could not use the
|
|
|
|
@@ -315,23 +192,104 @@ Leaving a project revokes the keys the person made there, unless another
|
|
|
|
|
membership still reaches it. A personal key keeps no conversations or notes:
|
|
|
|
|
those are kept per project.
|
|
|
|
|
|
|
|
|
|
**Upgrading.** Migration 0010 turns everyone's project into a membership and
|
|
|
|
|
attributes past usage to people where a key had an owner. On SQLite it rebuilds
|
|
|
|
|
the keys table; it was run against a copy of a production PostgreSQL store
|
|
|
|
|
before release. `llmgw users move` is gone: use `join` and `leave`.
|
|
|
|
|
### An organization's own page
|
|
|
|
|
|
|
|
|
|
### One menu on every page
|
|
|
|
|
A member of an organization can be made its **manager** (`users join --org …
|
|
|
|
|
--role manager`, or the star on the people page). A manager gets `/org`: the
|
|
|
|
|
organization's usage over time by team, project or person, its teams and
|
|
|
|
|
projects with limits and spend, and its members — read-only, and only what was
|
|
|
|
|
spent in its projects. Administrators can open every organization's page.
|
|
|
|
|
|
|
|
|
|
Each page drew its own menu, and no two agreed: different links, in a different
|
|
|
|
|
### The chat page
|
|
|
|
|
|
|
|
|
|
- A model that is not running shows a progress bar while it loads — waiting
|
|
|
|
|
for a node, downloading, starting — and the message is sent once it is up,
|
|
|
|
|
rather than failing after 45 seconds with "the model is loading".
|
|
|
|
|
- A finished reply is rendered as markdown, by the server under the model
|
|
|
|
|
cards' rules, with its maths typeset by KaTeX: "$5 and $10" stays prose.
|
|
|
|
|
- Stop, while a reply streams; its tokens, speed and time to first token under
|
|
|
|
|
it; a system prompt and a temperature; and the conversation kept in the
|
|
|
|
|
browser across a reload — the gateway still keeps none.
|
|
|
|
|
|
|
|
|
|
### Connect an editor, and the Responses API
|
|
|
|
|
|
|
|
|
|
- The account page makes a key for **Continue**, **Codex** or **Cline** and
|
|
|
|
|
shows what the tool needs, filled in: Continue's assistant file, Codex's
|
|
|
|
|
provider and profile, Cline's four settings. Continue and Codex get a one-line
|
|
|
|
|
command (`curl -fsSL <gateway>/setup/<tool>?model=… | sh`) whose script holds
|
|
|
|
|
no secret — the key comes from the environment — keeps a copy of whatever it
|
|
|
|
|
replaces, and leaves the rest of the file alone.
|
|
|
|
|
- **`POST /v1/responses`** translates the Responses API, which Codex speaks, to
|
|
|
|
|
chat completions and back — instructions, messages, function and custom
|
|
|
|
|
tools with their calls and outputs, usage, streamed or not — down the same
|
|
|
|
|
path as any request.
|
|
|
|
|
|
|
|
|
|
### Your own use
|
|
|
|
|
|
|
|
|
|
The account page (was "Your keys") shows the last thirty days of a person's own
|
|
|
|
|
use and, for each place they spend, a bar for every budget that applies — their
|
|
|
|
|
own, or a project's shared with others, said as such.
|
|
|
|
|
|
|
|
|
|
### The mirror page
|
|
|
|
|
|
|
|
|
|
- **Explore models** opens on what is trending on Hugging Face, with views for
|
|
|
|
|
new and most downloaded, filters for task and size, and each repository's
|
|
|
|
|
parameters, architecture (✓ where the nodes' llama.cpp loads it), context,
|
|
|
|
|
base model and license. "Only what these nodes can run" is on by default.
|
|
|
|
|
- A repository opens beneath its row with its model card, and a download
|
|
|
|
|
started there starts in place, its progress followed in the table above.
|
|
|
|
|
- What the mirror holds is grouped by model family.
|
|
|
|
|
- An importance matrix published beside a model's quantisations is no longer
|
|
|
|
|
offered as a model.
|
|
|
|
|
|
|
|
|
|
**How a model runs.** Each model a mirrored file is serving now shows how it runs — always on, on
|
|
|
|
|
demand, or stopped — and it can be changed there, without the CLI. The serve
|
|
|
|
|
form asks the same: on demand loads the model when somebody asks for it and
|
|
|
|
|
stops it after 15 minutes unused. A model declared in `server.yaml` shows its
|
|
|
|
|
mode and says to change it there, because the file is read on every start.
|
|
|
|
|
|
|
|
|
|
**Weights the nodes cannot load are not fetched.** `kev-0.5b` was a GGUF whose architecture, `ggmlc`, llama.cpp does not know. It
|
|
|
|
|
was downloaded, catalogued and placed, and failed on every node it was tried on.
|
|
|
|
|
The architecture was in the first kilobytes of the file all along. Now:
|
|
|
|
|
|
|
|
|
|
- The mirror page's registry listing shows each file's architecture and, for
|
|
|
|
|
one the nodes' llama.cpp cannot load, says so instead of offering to fetch
|
|
|
|
|
it. A vision projector (`mmproj`) is marked as not a model on its own.
|
|
|
|
|
- A fetch reads the file's header first — a range request, not a download —
|
|
|
|
|
and refuses one that cannot be served, unless "fetch anyway" is ticked.
|
|
|
|
|
- Serving from the mirror and `llmgw ctl model add` refuse the same.
|
|
|
|
|
|
|
|
|
|
The list of architectures is llama.cpp's own (`LLM_ARCH_NAMES`) at the build
|
|
|
|
|
llmgw pins, b10991: 152 of them. A different build configured in `engines:` is
|
|
|
|
|
not refused, only not checked.
|
|
|
|
|
|
|
|
|
|
### Each node's CPU and memory, and a page per node
|
|
|
|
|
|
|
|
|
|
The agent reads `/proc` between heartbeats: CPU across all cores, load, memory
|
|
|
|
|
used and available, and each engine's cores and resident memory. The cluster
|
|
|
|
|
page's node table shows CPU and RAM as bars; each node has a page with its
|
|
|
|
|
current load, its engines, its fabric ports, its GPUs, and an hour of charts
|
|
|
|
|
from Prometheus.
|
|
|
|
|
|
|
|
|
|
### The admin pages
|
|
|
|
|
|
|
|
|
|
- The cluster page leads with capacity, nodes, models and replicas, then usage;
|
|
|
|
|
it refreshes in place instead of reloading, and its usage table names people
|
|
|
|
|
and projects instead of ids.
|
|
|
|
|
- The usage page's legend gives each line's total, not its last hour.
|
|
|
|
|
- The people page keeps an account on one line and its rarer actions in a menu.
|
|
|
|
|
- Every page fits a phone: a wide table scrolls in its own box.
|
|
|
|
|
- Static files are named by their content, so a deploy's new styles and
|
|
|
|
|
scripts reach browsers at once instead of a day later.
|
|
|
|
|
|
|
|
|
|
**One menu on every page.** Each page drew its own menu, and no two agreed: different links, in a different
|
|
|
|
|
order, the page you were on sometimes listed and sometimes not. Every signed-in
|
|
|
|
|
page now draws the same one — Cluster, People, Quotas, Usage, Mirror on the
|
|
|
|
|
left for an administrator, Chat, Account and Sign out on the right for
|
|
|
|
|
everyone — with the current page marked. The chat and key pages name the
|
|
|
|
|
person's project by its path rather than its id.
|
|
|
|
|
|
|
|
|
|
### The quotas page says what actually applies
|
|
|
|
|
|
|
|
|
|
A branch that sets no limit of its own used to read "—" in every column while
|
|
|
|
|
**The quotas page says what actually applies.** A branch that sets no limit of its own used to read "—" in every column while
|
|
|
|
|
its organization held it to sixty requests a minute. Each limit now shows the
|
|
|
|
|
value it is held to from above, muted and marked ↑, with where it comes from on
|
|
|
|
|
hover; a larger value set below a stricter one says what caps it, for rate and
|
|
|
|
@@ -342,6 +300,54 @@ form or saving one no longer reloads the page, and the explanations moved under
|
|
|
|
|
the start of their path, which misplaced them when two organizations shared a
|
|
|
|
|
name.
|
|
|
|
|
|
|
|
|
|
### The project's website, in every gateway
|
|
|
|
|
|
|
|
|
|
`/about` is llmgw's own page: what it does, InfiniBand and NVLink, installing,
|
|
|
|
|
the reverse proxy configurations, and the release's files to download. It is
|
|
|
|
|
served without a key like `/`, whose footer links to it, and says nothing about
|
|
|
|
|
the installation. Its fonts are in the binary, so it makes no request elsewhere.
|
|
|
|
|
The published website is rendered from the same template.
|
|
|
|
|
|
|
|
|
|
### Metrics
|
|
|
|
|
|
|
|
|
|
`llmgw_node_cpu_utilization_percent`, `llmgw_node_cpus`, `llmgw_node_load`,
|
|
|
|
|
`llmgw_node_memory_used_bytes`, `llmgw_node_memory_available_bytes`,
|
|
|
|
|
`llmgw_shard_cpu_cores`, `llmgw_shard_memory_bytes`, `llmgw_ib_port_active`,
|
|
|
|
|
`llmgw_ib_port_rate_gbps`, `llmgw_ib_port_counter_total`,
|
|
|
|
|
`llmgw_fabric_master_sm`, `llmgw_fabric_subnet_managers`,
|
|
|
|
|
`llmgw_fabric_problems`, `llmgw_gpu_nvlink_joined`, `llmgw_imex_up`.
|
|
|
|
|
|
|
|
|
|
### Fixes
|
|
|
|
|
|
|
|
|
|
- PostgreSQL queries with an apostrophe in a comment failed: the placeholder
|
|
|
|
|
rewriting read it as the start of a string. Comments are now skipped.
|
|
|
|
|
- Chart axes read "20.0BB" for sizes and 0% for a quiet CPU.
|
|
|
|
|
- Usage by team, project or organization, and the spend on the quotas page,
|
|
|
|
|
left out every request made on the chat page: they were joined through the
|
|
|
|
|
key, and the chat page has none. They now count where a request was spent.
|
|
|
|
|
- `/org` was answered with "no api key was provided" for a signed-in browser;
|
|
|
|
|
a test now drives every session page through the authentication middleware.
|
|
|
|
|
|
|
|
|
|
### Upgrading
|
|
|
|
|
|
|
|
|
|
Upgrade the control plane first, then the agents: an older agent is served as
|
|
|
|
|
before and simply reports none of the new measurements. Existing agent units
|
|
|
|
|
keep their old limits until `LimitMEMLOCK=infinity` is added to them. A node
|
|
|
|
|
that set `mgmt_interface` keeps it; one that did not now uses the interface
|
|
|
|
|
holding its management address rather than `eth0`. Multi-node vLLM replicas
|
|
|
|
|
start with `mp` unless `launch_mode: ray` is set.
|
|
|
|
|
|
|
|
|
|
The store migrates on its first start. Migration 0010 turns everyone's project into a membership and
|
|
|
|
|
attributes past usage to people where a key had an owner. On SQLite it rebuilds
|
|
|
|
|
the keys table; it was run against a copy of a production PostgreSQL store
|
|
|
|
|
before release. `llmgw users move` is gone: use `join` and `leave`.
|
|
|
|
|
|
|
|
|
|
vLLM is installed on every GPU node that can run it, once its agent is
|
|
|
|
|
upgraded: about 8 GB each, from PyPI. Set `engines.vllm: ""` in `server.yaml`
|
|
|
|
|
before upgrading to keep vLLM your own. `deploy/bootstrap/onboard-node.sh` is
|
|
|
|
|
replaced by `llmgw node add`; it still works, for a machine holding the
|
|
|
|
|
authority but not this llmgw.
|
|
|
|
|
|
|
|
|
|
## v0.9.2 — 2026-09-28
|
|
|
|
|
|
|
|
|
|
### A new control plane serves a model by itself
|
|
|
|
|