-
v0.10.6-rc.8
Pre-Release
gitea-actions
released this
2026-10-01 07:39:12 +02:00 | 544 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
LoRA adapters
A llama.cpp model can carry
adapters, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (--lora-scaled FILE:0); a request that names an
adapter goes to the model's replicas with"lora": [{"id": i, "scale": s}],
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in/v1/models, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.Finding and attaching adapters from the mirror page
- Explore has a LoRA adapters view: adapters for any model, or — from a
held model's card, find adapters on Hugging Face — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original. - Opening a repository reads the first bytes of each GGUF file and marks it
a LoRA adapter or a model. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says. - Held adapter files are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog. llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]
lists and changes them from the command line, through
POST /api/v1/models/{name}/adapters, with the same checks.
Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere:
external: {url, model, api_key_file}and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model'sfallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md).ctl modelsand the cluster page mark it
external.Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.7
Pre-Release
gitea-actions
released this
2026-10-01 07:30:13 +02:00 | 545 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
LoRA adapters
A llama.cpp model can carry
adapters, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (--lora-init-without-apply); a request that names an
adapter goes to the model's replicas with"lora": [{"id": i, "scale": s}],
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in/v1/models, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.Finding and attaching adapters from the mirror page
- Explore has a LoRA adapters view: adapters for any model, or — from a
held model's card, find adapters on Hugging Face — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original. - Opening a repository reads the first bytes of each GGUF file and marks it
a LoRA adapter or a model. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says. - Held adapter files are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog. llmgw ctl model adapters MODEL [--add NAME=URL[@SCALE]] [--remove NAME]
lists and changes them from the command line, through
POST /api/v1/models/{name}/adapters, with the same checks.
Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere:
external: {url, model, api_key_file}and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model'sfallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md).ctl modelsand the cluster page mark it
external.Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.6
Pre-Release
gitea-actions
released this
2026-10-01 07:20:35 +02:00 | 546 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
LoRA adapters
A llama.cpp model can carry
adapters, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (--lora-init-without-apply); a request that names an
adapter goes to the model's replicas with"lora": [{"id": i, "scale": s}],
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in/v1/models, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.Finding and attaching adapters from the mirror page
- Explore has a LoRA adapters view: adapters for any model, or — from a
held model's card, find adapters on Hugging Face — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original. - Opening a repository reads the first bytes of each GGUF file and marks it
a LoRA adapter or a model. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says. - Held adapter files are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere:
external: {url, model, api_key_file}and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model'sfallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md).ctl modelsand the cluster page mark it
external.Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.5
Pre-Release
gitea-actions
released this
2026-10-01 07:19:08 +02:00 | 547 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
LoRA adapters
A llama.cpp model can carry
adapters, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (--lora-init-without-apply); a request that names an
adapter goes to the model's replicas with"lora": [{"id": i, "scale": s}],
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in/v1/models, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.Finding and attaching adapters from the mirror page
- Explore has a LoRA adapters view: adapters for any model, or — from a
held model's card, find adapters on Hugging Face — those made for the model
its repository was made from, since a GGUF repository is a quantisation and
adapters are made for the original. - Opening a repository reads the first bytes of each GGUF file and marks it
a LoRA adapter or a model. Repositories tagged "lora" hold a model with
the adapter merged in about as often as an adapter, and only the file says. - Held adapter files are marked in the list, and a model's card lists its
adapters, offers the held adapter files of its architecture to attach under a
name and scale, and removes them. A file that is not an adapter, or is for
another architecture, is refused before it reaches the catalog.
Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere:
external: {url, model, api_key_file}and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model'sfallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md).ctl modelsand the cluster page mark it
external.Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.4
Pre-Release
gitea-actions
released this
2026-09-30 23:40:47 +02:00 | 548 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
LoRA adapters
A llama.cpp model can carry
adapters, each a name and a GGUF LoRA file. The
nodes fetch them with the model's weights and llama.cpp loads them all at start,
applied to nothing (--lora-init-without-apply); a request that names an
adapter goes to the model's replicas with"lora": [{"id": i, "scale": s}],
so one replica serves the model and all its adapters. Adapter names share the
namespace of models, are listed in/v1/models, and follow key allowlists on
their own — a key allowed an adapter does not reach the model under it. Adapter
files are kept from weight cleanup like the model's own. Not yet for vLLM or
SGLang profiles, whose adapters are loaded differently.Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere:
external: {url, model, api_key_file}and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model'sfallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md).ctl modelsand the cluster page mark it
external.Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.3
Pre-Release
gitea-actions
released this
2026-09-30 23:29:01 +02:00 | 549 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
Models served outside the cluster
A catalog model can be an OpenAI-compatible API elsewhere:
external: {url, model, api_key_file}and no profiles. Nothing is placed for it; the router
sends its requests to the provider, rewritten to the provider's name for the
model and with its key, read from the file on every publish so a rotated key is
picked up. Named in another model'sfallbacks, it is used only when the
cluster has no capacity left. It is never scraped for metrics, and never reads
as stale. Prices, budgets, limits, usage and key allowlists apply as to any
model — and since its prompts leave the cluster, the allowlist is how to decide
who may send them (docs/install.md).ctl modelsand the cluster page mark it
external.Pool spillover
A model served from pools of different priority — H100 before A100, say — was
routed to the preferred pool however full it was. Now a pool whose replicas are
saturated, by the same measure admission uses (queue past its healthy depth, or
KV cache past its ceiling), passes requests to the next pool that is not. When
every pool is saturated the preferred one is kept: a pool that is just as full
gains nothing.Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.2
Pre-Release
gitea-actions
released this
2026-09-30 23:07:53 +02:00 | 551 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
Displacement: a higher band is not refused while a lower one waits
Each band's queue was bounded, and nothing bounded them together, so a full
gateway turned an interactive request away while batch requests sat queued
behind it.admission.queue_capacity_totalnow bounds everything queued —
by default half of every band's capacity together. Reached, a request is not
refused while a lower band has any queued: the newest request of the lowest
band is turned away instead, with a 503 andRetry-After: 5, so it backs off
and returns. The newest, because the oldest have waited longest and are
nearest their TTL. Counted asdisplacedin the admission rejection metric.Releases can be signed
llmgw release keygenmakes an ed25519 key pair. With the private key in the
CI secretRELEASE_SIGNING_KEY, releases carrychecksums.txt.sig; with the
public key ininternal/release/trusted.pub, every binary built after refuses
a release not signed with it, wherever it installs one. Off until then, and
docs/install.md says in which order to turn it on.Restarts borrow a lower class's room
A restart places the replacement beside the replica it replaces, and where
there is no free room it now makes some the way any placement does: by
stopping a replica its class outranks, which gets its room back once the old
replica has gone. The in-place restart — the model unserved while its
replacement loads — is left for when nothing of a lower class is running. On a
cluster where every model is standard, that is every full one: mark the models
that can waitopportunistic(llmgw ctl model class, or the model's card).Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.6-rc.1
Pre-Release
gitea-actions
released this
2026-09-30 22:40:45 +02:00 | 555 commits to main since this releaseChanges since v0.10.5, published as release candidates (v0.10.6-rc.N) until
there are enough for v0.10.6.Capacity from the admin pages
What v0.10.5 made settable from the command line is now on the pages too:
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
cores and memory beside what it holds, with a form to set, change or remove
one. It goes through the same checks asllmgw ctl quota. - A model's card on the mirror page sets its class, its owner (when there
are quotas to count against), and the memory and cores a replica reserves,
beside its replica counts. Only what changed is applied, and each change is
said. - Room counts what a class may take. The card,
llmgw ctl model targetand
the headroom API count the replicas a model could get by stopping those its
class outranks, separately from free room, and name whose they would be. A
critical model's card no longer refuses a replica that preemption would make
room for.
Downloads
- Quotas has a Capacity section: every scope's guaranteed and most GPUs,
-
v0.10.5
Stable
gitea-actions
released this
2026-09-30 22:26:19 +02:00 | 556 commits to main since this releasePublished first as release candidates v0.10.5-rc.1 to rc.14; rc.5 failed its
format check and has no binaries.After upgrading:
- Declare what CPU models need. A CPU model added from now on must say its
memory; one stored before this still loads and is flagged inctl models
untilllmgw ctl model resources MODEL --memory GB --cores Nis run. Cores
are then reserved per replica and become the engine's thread count, for
replicas started after the change:llmgw ctl restart --model MODEL. - Upgrade the mirror on its own host, now with one command:
sudo llmgw upgrade v0.10.5 --service llmgw-mirror(by hand once, for a
mirror older than this).llmgw ctl upgradesays so at the end.
CPU cores are reserved, and memory has to be declared
The first step of resource allocation for larger clusters: what a replica
holds of a node is reserved, not assumed.- Cores. A profile's
cpu_coresis reserved out of its node's CPUs, as
memory is, and is what its engine computes on (llama.cpp's--threads). A
node holds as many CPU replicas as it has cores for, and two on one node no
longer spin against each other. Agents report their CPU count; a node that
does not, and a profile that declares no cores, constrain nothing. - Memory is required of new CPU models. A model with no GPU is all host
memory, and one declaring none looked to placement as if it fitted
everywhere, any number of times. Adding one withoutmin_memory_gbis now
refused. Models stored before this still load, and say what is missing. - Declared against measured. Each model shows what one replica reserves
and the most one of its replicas uses — inctl models(RESERVES) and on the
cluster page — with a warning when nothing is declared or when it uses more
than a tenth over what it reserves. - Setting them.
llmgw ctl model resources MODEL --memory GB --cores N
(andPOST /api/v1/models/{name}/resources) for a model added at runtime.
ctl model serveand the mirror's serve form fill in cores — half the
smallest node the model fits on — beside the memory they already estimate.
Running replicas keep their old figures until restarted.
The commands are
llmgw ctl model resourcesandllmgw ctl model class; in
rc.7 the first was brieflyllmgw ctl resources.Capacity classes, and preemption
A model has a class —
critical,standardoropportunistic— saying what
its capacity is worth against another's when there is not enough for both.
Unset, an on-demand model is opportunistic and every other model standard,
which is how the catalog behaved before.llmgw ctl model class MODEL CLASS
sets it.When a model cannot be placed, replicas are stopped to make room for it, by
these rules:- A lower class gives way to a higher one.
- Within a class, only replicas above a model's minimum — room it grew into
under load — and only for a model still below its own minimum. - A critical model's minimum is never taken, nor a replica that has only just
loaded, nor one spanning several nodes. - Idle on-demand replicas go first, then what lower classes grew into, then
the same within the class, then lower classes' minimums; the longest unused
first within each, and of equally idle ones the smallest. - The set is tried on a copy of the inventory first, at most three replicas,
and nothing is stopped unless it makes the room. Of a set that does, each
replica whose stopping turns out not to be needed is left running, so an
idle replica tried first on the wrong node is not stopped with the one that
made the room.
A replica stopped this way is drained and given 30 seconds to finish what it
is serving. Its model is held back for up to five minutes, or until the model
it gave way to has its replicas, so it does not take the room straight back.
ctl modelsshows each model's class and "gave room to X" while held; the
cluster page shows the class and how many replicas were lost; the metric is
llmgw_model_preempted_total{model,class}, and every preemption is logged.Capacity quotas for the tenancy tree
An organization, team or project can be guaranteed GPUs, CPU cores and
memory, and limited to at most some amount of each. What it holds is what the
replicas of the models it owns hold — and those of the teams and projects
beneath it.llmgw ctl model owner MODEL team:Acme/mlsays whose a model is;
one with no owner is shared and counts against nobody.- A guarantee is kept. A replica within its owner's guarantee is never
taken for another scope's model, whatever the classes: a guarantee a higher
class elsewhere could take would not be one. - Beyond it is borrowing. Free room is used; when a scope within its own
guarantee needs room, what others hold beyond theirs is taken back first,
whatever their class — except a critical model's minimum. - A limit is a ceiling. A model whose owner would go past a limit anywhere
up the tree gets no further replica, and says which limit; headroom and the
mirror card stop there too.
llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16(also--cores,
--memoryand their limits),llmgw ctl quota listwith what each scope holds,
llmgw ctl quota clear. Scopes are named by path or ID; a name that could mean
two is refused. Changes take effect within half a minute. Metrics:
llmgw_capacity_holds,llmgw_capacity_guaranteeandllmgw_capacity_limit
by scope and resource; alert: CapacityAtLimit.Sized by what a replica was measured to serve
What
llmgw ctl bench MODEL --savemeasured — the most requests at once one
replica served while meeting its SLO, and its throughput there — now drives
decisions rather than only being printed:- Scaling up when requests per replica pass what was measured, not only
when the engines' queues fill: an engine takes requests into its slots long
after latency has gone. - Scaling down only when the replicas left would stay under 70% of what
they were measured to hold, so a replica taken back does not put the rest
straight over. - Choosing a profile. Among profiles of equal priority, the one measured
to serve the most per GPU — or per core — is tried first. Priority set by an
operator still wins. - Sizing.
llmgw ctl model target MODEL --tokens-per-sec Nsays how many
replicas that takes, how many run, and whether the rest fit, with the reason
when they do not;--applysets it if they do. Headroom reports what one
more replica would serve.
A model never measured behaves exactly as before. Measurements are read from
the store every half minute.Upgrading a machine the control plane does not reach
sudo llmgw upgrade VERSION --service UNITupgrades the machine it runs on —
the mirror's host above all, whose service runs unprivileged and cannot replace
its own binary as an agent does. The release's checksums are checked, the old
binary kept, the unit restarted and watched until it stays up.Nodes report their own heat and power
The first step of facility-aware placement (docs/facility-aware-placement.md),
observe-only: nothing is decided on these yet. Every agent reads, at most every
ten seconds, and sends with its heartbeat:- GPUs from nvidia-smi: temperature, HBM temperature where the driver has
it, power draw and the enforced limit, and what is holding the clocks down —
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
at registration; it is now current. - CPUs from hwmon, which is what lm_sensors reads, with nothing to install:
the package temperature, and the board's inlet air sensor where it has one. - CPU power from RAPL's energy counters.
A sensor a machine does not have is not reported, never shown as zero. New
metrics:llmgw_gpu_power_watts,llmgw_gpu_power_limit_watts,
llmgw_gpu_memory_temperature_celsius,llmgw_gpu_throttled{reason},
llmgw_node_cpu_temperature_celsius,llmgw_node_cpu_power_watts,
llmgw_node_inlet_temperature_celsius. Each node's page shows them, with a GPU
table of temperature, power against its limit, and what is throttling it.The mirror page in tabs
/admin/mirror was one long page: what the mirror holds, the Hugging Face
explorer and the fetch form, one under another, with every visit waiting on
Hugging Face for a list it might not show. It is now three tabs — Held (with
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
URL — each with its own address. Only the tab shown does its work, and only
Held reloads while something downloads. A download started from the explorer
links to Held, where its progress is; a fetch the mirror refused comes back on
the form it was sent from.A held model opens to where it runs, and how many
Clicking a model under Held opens a card beneath it: each replica with its
node, state, age and the CPU and memory its engine uses, and how many more
replicas the cluster has room for now, on which nodes, and why not more. The
replicas it always runs and the most it grows to under load are changed there.
Room is found by trying the scheduler itself on a copy of the inventory — the
same decisions the next passes would make, not an estimate beside them — and a
number of always-running replicas it cannot place is refused with the reason,
rather than accepted and left short. A ceiling above the room is taken, and
said. The same answer is atGET /api/v1/models/{name}/headroomon the admin
API.Each row also says how its model runs without being opened: replicas ready of
those it should have, how many are still starting, and the nodes they are on —
in amber when short, in red when parked or when it does not fit. It is one line
per model beside its mode, and nodes drop the part of the name every node
shares (node05 for llmgw-node05, the full name on hover).One line per record, in every table
The same rule across the admin pages. Headers do not wrap; a record is one
line; free text is cut to its column with the whole of it on hover rather than
given a row of its own; a row across the whole table is only a note or an open
form. On the cluster page that means a node is one line (its address and logs
beside its name, not under it), a model's description is a column rather than a
second row, node names drop the prefix they all share, and the GPUs, classes,
fabrics and fallbacks columns appear only when some row has something in them —
on a CPU-only cluster they were three columns of dashes pushing the table past
the page. On People, a person in many teams shows two and a count, the rest on
hover.Downloads
- Declare what CPU models need. A CPU model added from now on must say its
-
v0.10.5-rc.14
Pre-Release
gitea-actions
released this
2026-09-30 19:49:08 +02:00 | 558 commits to main since this releaseChanges since v0.10.4, published as release candidates (v0.10.5-rc.N) until
there are enough for v0.10.5.CPU cores are reserved, and memory has to be declared
The first step of resource allocation for larger clusters: what a replica
holds of a node is reserved, not assumed.- Cores. A profile's
cpu_coresis reserved out of its node's CPUs, as
memory is, and is what its engine computes on (llama.cpp's--threads). A
node holds as many CPU replicas as it has cores for, and two on one node no
longer spin against each other. Agents report their CPU count; a node that
does not, and a profile that declares no cores, constrain nothing. - Memory is required of new CPU models. A model with no GPU is all host
memory, and one declaring none looked to placement as if it fitted
everywhere, any number of times. Adding one withoutmin_memory_gbis now
refused. Models stored before this still load, and say what is missing. - Declared against measured. Each model shows what one replica reserves
and the most one of its replicas uses — inctl models(RESERVES) and on the
cluster page — with a warning when nothing is declared or when it uses more
than a tenth over what it reserves. - Setting them.
llmgw ctl model resources MODEL --memory GB --cores N
(andPOST /api/v1/models/{name}/resources) for a model added at runtime.
ctl model serveand the mirror's serve form fill in cores — half the
smallest node the model fits on — beside the memory they already estimate.
Running replicas keep their old figures until restarted.
The commands are
llmgw ctl model resourcesandllmgw ctl model class; in
rc.7 the first was brieflyllmgw ctl resources.Capacity classes, and preemption
A model has a class —
critical,standardoropportunistic— saying what
its capacity is worth against another's when there is not enough for both.
Unset, an on-demand model is opportunistic and every other model standard,
which is how the catalog behaved before.llmgw ctl model class MODEL CLASS
sets it.When a model cannot be placed, replicas are stopped to make room for it, by
these rules:- A lower class gives way to a higher one.
- Within a class, only replicas above a model's minimum — room it grew into
under load — and only for a model still below its own minimum. - A critical model's minimum is never taken, nor a replica that has only just
loaded, nor one spanning several nodes. - Idle on-demand replicas go first, then what lower classes grew into, then
the same within the class, then lower classes' minimums; the longest unused
first within each, and of equally idle ones the smallest. - The set is tried on a copy of the inventory first, at most three replicas,
and nothing is stopped unless it makes the room. Of a set that does, each
replica whose stopping turns out not to be needed is left running, so an
idle replica tried first on the wrong node is not stopped with the one that
made the room.
A replica stopped this way is drained and given 30 seconds to finish what it
is serving. Its model is held back for up to five minutes, or until the model
it gave way to has its replicas, so it does not take the room straight back.
ctl modelsshows each model's class and "gave room to X" while held; the
cluster page shows the class and how many replicas were lost; the metric is
llmgw_model_preempted_total{model,class}, and every preemption is logged.Capacity quotas for the tenancy tree
An organization, team or project can be guaranteed GPUs, CPU cores and
memory, and limited to at most some amount of each. What it holds is what the
replicas of the models it owns hold — and those of the teams and projects
beneath it.llmgw ctl model owner MODEL team:Acme/mlsays whose a model is;
one with no owner is shared and counts against nobody.- A guarantee is kept. A replica within its owner's guarantee is never
taken for another scope's model, whatever the classes: a guarantee a higher
class elsewhere could take would not be one. - Beyond it is borrowing. Free room is used; when a scope within its own
guarantee needs room, what others hold beyond theirs is taken back first,
whatever their class — except a critical model's minimum. - A limit is a ceiling. A model whose owner would go past a limit anywhere
up the tree gets no further replica, and says which limit; headroom and the
mirror card stop there too.
llmgw ctl quota set team:Acme/ml --gpus 8 --limit-gpus 16(also--cores,
--memoryand their limits),llmgw ctl quota listwith what each scope holds,
llmgw ctl quota clear. Scopes are named by path or ID; a name that could mean
two is refused. Changes take effect within half a minute. Metrics:
llmgw_capacity_holds,llmgw_capacity_guaranteeandllmgw_capacity_limit
by scope and resource; alert: CapacityAtLimit.Sized by what a replica was measured to serve
What
llmgw ctl bench MODEL --savemeasured — the most requests at once one
replica served while meeting its SLO, and its throughput there — now drives
decisions rather than only being printed:- Scaling up when requests per replica pass what was measured, not only
when the engines' queues fill: an engine takes requests into its slots long
after latency has gone. - Scaling down only when the replicas left would stay under 70% of what
they were measured to hold, so a replica taken back does not put the rest
straight over. - Choosing a profile. Among profiles of equal priority, the one measured
to serve the most per GPU — or per core — is tried first. Priority set by an
operator still wins. - Sizing.
llmgw ctl model target MODEL --tokens-per-sec Nsays how many
replicas that takes, how many run, and whether the rest fit, with the reason
when they do not;--applysets it if they do. Headroom reports what one
more replica would serve.
A model never measured behaves exactly as before. Measurements are read from
the store every half minute.Nodes report their own heat and power
The first step of facility-aware placement (docs/facility-aware-placement.md),
observe-only: nothing is decided on these yet. Every agent reads, at most every
ten seconds, and sends with its heartbeat:- GPUs from nvidia-smi: temperature, HBM temperature where the driver has
it, power draw and the enforced limit, and what is holding the clocks down —
power cap, thermal slowdown, power brake. A GPU's temperature was read once,
at registration; it is now current. - CPUs from hwmon, which is what lm_sensors reads, with nothing to install:
the package temperature, and the board's inlet air sensor where it has one. - CPU power from RAPL's energy counters.
A sensor a machine does not have is not reported, never shown as zero. New
metrics:llmgw_gpu_power_watts,llmgw_gpu_power_limit_watts,
llmgw_gpu_memory_temperature_celsius,llmgw_gpu_throttled{reason},
llmgw_node_cpu_temperature_celsius,llmgw_node_cpu_power_watts,
llmgw_node_inlet_temperature_celsius. Each node's page shows them, with a GPU
table of temperature, power against its limit, and what is throttling it.The mirror page in tabs
/admin/mirror was one long page: what the mirror holds, the Hugging Face
explorer and the fetch form, one under another, with every visit waiting on
Hugging Face for a list it might not show. It is now three tabs — Held (with
how many are held, downloading and failed), Explore Hugging Face, and Fetch by
URL — each with its own address. Only the tab shown does its work, and only
Held reloads while something downloads. A download started from the explorer
links to Held, where its progress is; a fetch the mirror refused comes back on
the form it was sent from.A held model opens to where it runs, and how many
Clicking a model under Held opens a card beneath it: each replica with its
node, state, age and the CPU and memory its engine uses, and how many more
replicas the cluster has room for now, on which nodes, and why not more. The
replicas it always runs and the most it grows to under load are changed there.
Room is found by trying the scheduler itself on a copy of the inventory — the
same decisions the next passes would make, not an estimate beside them — and a
number of always-running replicas it cannot place is refused with the reason,
rather than accepted and left short. A ceiling above the room is taken, and
said. The same answer is atGET /api/v1/models/{name}/headroomon the admin
API.Each row also says how its model runs without being opened: replicas ready of
those it should have, how many are still starting, and the nodes they are on —
in amber when short, in red when parked or when it does not fit. It is one line
per model beside its mode, and nodes drop the part of the name every node
shares (node05 for llmgw-node05, the full name on hover).One line per record, in every table
The same rule across the admin pages. Headers do not wrap; a record is one
line; free text is cut to its column with the whole of it on hover rather than
given a row of its own; a row across the whole table is only a note or an open
form. On the cluster page that means a node is one line (its address and logs
beside its name, not under it), a model's description is a column rather than a
second row, node names drop the prefix they all share, and the GPUs, classes,
fabrics and fallbacks columns appear only when some row has something in them —
on a CPU-only cluster they were three columns of dashes pushing the table past
the page. On People, a person in many teams shows two and a count, the rest on
hover.Downloads
- Cores. A profile's